PS2 wLaunchELF Release Thread

Discussion in 'PS2 Homebrew' started by sp193, May 10, 2017.

  1. 68
    94
    42
    El_Patas

    El_Patas Member

    Joined:
    Aug 10, 2017
    Messages:
    68
    Likes Received:
    94
    Trophy Points:
    42
    Gender:
    Male
    Hi @sp193,

    thank you very much for the relaxation on the Unicode, i tested it and the characters works fine.
    I hope you can find a good way for add complete UTF-8 support for the USBHDFSD, i appreciate
    your work.

    And the internal HDD do not support complete UTF-8? when i wrote the name of a game with
    a special spanish character, the character do not appears in the games list of the OPL,
    and the name appears cutted.
    Or is the OPL which do not have support for UTF-8?
    Would there be some way to fix this?
    Well sorry, maybe this will go in another thread.

    Best regards.
     
  2. 1,504
    1,191
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,504
    Likes Received:
    1,191
    Trophy Points:
    347
    Gender:
    Male
    Are there any differences between "BOOT-181028.ELF", "BOOT-181021.ELF"?
    Because when I go into page "About uLE", both versions have the same commit 9db3bcc from Oct 21.
     
  3. 728
    1,302
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    728
    Likes Received:
    1,302
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    Yes, USBHDFSD changed.
    This bug, although it is visible through your activities within LaunchELF, is not actually a LaunchELF bug. So when we change USBHDFSD by changing the PS2SDK, LaunchELF receives no new commits and so its last commit hash remains the same.

    I am not sure how you did it, but APA is officially documented to only accept some ASCII characters for partition names. So don't enter foreign characters in, even if the tool you use allows it. HDLGameInstaller will replace all unsupported characters within the partition name with underscores (_). The maximum length is 32.

    However, the HDLoader game title we have is stored in UTF-8. OPL supports this. And because UTF-8 is backward-compatible with ASCII strings, it will still work with HDLoader and LaunchELF. So please change the title instead of the partition name.

    In comparison to FAT, the FAT LFN was documented to have its filenames stored in the "OEM" encoding. Although this meant that the encoding was not specified, Windows was the primary user of it, so the encoding used by Windows should be what we use for compatibility. Which means UCS-2, which can represent Unicode.
     
    Last edited: Oct 28, 2018
  4. 62
    19
    37
    Agrippa

    Agrippa Member

    Joined:
    Sep 25, 2018
    Messages:
    62
    Likes Received:
    19
    Trophy Points:
    37
    Gender:
    Male
    I've downloaded this release to test the USB changes to make sure everything works correctly. And yes, it does, but I'm generally using the uLE_kHn_20160723 version for POPS's VCD support. I don't remember if kHn additionally tweaked the USB drivers himself. But the performance is better in his version rather than in yours. I've tested it by performing simple moving operations. By using the USB Flash Disk (the one which was incompatible till your recent changes) the average speed is:
    -uLE_kHn : about 130 kB/s
    -your uLE: about 35 kB/s.

    Comparing to the uLE_kHn, the average speeds of my USB HDD disk are also lower on your newest version by about 15 kB/s.
     
  5. 728
    1,302
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    728
    Likes Received:
    1,302
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    Have you tried the new USB drivers with that old version of LaunchELF?
    It is true that the driver is now slightly slower because we are more compliant with the USB mass storage standard.
    35KB/s is very slow though. What sort of files are you copying? Are there many, but small files?

    We also recently changed the way it transferred large amounts of data, for better efficiency. Maximus32 did remark that this new method of transferring data could allow slightly over 1000KB/s, unlike the old driver design from 2016. It would probably still be worse for transferring small files.
     
  6. 62
    19
    37
    Agrippa

    Agrippa Member

    Joined:
    Sep 25, 2018
    Messages:
    62
    Likes Received:
    19
    Trophy Points:
    37
    Gender:
    Male
    I'm copying two ELFs from the root of the flash disk to the folder. By forcing uLE_kHn 4.42 to use the new USBHDFSD driver (taken from the newest FMCB release) it freezes when accessing mass:/ (hangs up on empty drive contents list).

    EDIT 31.10.2018 11:22 GMT:
    I've tested the newest LaunchELF with the USBHDFSD from the FMCB 1.953 which is installed on my MC. Average speed is the same. If I try it with the uLE_kHn, every paste operation fails. It creates an empty folder instead.
     
    Last edited: Oct 31, 2018
    jolek likes this.
  7. 728
    1,302
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    728
    Likes Received:
    1,302
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    Please try this: https://www.sendspace.com/file/k66gcd
    I have fixed its buffer allocation code to always allocate 1MB, instead of only giving such a buffer for small transfers.
    I have also increased the FILEIO (fileXio here) R/W buffer size from 16KB to 128KB, to improve efficiency of the IOP.

    I believe the poor performance comes from a few issues:
    • Although it was changed to align its buffers, it will try to allocate 50000 bytes of space when a USB device is used. 50000 is neither a multiple of 512 nor 64.
    • With a 50000-byte buffer, it would eventually cause the buffer address to become unaligned, which will force FILEIO (fileXio in our case) to correct the misalignment in software.
    • With a 50000-byte buffer, it would eventually cause the buffer offset to become unaligned with the sector boundaries. So it will eventually hit a point where USBHDFSD will have to read a sector into memory and copy part of it out, in software.
    • The buffer size can be smaller or larger, depending on the device. As LaunchELF allocates a different size for different devices.
    Despite of the improvements to USBHDFSD, we're still constrained by the design of FILEIO (which fileXio copies), which will half the speed of everything. As of now, reading/writing to one device with the standard functions will block access to another.

    Perhaps in the past, it was also "better" when copying between certain devices (i.e. copying between HDD and USB disk) because LaunchELF once had a weird design trait: it would use both fileXio and FILEIO RPCs, depending on the devices involved.

    I don't know what this means. Although hundreds of commits have been made since 2010, the semantics for IOMAN have not been changed because we have to comply with what Sony laid down. So I doubt the module itself is incompatible with a software design from 2010.

    Just this year or so, following a recommendation by Ragnarok2040, I agreed that we should use the standard mode bits (FIO_S_* instead of FIO_SO_*) because the latter was actually meant for use by only the memory card filesystem. Even within the Sony SDK, the memory card module is the only one to use those older bit definitions.
    Even if the module worked with the old LaunchELF version (and it did not trigger the unpatched bug within FILEIO etc), then LaunchELF would be unable to list the contents (since it will skip over any entry that is neither a file nor directory).

    This was due to a Sony bug, within FILEIO. For a really long time, we also used to have a workaround for FILEIO calling mkdir() after remove(), within USBHDFSD. The workaround involved faking the mkdir() operation, if it was deemed that a remove() operation was just executed for the same file.

    The Sony FILEIO dread() and getstat() were found to be also bugged and unsafe to use (violates the critical section within sceSifSetDma). I made a new patch that would fix these functions, together. The workaround for remove() within USBHDFSD was removed.
     
    Last edited: Oct 31, 2018
    jolek and Agrippa like this.
  8. 62
    19
    37
    Agrippa

    Agrippa Member

    Joined:
    Sep 25, 2018
    Messages:
    62
    Likes Received:
    19
    Trophy Points:
    37
    Gender:
    Male
    The only difference is the newest build doesn't update the speed and bytes left on-the-fly. When it finishes copying one ELF it shows the average speed is 38 kB/s however. So no improvement.
     
  9. 728
    1,302
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    728
    Likes Received:
    1,302
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    The problem is with how LaunchELF is single-threaded.
    It'll probably be bad to leave users in the lurch, as it copies files to/from the memory card, so I'll probably stay with the smaller buffers that it used to have for some devices, but I'll round up the sizes to be multiples of their sector/page sizes. Thanks for sharing.

    But did it feel like it was going at 38KB/s? 38KB/s would mean that it would have needed 13s to transfer a 500KB file. Likewise, was the old LaunchELF version able to copy the 500KB file in 5s?
    If you cannot tell the difference, you might need to copy longer files.
     
    Agrippa likes this.
  10. 62
    19
    37
    Agrippa

    Agrippa Member

    Joined:
    Sep 25, 2018
    Messages:
    62
    Likes Received:
    19
    Trophy Points:
    37
    Gender:
    Male
    Yes, it did. OPL's 887 kB ELF file took the uLE_kHn 7,59 seconds to copy. Today's version and the previous one need 23,24 seconds to do that. Arithmetic is simple: 117 kB/s vs 38 kB/s.

    By the way, does your LaunchELF support creating partitions without the "+" symbol?
     
    Last edited: Oct 31, 2018
  11. 728
    1,302
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    728
    Likes Received:
    1,302
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    I see. Well, I don't see any further reasons as to why you get such results.

    Now to think about it, even if it manages 100KB/s, that is very slow. Is your disk very fragmented or has small clusters? If you have 4KB clusters (typical for disks up to less than 8GB), is there any chance that you can find a larger disk (ideally, >= 32GB) or to reformat a disk to have larger clusters?

    This restriction was lifted from LaunchELF by @akuhak in 2013.
     
  12. 68
    94
    42
    El_Patas

    El_Patas Member

    Joined:
    Aug 10, 2017
    Messages:
    68
    Likes Received:
    94
    Trophy Points:
    42
    Gender:
    Male
    Hi @sp193,

    i don't use any tool and i don't change the names in the partitions tool of the uLE.

    I mean, i simply use the "rename" option of the OPL to change the name of a game in the list of games,
    and if i write it well in spanish using one of the specific characters of the spanish language then that
    character is not shown being the name cut at that point.

    For example, if I write:

    Arc el Crepúsculo de las Almas

    It appears as:

    Arc el Crep

    Since it must not support the "ú" character.

    Best regards.
     
  13. 62
    19
    37
    Agrippa

    Agrippa Member

    Joined:
    Sep 25, 2018
    Messages:
    62
    Likes Received:
    19
    Trophy Points:
    37
    Gender:
    Male
    So what speeds do you achieve? Yes, the flash disk had 4 kB clusters before. I've changed the size to the 16 kB. It is copying much faster now with the average speed about 128 kB/s.
     
  14. 728
    1,302
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    728
    Likes Received:
    1,302
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    But OPL does not support changing the partition name. It just changes the game title.
    So even if it does not support UTF-8 correctly in titles (even though using UTF-8 in titles probably started with OPL), it would be an OPL bug.

    There is no 'ú' character in the soft keyboard, however.

    I'll have to check. I don't actually use my PS2 much due to a lack of time and reason. When I do, I'll update this post.

    But it is known that original design of USBHDFSD is capable of something like ~700KB/s while reading. Writing will be slower, due to having a need to allocate (need to update the FAT) clusters.

    When additional checks were added to make it more compliant with the USB mass storage device specification, there was additional overhead that resulted in the overall speed going down. But after we changed the way we break up USB-level transfers (to not be issued from the caller thread) and to bypass the cache to allow for longer read requests, reading went up to slightly over 1MB/s. Naturally, this also means that the smaller the clusters, the higher the overhead there will be. It is still bad for making very small reads, which can happen due to having a heavily-fragmented filesystem that has small clusters.

    If the software does not use word-aligned buffers while writing, then USBHDFSD will use a software approach to work around the hardware errata, by using scache as an intermediate buffer. The 2nd last commit to LaunchELF was meant to fix this problem with LaunchELF.

    EDIT: with my 4GB Sony USB disk, I got about 35~50KB/s when copying a file from one directory to another. With my 60GB HDD (and generic USB enclosure), I got around 180KB/s. When copying from the USB disk to my internal HDD, I got roughly 600KB/s.

    The Sony flash disk has 4KB clusters, while the 60GB disk has 32KB clusters.

    It's all so disappointing. I am sure it has got to do with how LaunchELF copies files at the moment with FILEIO. Unfortunately, cloning the new FILEIO design (which offers true multi-threaded I/O) and getting LaunchELF to be mulithreaded, is not a short task. So it will have to be a long-term thing.

    Anyway, thanks for bringing this up. Perhaps someday, LaunchELF will be able to do well at what it was meant to do.
     
    Last edited: Nov 3, 2018
    Algol, kozarovv and Agrippa like this.
  15. 1,504
    1,191
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,504
    Likes Received:
    1,191
    Trophy Points:
    347
    Gender:
    Male
    I've notice that recently wLe is loading HDD modules when they are needed (not at the beginning),
    which shortened loading time (especially on slims).
    Thanks.
     
    Algol and kozarovv like this.
  16. 184
    261
    122
    Krah

    Krah Developer

    Joined:
    Jul 20, 2018
    Messages:
    184
    Likes Received:
    261
    Trophy Points:
    122
    Was reading old development threads of opl over on PSX scene out of curiosity and saw this

    Might be helpful, might not, youre probably already doing this but when I saw it, it reminded me of your problem.
     
  17. 68
    94
    42
    El_Patas

    El_Patas Member

    Joined:
    Aug 10, 2017
    Messages:
    68
    Likes Received:
    94
    Trophy Points:
    42
    Gender:
    Male
    Umm, you are right, there is no "ú" character in the soft keyboard, sorry i remembered it badly,
    i installed an ISO now having the specific latin characters in the PC, then in the PS2 the name
    appears cutted, and i tried to fix the name using the soft keyboard of OPL and uLE but
    this characters do not exist on them.

    Well, if there is nothing to do, thank you anyway for your time and for all your hard work in PS2.

    Best regards.

    I have this file already always, it helps with the strings but not with this,
    thank you anyway also.

    Best regards.
     
  18. 107
    66
    82
    Algol

    Algol Member

    Joined:
    Sep 20, 2016
    Messages:
    107
    Likes Received:
    66
    Trophy Points:
    82
    Gender:
    Male
    Bonjour.

    La dernière version de wLE fonctionne mal sur une clef USB2 rapide. L'effacement des données, qu'il soit sur une seule à la fois ou plusieurs (multi-sélection) ne se termine jamais. La Led d'activité de la clef USB clignote mais wLE est bloqué, RESET obligatoire de la PS2 pour s'en sortir. Un contrôle du contenu de la clef USB dit qu'elle est corrompue, les données sont mal effacées.

    Un vieil uLE (le uLE_4.42e_HDD) fait parfaitement le travail et de façon très rapide. Sa vitesse d'écriture est de 200ko/s en moyenne.

    La clef utilisée est une PATRIOT XPORTER de 8Go, la PS2 est une SCPH50004.

    Très cordialement.


    In Engrish by Google :

    Hello.

    The latest version of wLE works poorly on a fast USB2 key. Erasing data, whether on one at a time or multiple (multi-selection) never ends. The activity LED of the USB key flashes but wLE is blocked, RESET mandatory of the PS2 to get out of it. A control of the contents of the USB key says that it is corrupted, the data is badly erased.

    An old uLE (the uLE_4.42e_HDD) does the job perfectly and very quickly. Its writing speed is 200ko/s on average.

    The key used is an 8GB PATRIOT XPORTER, the PS2 is a SCPH50004.

    Very cordially.
     
  19. 728
    1,302
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    728
    Likes Received:
    1,302
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    Just a few posts above (#84), we were discussing the performance limitations of the newer USBHDFSD modules.
    In summary of posts #84-#94:
    • USBHDFSD, the USB Mass Storage Device driver, was changed for better compliance with the USB Mass Storage specification. This was for better stability, so you will not experience crashes due to missing error handling. Simply going back to 2014, is likely not a good idea.
    • Software made with PS2SDK revisions from before 2014 will hence have earlier versions of USBHDFSD that are "faster".
    • As of June 2018, the performance of USBHDFSD was increased, but it only helps if longer transfers are done.
    • If your USB disk is small, the cluster size is small (i.e. for a 4GB disk, you will likely have 4KB clusters). This will make performance bad. https://support.microsoft.com/en-my/help/140365/default-cluster-size-for-ntfs-fat-and-exfat
    • Hence for better performance, the disk should have larger clusters. This comes naturally for larger disks, especially if it is 32GB or above (32KB clusters).
    • LaunchELF does not allow reading & writing to be done simultaneously. So even if you copy a file to 2 different devices, it will be slow.
     
  20. 107
    66
    82
    Algol

    Algol Member

    Joined:
    Sep 20, 2016
    Messages:
    107
    Likes Received:
    66
    Trophy Points:
    82
    Gender:
    Male
    Bonsoir.

    Mon intention n'était pas de parler de vitesse d'écriture même si à la fin de mon exposé j'en ai parlé.

    Le problème principal est qu'aucun effacement n'est possible chez moi sur cette clef et aussi sûrement sur d'autre plus ou moins grosses. wLE se plante, il est incapable d'effacer des données fichier par fichier ou plusieurs fichiers en même temps (sélections multiples). Un uLE bien plus ancien le fait très bien, aucun probème n'apparait, l'opération d'effacement se finie très bien. La copie, transfert de fichiers se passe parfaitement.

    Où est, d'où provient le problème ??? Je ne suis sûrement pas le seul qui ai se mauvais fonctionnement sur de l'USB !!!
    Quelle est la taille de "cluster" la plus efficace avec une clef de 8Go ?

    Merci par avance pour votre soutien.


    In Engrish by Google :

    Good evening.

    It was not my intention to talk about writing speed even though at the end of my talk I talked about it.

    The main problem is that no erasure is possible at home on this key and also surely on other more or less big. wLE crashes, unable to erase file data by file or multiple files at the same time (multiple selections). A much older ULE does it very well, no problem appears, the erase operation is finished very well. The copy, file transfer goes perfectly.

    Where is, where does the problem come from ??? I am surely not the only one who has malfunction on USB !!!
    What is the most efficient "cluster" size with an 8GB key?

    Thank you in advance for your support.

    PS : désolé SP193, j'espère avoir bien compris votre réponse.
     
Tags:

Share This Page