PS2 wLaunchELF Release Thread

[El_Patas]

Hi @sp193,

thank you very much for the relaxation on the Unicode, i tested it and the characters works fine.
I hope you can find a good way for add complete UTF-8 support for the USBHDFSD, i appreciate
your work.

And the internal HDD do not support complete UTF-8? when i wrote the name of a game with
a special spanish character, the character do not appears in the games list of the OPL,
and the name appears cutted.
Or is the OPL which do not have support for UTF-8?
Would there be some way to fix this?
Well sorry, maybe this will go in another thread.

Best regards.

 

[jolek]

If there are no other problems, we could probably crown this as the next stable release: https://www.sendspace.com/file/r2168y

LaunchELF commit 9db3bcc, build 2018/10/27

Changelog:

• Updated USBHDFSD: fix bug that prevented correct reading & writing. Also relaxed restriction on Unicode codepoints to support glyphs up to index 255.

Downloads/Links
LaunchELF: https://www.sendspace.com/file/wcqn8o

Are there any differences between "BOOT-181028.ELF", "BOOT-181021.ELF"?
Because when I go into page "About uLE", both versions have the same commit 9db3bcc from Oct 21.

 

[sp193]

Yes, USBHDFSD changed.
This bug, although it is visible through your activities within LaunchELF, is not actually a LaunchELF bug. So when we change USBHDFSD by changing the PS2SDK, LaunchELF receives no new commits and so its last commit hash remains the same.

internal HDD do not support complete UTF-8? when i wrote the name of a game with
a special spanish character, the character do not appears in the games list of the OPL,
and the name appears cutted.
Or is the OPL which do not have support for UTF-8?
Would there be some way to fix this?
Well sorry, maybe this will go in another thread.

I am not sure how you did it, but APA is officially documented to only accept some ASCII characters for partition names. So don't enter foreign characters in, even if the tool you use allows it. HDLGameInstaller will replace all unsupported characters within the partition name with underscores (_). The maximum length is 32.

However, the HDLoader game title we have is stored in UTF-8. OPL supports this. And because UTF-8 is backward-compatible with ASCII strings, it will still work with HDLoader and LaunchELF. So please change the title instead of the partition name.

In comparison to FAT, the FAT LFN was documented to have its filenames stored in the "OEM" encoding. Although this meant that the encoding was not specified, Windows was the primary user of it, so the encoding used by Windows should be what we use for compatibility. Which means UCS-2, which can represent Unicode.

 

[Agrippa]

LaunchELF commit 9db3bcc, build 2018/10/27

Changelog:

• Updated USBHDFSD: fix bug that prevented correct reading & writing. Also relaxed restriction on Unicode codepoints to support glyphs up to index 255.

Downloads/Links
LaunchELF: https://www.sendspace.com/file/wcqn8o

This may be a bit too early to say, but I have no time to actually spend here these two weeks. However, due to the severity of this sort of bug, I have made new releases of LaunchELF, FMCB, PS2Ident and HDDChecker.
If @akuhak's bot will build with the patched SDK, then it is fine. But in case it does not, here is a good copy.

Well, I'm hoping the new commit fixes the last of the problems from June 8th. If not, then I will be feeling sad. But if I knew the reason why it has been so defective, then it would have been fixed long ago.

I've downloaded this release to test the USB changes to make sure everything works correctly. And yes, it does, but I'm generally using the uLE_kHn_20160723 version for POPS's VCD support. I don't remember if kHn additionally tweaked the USB drivers himself. But the performance is better in his version rather than in yours. I've tested it by performing simple moving operations. By using the USB Flash Disk (the one which was incompatible till your recent changes) the average speed is:
-uLE_kHn : about 130 kB/s
-your uLE: about 35 kB/s.

Comparing to the uLE_kHn, the average speeds of my USB HDD disk are also lower on your newest version by about 15 kB/s.

 

[sp193]

Have you tried the new USB drivers with that old version of LaunchELF?
It is true that the driver is now slightly slower because we are more compliant with the USB mass storage standard.
35KB/s is very slow though. What sort of files are you copying? Are there many, but small files?

We also recently changed the way it transferred large amounts of data, for better efficiency. Maximus32 did remark that this new method of transferring data could allow slightly over 1000KB/s, unlike the old driver design from 2016. It would probably still be worse for transferring small files.

 

[Agrippa]

I'm copying two ELFs from the root of the flash disk to the folder. By forcing uLE_kHn 4.42 to use the new USBHDFSD driver (taken from the newest FMCB release) it freezes when accessing mass:/ (hangs up on empty drive contents list).

EDIT 31.10.2018 11:22 GMT:
I've tested the newest LaunchELF with the USBHDFSD from the FMCB 1.953 which is installed on my MC. Average speed is the same. If I try it with the uLE_kHn, every paste operation fails. It creates an empty folder instead.

 

[sp193]

Please try this: https://www.sendspace.com/file/k66gcd
I have fixed its buffer allocation code to always allocate 1MB, instead of only giving such a buffer for small transfers.
I have also increased the FILEIO (fileXio here) R/W buffer size from 16KB to 128KB, to improve efficiency of the IOP.

I believe the poor performance comes from a few issues:

• Although it was changed to align its buffers, it will try to allocate 50000 bytes of space when a USB device is used. 50000 is neither a multiple of 512 nor 64.
• With a 50000-byte buffer, it would eventually cause the buffer address to become unaligned, which will force FILEIO (fileXio in our case) to correct the misalignment in software.
• With a 50000-byte buffer, it would eventually cause the buffer offset to become unaligned with the sector boundaries. So it will eventually hit a point where USBHDFSD will have to read a sector into memory and copy part of it out, in software.
• The buffer size can be smaller or larger, depending on the device. As LaunchELF allocates a different size for different devices.

Despite of the improvements to USBHDFSD, we're still constrained by the design of FILEIO (which fileXio copies), which will half the speed of everything. As of now, reading/writing to one device with the standard functions will block access to another.

Perhaps in the past, it was also "better" when copying between certain devices (i.e. copying between HDD and USB disk) because LaunchELF once had a weird design trait: it would use both fileXio and FILEIO RPCs, depending on the devices involved.

I'm copying two ELFs from the root of the flash disk to the folder. By forcing uLE_kHn 4.42 to use the new USBHDFSD driver (taken from the newest FMCB release) it freezes when accessing mass:/ (hangs up on empty drive contents list).

I don't know what this means. Although hundreds of commits have been made since 2010, the semantics for IOMAN have not been changed because we have to comply with what Sony laid down. So I doubt the module itself is incompatible with a software design from 2010.

Just this year or so, following a recommendation by Ragnarok2040, I agreed that we should use the standard mode bits (FIO_S_* instead of FIO_SO_*) because the latter was actually meant for use by only the memory card filesystem. Even within the Sony SDK, the memory card module is the only one to use those older bit definitions.
Even if the module worked with the old LaunchELF version (and it did not trigger the unpatched bug within FILEIO etc), then LaunchELF would be unable to list the contents (since it will skip over any entry that is neither a file nor directory).

very paste operation fails. It creates an empty folder instead.

This was due to a Sony bug, within FILEIO. For a really long time, we also used to have a workaround for FILEIO calling mkdir() after remove(), within USBHDFSD. The workaround involved faking the mkdir() operation, if it was deemed that a remove() operation was just executed for the same file.

The Sony FILEIO dread() and getstat() were found to be also bugged and unsafe to use (violates the critical section within sceSifSetDma). I made a new patch that would fix these functions, together. The workaround for remove() within USBHDFSD was removed.

 

[Agrippa]

The only difference is the newest build doesn't update the speed and bytes left on-the-fly. When it finishes copying one ELF it shows the average speed is 38 kB/s however. So no improvement.

 

[sp193]

The problem is with how LaunchELF is single-threaded.
It'll probably be bad to leave users in the lurch, as it copies files to/from the memory card, so I'll probably stay with the smaller buffers that it used to have for some devices, but I'll round up the sizes to be multiples of their sector/page sizes. Thanks for sharing.

But did it feel like it was going at 38KB/s? 38KB/s would mean that it would have needed 13s to transfer a 500KB file. Likewise, was the old LaunchELF version able to copy the 500KB file in 5s?
If you cannot tell the difference, you might need to copy longer files.

 

[Agrippa]

Yes, it did. OPL's 887 kB ELF file took the uLE_kHn 7,59 seconds to copy. Today's version and the previous one need 23,24 seconds to do that. Arithmetic is simple: 117 kB/s vs 38 kB/s.

By the way, does your LaunchELF support creating partitions without the "+" symbol?

 

[sp193]

Yes, it did. OPL's 887 kB ELF file took the uLE_kHn 7,59 seconds to copy. Today's version and the previous one need 23,24 seconds to do that. Arithmetic is simple: 117 kB/s vs 38 kB/s.

I see. Well, I don't see any further reasons as to why you get such results.

Now to think about it, even if it manages 100KB/s, that is very slow. Is your disk very fragmented or has small clusters? If you have 4KB clusters (typical for disks up to less than 8GB), is there any chance that you can find a larger disk (ideally, >= 32GB) or to reformat a disk to have larger clusters?

By the way, does your LaunchELF support creating partitions without the "+" symbol?

This restriction was lifted from LaunchELF by @akuhak in 2013.

 

[El_Patas]

I am not sure how you did it, but APA is officially documented to only accept some ASCII characters for partition names. So don't enter foreign characters in, even if the tool you use allows it. HDLGameInstaller will replace all unsupported characters within the partition name with underscores (_). The maximum length is 32.

However, the HDLoader game title we have is stored in UTF-8. OPL supports this. And because UTF-8 is backward-compatible with ASCII strings, it will still work with HDLoader and LaunchELF. So please change the title instead of the partition name.

In comparison to FAT, the FAT LFN was documented to have its filenames stored in the "OEM" encoding. Although this meant that the encoding was not specified, Windows was the primary user of it, so the encoding used by Windows should be what we use for compatibility. Which means UCS-2, which can represent Unicode.

Hi @sp193,

i don't use any tool and i don't change the names in the partitions tool of the uLE.

I mean, i simply use the "rename" option of the OPL to change the name of a game in the list of games,
and if i write it well in spanish using one of the specific characters of the spanish language then that
character is not shown being the name cut at that point.

For example, if I write:

Arc el Crepúsculo de las Almas

It appears as:

Arc el Crep

Since it must not support the "ú" character.

Best regards.

 

[Agrippa]

I see. Well, I don't see any further reasons as to why you get such results.

Now to think about it, even if it manages 100KB/s, that is very slow. Is your disk very fragmented or has small clusters? If you have 4KB clusters (typical for disks up to less than 8GB), is there any chance that you can find a larger disk (ideally, >= 32GB) or to reformat a disk to have larger clusters?

So what speeds do you achieve? Yes, the flash disk had 4 kB clusters before. I've changed the size to the 16 kB. It is copying much faster now with the average speed about 128 kB/s.

 

[sp193]

I mean, i simply use the "rename" option of the OPL to change the name of a game in the list of games,
and if i write it well in spanish using one of the specific characters of the spanish language then that
character is not shown being the name cut at that point.

But OPL does not support changing the partition name. It just changes the game title.
So even if it does not support UTF-8 correctly in titles (even though using UTF-8 in titles probably started with OPL), it would be an OPL bug.

There is no 'ú' character in the soft keyboard, however.

So what speeds do you achieve? Yes, the flash disk had 4 kB clusters before. I've changed the size to the 16 kB. It is copying much faster now with the average speed about 128 kB/s.

I'll have to check. I don't actually use my PS2 much due to a lack of time and reason. When I do, I'll update this post.

But it is known that original design of USBHDFSD is capable of something like ~700KB/s while reading. Writing will be slower, due to having a need to allocate (need to update the FAT) clusters.

When additional checks were added to make it more compliant with the USB mass storage device specification, there was additional overhead that resulted in the overall speed going down. But after we changed the way we break up USB-level transfers (to not be issued from the caller thread) and to bypass the cache to allow for longer read requests, reading went up to slightly over 1MB/s. Naturally, this also means that the smaller the clusters, the higher the overhead there will be. It is still bad for making very small reads, which can happen due to having a heavily-fragmented filesystem that has small clusters.

If the software does not use word-aligned buffers while writing, then USBHDFSD will use a software approach to work around the hardware errata, by using scache as an intermediate buffer. The 2nd last commit to LaunchELF was meant to fix this problem with LaunchELF.

EDIT: with my 4GB Sony USB disk, I got about 35~50KB/s when copying a file from one directory to another. With my 60GB HDD (and generic USB enclosure), I got around 180KB/s. When copying from the USB disk to my internal HDD, I got roughly 600KB/s.

The Sony flash disk has 4KB clusters, while the 60GB disk has 32KB clusters.

It's all so disappointing. I am sure it has got to do with how LaunchELF copies files at the moment with FILEIO. Unfortunately, cloning the new FILEIO design (which offers true multi-threaded I/O) and getting LaunchELF to be mulithreaded, is not a short task. So it will have to be a long-term thing.

Anyway, thanks for bringing this up. Perhaps someday, LaunchELF will be able to do well at what it was meant to do.

 

[jolek]

I've notice that recently wLe is loading HDD modules when they are needed (not at the beginning),
which shortened loading time (especially on slims).
Thanks.

 

[Krah]

For example, if I write:

Arc el Crepúsculo de las Almas

It appears as:

Arc el Crep

Since it must not support the "ú" character.

Was reading old development threads of opl over on PSX scene out of curiosity and saw this

To use specific characters you need an "enhanced" font file. The embedded one into OPL was trimed down to basic "latin" charset to reduce its size.

Copy the font_Spanish.ttf from the thirdparty folder to your mc?:OPL/ folder.

Might be helpful, might not, youre probably already doing this but when I saw it, it reminded me of your problem.

 

[El_Patas]

But OPL does not support changing the partition name. It just changes the game title.
So even if it does not support UTF-8 correctly in titles (even though using UTF-8 in titles probably started with OPL), it would be an OPL bug.

There is no 'ú' character in the soft keyboard, however.

Umm, you are right, there is no "ú" character in the soft keyboard, sorry i remembered it badly,
i installed an ISO now having the specific latin characters in the PC, then in the PS2 the name
appears cutted, and i tried to fix the name using the soft keyboard of OPL and uLE but
this characters do not exist on them.

Well, if there is nothing to do, thank you anyway for your time and for all your hard work in PS2.

Best regards.

Was reading old development threads of opl over on PSX scene out of curiosity and saw this

Might be helpful, might not, youre probably already doing this but when I saw it, it reminded me of your problem.

I have this file already always, it helps with the strings but not with this,
thank you anyway also.

Best regards.

 

[Algol]

Bonjour.

La dernière version de wLE fonctionne mal sur une clef USB2 rapide. L'effacement des données, qu'il soit sur une seule à la fois ou plusieurs (multi-sélection) ne se termine jamais. La Led d'activité de la clef USB clignote mais wLE est bloqué, RESET obligatoire de la PS2 pour s'en sortir. Un contrôle du contenu de la clef USB dit qu'elle est corrompue, les données sont mal effacées.

Un vieil uLE (le uLE_4.42e_HDD) fait parfaitement le travail et de façon très rapide. Sa vitesse d'écriture est de 200ko/s en moyenne.

La clef utilisée est une PATRIOT XPORTER de 8Go, la PS2 est une SCPH50004.

Très cordialement.


In Engrish by Google :

Hello.

The latest version of wLE works poorly on a fast USB2 key. Erasing data, whether on one at a time or multiple (multi-selection) never ends. The activity LED of the USB key flashes but wLE is blocked, RESET mandatory of the PS2 to get out of it. A control of the contents of the USB key says that it is corrupted, the data is badly erased.

An old uLE (the uLE_4.42e_HDD) does the job perfectly and very quickly. Its writing speed is 200ko/s on average.

The key used is an 8GB PATRIOT XPORTER, the PS2 is a SCPH50004.

Very cordially.

 

[sp193]

Hello.

The latest version of wLE works poorly on a fast USB2 key. Erasing data, whether on one at a time or multiple (multi-selection) never ends. The activity LED of the USB key flashes but wLE is blocked, RESET mandatory of the PS2 to get out of it. A control of the contents of the USB key says that it is corrupted, the data is badly erased.

An old uLE (the uLE_4.42e_HDD) does the job perfectly and very quickly. Its writing speed is 200ko/s on average.

The key used is an 8GB PATRIOT XPORTER, the PS2 is a SCPH50004.

Very cordially.

Just a few posts above (#84), we were discussing the performance limitations of the newer USBHDFSD modules.
In summary of posts #84-#94:

• USBHDFSD, the USB Mass Storage Device driver, was changed for better compliance with the USB Mass Storage specification. This was for better stability, so you will not experience crashes due to missing error handling. Simply going back to 2014, is likely not a good idea.
  
• Software made with PS2SDK revisions from before 2014 will hence have earlier versions of USBHDFSD that are "faster".
• As of June 2018, the performance of USBHDFSD was increased, but it only helps if longer transfers are done.
• If your USB disk is small, the cluster size is small (i.e. for a 4GB disk, you will likely have 4KB clusters). This will make performance bad. https://support.microsoft.com/en-my/help/140365/default-cluster-size-for-ntfs-fat-and-exfat
• Hence for better performance, the disk should have larger clusters. This comes naturally for larger disks, especially if it is 32GB or above (32KB clusters).
• LaunchELF does not allow reading & writing to be done simultaneously. So even if you copy a file to 2 different devices, it will be slow.

 

[Algol]

Bonsoir.

Mon intention n'était pas de parler de vitesse d'écriture même si à la fin de mon exposé j'en ai parlé.

Le problème principal est qu'aucun effacement n'est possible chez moi sur cette clef et aussi sûrement sur d'autre plus ou moins grosses. wLE se plante, il est incapable d'effacer des données fichier par fichier ou plusieurs fichiers en même temps (sélections multiples). Un uLE bien plus ancien le fait très bien, aucun probème n'apparait, l'opération d'effacement se finie très bien. La copie, transfert de fichiers se passe parfaitement.

Où est, d'où provient le problème ??? Je ne suis sûrement pas le seul qui ai se mauvais fonctionnement sur de l'USB !!!
Quelle est la taille de "cluster" la plus efficace avec une clef de 8Go ?

Merci par avance pour votre soutien.


In Engrish by Google :

Good evening.

It was not my intention to talk about writing speed even though at the end of my talk I talked about it.

The main problem is that no erasure is possible at home on this key and also surely on other more or less big. wLE crashes, unable to erase file data by file or multiple files at the same time (multiple selections). A much older ULE does it very well, no problem appears, the erase operation is finished very well. The copy, file transfer goes perfectly.

Where is, where does the problem come from ??? I am surely not the only one who has malfunction on USB !!!
What is the most efficient "cluster" size with an 8GB key?

Thank you in advance for your support.

PS : désolé SP193, j'espère avoir bien compris votre réponse.