OPL with prototype game list cache

[sp193]

Related Github issue: https://github.com/ifcaro/Open-PS2-Loader/issues/158

Summary

OPL seems to have issues with extremely long loading times at boot, when there are many games installed. This happens when:

• There are many games on the HDD unit.
• Or for USB devices/SMB share: there are many games in the CD and/or DVD directories, with no boot filename in their filenames.

For the HDD unit:
the cause is the design of APA, which is the partitioning scheme that we use on the PS2. For forward and backward-compatibility with Sony software, as well as other software.
Due to the design of the APA driver and the IOP only having 2MB, it becomes impossible to cache the existence of all partitions on the disk and hence things slow down when there are many games in existence. Also because of the design of APA, a game can consist of multiple partitions.
There is also no partition table in this format, meaning that all partitions have to be scanned through, which results in long loading times (probably because of seeking, since SSDs do not have this problem). It seems to still take quite a while to list the hundreds of partitions on a sizable disk (perhaps 400~800 partitions on a 2TB disk).

For USB devices & SMB:
USB is generally slow, but I think the actual problem might come from users who do not name their ISO images in the old format:

SXXX_YYY.ZZ.Game Name.iso

This would cause OPL to mount the ISO and parse SYSTEM.CNF, to obtain the boot filename, which used to be specified within the filename of the file (SXXX_YYY.ZZ).
If the performance of iterating through (without mounting) 400 game titles is actually acceptable, then it probably becomes quite easy to automatically maintain a games list: it just has to iterate through all ISO files to identify what has changed, before updating the game list.

***

So now I have wrote up a prototype that will cache the game list.
For USB & SMB shares, the list is automatically maintained by OPL and it will cache ISO files that do not have the boot filename in their names (i.e. mygametitle.iso).

For the HDD unit, the game cache is used by default and will only be refreshed if you press the refresh button. Automatic refresh of the HDD unit is disabled because it does not make much sense; the HDD cannot be hot-plugged.

If you have a large number of games, please consider helping us by trying this out and letting us know how the user experience is like. There is nearly no risk, given that the game list cache is a new file.

We should aim to make the user experience comfortable. But with hardware restrictions, a line has to be drawn to make the requirements realistic.
Given a 2TB disk, users with a "high number" of games could have about 500x 4GB titles. On the PS2 HDD unit, this could translate to something like 800 partitions, due to the design of APA (i.e. if the maximum partition size is 4GB, then a 4.1GB game would consist of 2 partitions).

Testing on my end has been minimal so far.

Download: https://www.sendspace.com/file/h1wjyl
Github branch: https://github.com/sp193/Open-PS2-Loader/tree/gamelistcache

 

[Tupakaveli]

Wow.

My load time with a 2Tb went from ~50 seconds to ~12 seconds.

However when I try to run a game the console freezes on 'Saving config'.

 

[TnA]

You are just incredible... You are delivering non-pause-action on the PS2!

Great work!


I will try it later with an HDD, because my SSD only needs around 3seconds, once i start the device (manually) in OPL's GUI.

 

[sp193]

You are just incredible... You are delivering non-pause-action on the PS2!

Thank you. Right now, I just happened to have some time. But I'll fade away again sooner or later.

I will try it later with an HDD, because my SSD only needs around 3seconds, once i start the device (manually) in OPL's GUI.

Thanks, but then this won't help your case because you already got something much, much better.

However when I try to run a game the console freezes on 'Saving config'.

I realized that there were problems with ETH and HDD shutting down DEV9, caused by their deinit() functions being always called by the UI regardless of whether these modes were used or not.
Please use the fixed file: https://www.sendspace.com/file/pnlgxg

 

[jolek]

Thanks for new version.
I've tried it unfortunately just couple times (limited time today) only through HDD.
At first "load-out"... hmm... maybe I do not pay enough attention, but the time was the same as with OPL 1229.
Currently when I measure time to load and shown cover art:
OPNPS2LD-GAMELIST-2.ELF needs 9s.
OPL 1229 needs 13s.

I've got "only" ~140 Games on 500 GB drive, but the time to load has been reduced.
I hope that people with 2TB disks will notice much more noticeable difference.

For me it is a very handful feature, thanks once again.
BTW games now should launch normally, without a problem.

 

[Peppe90]

Related Github issue: https://github.com/ifcaro/Open-PS2-Loader/issues/158

Summary

OPL seems to have issues with extremely long loading times at boot, when there are many games installed. This happens when:

• There are many games on the HDD unit.
• Or for USB devices/SMB share: there are many games in the CD and/or DVD directories, with no boot filename in their filenames.

For the HDD unit:
the cause is the design of APA, which is the partitioning scheme that we use on the PS2. For forward and backward-compatibility with Sony software, as well as other software.
Due to the design of the APA driver and the IOP only having 2MB, it becomes impossible to cache the existence of all partitions on the disk and hence things slow down when there are many games in existence. Also because of the design of APA, a game can consist of multiple partitions.
There is also no partition table in this format, meaning that all partitions have to be scanned through, which results in long loading times (probably because of seeking, since SSDs do not have this problem). It seems to still take quite a while to list the hundreds of partitions on a sizable disk (perhaps 400~800 partitions on a 2TB disk).

For USB devices & SMB:
USB is generally slow, but I think the actual problem might come from users who do not name their ISO images in the old format:

SXXX_YYY.ZZ.Game Name.iso

This would cause OPL to mount the ISO and parse SYSTEM.CNF, to obtain the boot filename, which used to be specified within the filename of the file (SXXX_YYY.ZZ).
If the performance of iterating through (without mounting) 400 game titles is actually acceptable, then it probably becomes quite easy to automatically maintain a games list: it just has to iterate through all ISO files to identify what has changed, before updating the game list.

***

So now I have wrote up a prototype that will cache the game list.
For USB & SMB shares, the list is automatically maintained by OPL and it will cache ISO files that do not have the boot filename in their names (i.e. mygametitle.iso).

For the HDD unit, the game cache is used by default and will only be refreshed if you press the refresh button. Automatic refresh of the HDD unit is disabled because it does not make much sense; the HDD cannot be hot-plugged.

If you have a large number of games, please consider helping us by trying this out and letting us know how the user experience is like. There is nearly no risk, given that the game list cache is a new file.

We should aim to make the user experience comfortable. But with hardware restrictions, a line has to be drawn to make the requirements realistic.
Given a 2TB disk, users with a "high number" of games could have about 500x 4GB titles. On the PS2 HDD unit, this could translate to something like 800 partitions, due to the design of APA (i.e. if the maximum partition size is 4GB, then a 4.1GB game would consist of 2 partitions).

Testing on my end has been minimal so far.

Download: https://www.sendspace.com/file/pnlgxg
Github branch: https://github.com/sp193/Open-PS2-Loader/tree/gamelistcache

It's really a healthy touch for beta testing purposes

 

[Peppe90]

For HDD users, where the file it is saved??

 

[jolek]

For HDD users, where the file it is saved??

For HDD, it should be stored in "hdd0:/+OPL/games.bin".

 

[sp193]

Thanks all. I realized that it might not always update the game list correctly, so please use this new file: https://www.sendspace.com/file/h1wjyl

As for where the game list cache is stored:

• HDD: hdd:/+OPL/games.bin
• Others: CD/games.bin & DVD/games.bin

 

[jolek]

BTW does the game list "scheme" was changed during 3 rev?
I mean, it is advised to leave it or delete old "games.bin"?

 

[Peppe90]

It's simply another world… It takes 6 secs for me on a cold boot, then, less than 5 secs!

And there's the surprise too!! The HDD spindown when leaving OPL, I have no words… Just thank you

Thanks all. I realized that it might not always update the game list correctly, so please use this new file: https://www.sendspace.com/file/h1wjyl

As for where the game list cache is stored:

HDD: hdd:/+OPL/games.bin
Others: CD/games.bin & DVD/games.bin

​

Yes I noticed it, here the reports:

1. one time Half Life had some strange sobing at the starting music, then in the menu after selecting Half life mode the music loop (one second of music Always looping). Couldn't reproduce the issue though.

2. One time after IGR from Ferrari Challenge, OPL couldn't load the list (it remains on the loading icon forever).

Btw I tried a lot of games (among the most problematic), aside those 2 bugs all is perfect

 

[Peppe90]

BTW does the game list "scheme" was changed during 3 rev?
I mean, it is advised to leave it or delete old "games.bin"?

Just refreshing would make OPL making a new one, right??

 

[Peppe90]

As for where the game list cache is stored:

• HDD: hdd:/+OPL/games.bin
• Others: CD/games.bin & DVD/games.bin

About what @jolek is saying: you think it matters how the HDD modules have been loaded at first??

I mean, if I have the HDD already turned on by another program (Ulaunchelf, FHDB, another old OPL version, etc...) it could give problems to this new OPL to correctly create the cache file (or correctly read it)??

 

[TnA]

Thank you. Right now, I just happened to have some time. But I'll fade away again sooner or later.

If you earn enough knowledge and have fun while doing it, you will become 100 years!

We need an SP193-AI (or like a Star Trek: TNG 'Data'), lol! But it can't match a human being (not only you), so this solely would be a half-arsed copy...

Anyway... A real life-Data from Doctor Soong ... erm... Yung would be insane!

Thanks, but then this won't help your case because you already got something much, much better.

Yeah! That's true for the SSD, but I want to revert to the 1TB-HDD and solely am doing some tests with the SSD.

I am very much interested, how fast it will scroll in the HDD-OSD and how fast it will load in OPL, once the SSD is FULL! ^^

However... The SSD is intended to be used in one of my PCs or in my upcoming Netbook.

THIS new feature we are talking about here, actually just contributes to that (because there is one less reason [OPL], to have a storage-device with no seek-times).

I realized that there were problems with ETH and HDD shutting down DEV9, caused by their deinit() functions being always called by the UI regardless of whether these modes were used or not.
Please use the fixed file: https://www.sendspace.com/file/pnlgxg

That's what testers are there for!

It's really a healthy touch for beta testing purposes

Indeed! For those who have a lot of games installed AND are testing these games AND newer OPL-revisions, it is perfect!
It will save hours of their life-time!

The best thing of it all [this feature] (IMO) however is, that it also could increase the life-time of the HDD WHILE increasing performance on boot-up!

It's simply another world… It takes 6 secs for me on a cold boot, then, less than 5 secs!

I suppose the physical position of the +OPL-Partition, COULD have quite an effect and that having it at the beginning of the HDD would probably yield a smaller amount of loadup-time...

I suppose if the +OPL-Partition is the last on the disc, that there wouldn't be any (or just a small) speed-improvement, but I am not 100% certain.

Please check, where your +OPL-Partition is located on the disc!

@sp193: Would you mind trying to change the path (just for 1 test) to a system-partition instead? These are (most often) in the beginning and those who have the partition quite at the end of the disc, are the perfect candidates for this kind of test!

And there's the surprise too!! The HDD spindown when leaving OPL, I have no words… Just thank you

That should be a choice IMO, because those who solely want their HDD being turned off, when it is not actually meant to be used anymore, wouldn't like their HDD spinning up and down...

However... If it's not implemented that way in IGR but solely exit/poweroff and IGR-Poweroff the, there are no big backdrops.

 

[Peppe90]

Indeed! For those who have a lot of games installed AND are testing these games AND newer OPL-revisions, it is perfect!
It will save hours of their life-time!

Exactly. I think it was an important thing to do as soon as possible, 'cause it could speed up subsequent tests/improvements.

I suppose the physical position of the +OPL-Partition, could have quite an effect and that having it at the beginning of the HDD would probably yield a smaller amount of loadup-time...

I suppose if the +OPL-Partition is the last on the disc, that there wouldn't be any (or just a small) speed-improvement, but I am not 100% certain.

Please check, where your +OPL-Partition is located on the disc!

​

I firstly made the +OPL and HDL settings partitions, just after formatting the HDD. But I think it wouldn't matter (if not for some milliseconds) where the partition is anyway.

The HDD takes some seconds to spin-up, it seems just the normal HDD times the Ps2 need to correctly enable it. I think it couldn't be faster than that.

Indeed when the HDD first spin up, the orange light is off. as soon as it lights up (just one quick blink) my game-list is displayed on screen ​

@sp193: Would you mind trying to change the path (just for 1 test) to a system-partition instead? These are (most often) in the beginning and those who have the partition quite at the end of the disc, are the perfect candidates for this kind of test!


That should be a choice IMO, because those who solely want their HDD being turned off, when it is not actually meant to be used anymore wouldn't like their HDD spinning up and down...

However... If it's not implemented that way in IGR, there are no big backdrops.

It would be a custom build just for tests anyway, 'cause I (and many other people) don't have those partitions into the HDD.
Plus one can make those partitions too subsequently having them on the middle/end of the HDD.

I think it's better as it is at the moment. However, assuming that the file position really affect the loading speed, I think the better alternative where place the file would be on the MC (together with the other CFG files within OPL folder).
But then you'd say: "there are FHDB users without MCs!!"

...Indeed the best thing is to leave the thing as it is, or make 2 versions of this OPL rev. One that save into +OPL and the other in your favorite partition

 

[Peppe90]

I made a quick comparison

 

[TnA]

Exactly. I think it was an important thing to do as soon as possible, 'cause it could speed up subsequent tests/improvements.

'As soon as possible', after ~9 years!

But I agree, that this feature will be VERY useful for testers especially, but also to John Doe(s)!

I firstly made the +OPL and HDL settings partitions, just after formatting the HDD. But I think it wouldn't matter (if not for some milliseconds) where the partition is anyway.

I am not so sure about it, because it would need to 'scan' through all APA-Chunks (sub-partitions or atleast the 'head'-partitions) and only would load the cache-file, when it finally found it!

That would lead to a higher amount of seek-times and thus waiting-times.

It wouldn't need to 'scan' for the Game-ID tho', so it would still need a bit less time...

The HDD takes some seconds to spin-up, it seems just the normal HDD times the Ps2 need to correctly enable it. I think it couldn't be faster than that.

Indeed... On your setup (sounds familiar? )!
But your partition is already in the beginning of the disc!

Indeed when the HDD first spin up, the orange light is off. as soon as it lights up (just one quick blink) my game-list is displayed on screen

Well, it is only 1-2 seconds faster on my SSD, so this feature seems to accelerate the aewk-up of the games quite fast!​

It would be a custom build just for tests anyway, 'cause I (and many other people) don't have those partitions into the HDD.
Plus one can make those partitions too subsequently having them on the middle/end of the HDD.

Indeed!
I think it is worth a test, trying it!

I think it's better as it is at the moment.

What are you referring to with this?

However, assuming that the file position really affect the loading speed, I think the better alternative where place the file would be on the MC (together with the other CFG files within OPL folder).

That could be a solution (at least for the game-cache), but would exclude FHDB and DEV.2-Users (if they don't use an MC as well)!

But then you'd say: "there are FHDB users without MCs!!"

You got me!
But I am merely stating facts!
Fortunately you are capable of understanding and finding them on your own!

...Indeed the best thing is to leave the thing as it is, or make 2 versions of this OPL rev. One that save into +OPL and the other in your favorite partition

The OPL-Partition is alright, but I really think this is worth a test!

If it is true, we can simply suggest thise users that they should create the OPL-partition in the beginning!

I made a quick comparison

Great! That's almost as fast as my SSD! Very cool indeed!

Now the only 'bigger' thing left to explore is, if it does load faster/slower, depending on the position the partition is located at!

 

[Peppe90]

'As soon as possible', after ~9 years!

I meant since this: http://www.ps2-home.com/forum/viewtopic.php?f=90&t=6207

Anyway there wasn't this need before, to have very huge game-lists is a quite new trend after all.

I am not so sure about it, because it would need to 'scan' through all APA-Chunks (sub-partitions or atleast the 'head'-partitions) and only would load the cache-file, when it finally found it!

That would lead to a higher amount of seek-times and thus waiting-times.

It wouldn't need to 'scan' for the Game-ID tho', so it would still need a bit less time...​

I'm curious to know the @sp193 opinion about it. If it really makes a noticeable difference in loading time it would be good to add an option in settings to specify where the file will be saved, i.e. these 3 options: hdd0:/+OPL, hdd0:/system-partition, mc?:/OPL.

Indeed... On your setup (sounds familiar? )!

But your partition is already in the beginning of the disc!

I've the "absolute" setup

You got me!

But I am merely stating facts!
Fortunately you are capable of understanding and finding them on your own!

Yes, I'm developing mind-reading powers

If it is true, we can simply suggest thise users that they should create the OPL-partition in the beginning!

Well, to have the +OPL partition at the beginning would be Always the best (you have, cfg, vmc, into it, to not talking about cover-art and screenshot!).
About it, I know people that enlarged the OPL partition when having the HDD almost full, and it don't seems to affect in a noticeable way the loading time of game CFGs and cover arts…

That should be a choice IMO, because those who solely want their HDD being turned off, when it is not actually meant to be used anymore, wouldn't like their HDD spinning up and down...

However... If it's not implemented that way in IGR but solely exit/poweroff and IGR-Poweroff the, there are no big backdrops.

It don't power off when IGR. I access Ulaunchelf from Apps in OPL and the HDD spindown.

For me it's a very good thing. For 2 resons:

1. If I'm leaving OPL to do something not related to the iHDD (like playing from USB with PS2ESDL, playing a DVD, etc...)

2. Even if I'm leaving i.e. for switching to another OPL version/HDLoader (or other applications that use the iHDD) it is better to restart the HDD modules with the new program (as we know it could make many differences).

Anyway I must thank you for this input, I'm going to try this:

Make a IGR, go to Ule (HDD will spin down), press square (my shortcut from Ule to OPL) and I bet that it would solve the "after IGR problems" in FC and SPPS

However, just an option in OPL main settings page to make the HDD spindown, would be good too.

 

[Tupakaveli]

Tested OPNPS2LD-GAMELIST-3.ELF

I deleted the previous games.bin and generated a new one.

Everything seems good, ~12 second load time is amazing, thank you!

This 2Tb HDD is almost full, the partitions are set up as:

__mbr
__net
__system
__sysconf
__common
+OPL

Then the games. 1100 games, ~104 of which are larger than 4Gb so all up ~1210 partitions.

Thank you. Right now, I just happened to have some time. But I'll fade away again sooner or later.

I sincerely hope it's the latter

 

[sp193]

BTW does the game list "scheme" was changed during 3 rev?
I mean, it is advised to leave it or delete old "games.bin"?

No, it was not changed. I changed the way it compares recorded game entries, as I realized that not every part of each game entry is initialized (i.e. the unused parts contained undefined content).

Everything seems good, ~12 second load time is amazing, thank you!

It's simply another world… It takes 6 secs for me on a cold boot, then, less than 5 secs!

Are all of you including the time taken for the HDD to spin up?

And there's the surprise too!! The HDD spindown when leaving OPL, I have no words… Just thank you

Yes, and that was not mentioned, but it was done because I think it is necessary.

It's to allow the PS2 to be switched off properly (cannot shut down when DEV9 is active) and also as a safety mechanism to protect some modern 2.5" HDDs (in USB enclosures or connected directly to the PS2).

You once had a question regarding the Idle command. The Idle command was meant for power-management. We have this option to configure the Standby Timer of this function because it used to be that some people complained that the HDD would go to sleep and the game would crash.
But IMO, it should not have been a problem in the first place.

​

Yes I noticed it, here the reports:

No, I was referring to a possibility that it might reject the game list and spend more time to generate a new one.

1. one time Half Life had some strange sobing at the starting music, then in the menu after selecting Half life mode the music loop (one second of music Always looping). Couldn't reproduce the issue though.

But this game was known to have issues. It might have some problem related to uninitialized hardware, since letting the PS2LOGO program run first would magically solve things.
We might have to either patch the game or to initialize the SPU2 in the way it expects, since the game would be normally booted after PS2LOGO.

2. One time after IGR from Ferrari Challenge, OPL couldn't load the list (it remains on the loading icon forever).

This might not be related to this feature, as I only cache the existence of games. As it is quite simple, it should either always work or not at all.
There are too many existing problems with OPL and the PS2SDK, unfortunately.

Just refreshing would make OPL making a new one, right??

Yes.

About what @jolek is saying: you think it matters how the HDD modules have been loaded at first??

I mean, if I have the HDD already turned on by another program (Ulaunchelf, FHDB, another old OPL version, etc...) it could give problems to this new OPL to correctly create the cache file (or correctly read it)??

No, the capabilities of a program are determined by the program itself. Usually, we don't share modules between programs, due to possible compatibility problems.

I am very much interested, how fast it will scroll in the HDD-OSD and how fast it will load in OPL, once the SSD is FULL! ^^

I doubt it'll change for the HDD OSD's case. Once you exceed the limits of the HDD.IRX cache, the cache becomes totally useless and cannot become any more useless.

I suppose the physical position of the +OPL-Partition, COULD have quite an effect and that having it at the beginning of the HDD would probably yield a smaller amount of loadup-time...

I suppose if the +OPL-Partition is the last on the disc, that there wouldn't be any (or just a small) speed-improvement, but I am not 100% certain.

Now that you mentioned it, this might be the problem. Well, we cannot solve all problems.

@sp193: Would you mind trying to change the path (just for 1 test) to a system-partition instead? These are (most often) in the beginning and those who have the partition quite at the end of the disc, are the perfect candidates for this kind of test!

I'm sure it's a plausible reason, because of how APA (unfortunately) works. If it's the 1100th partition, the poor thing must go through 1100 partitions to find the +OPL partition (or not). Anyway, since game settings are stored in +OPL, it needs to be accessed sooner or later.

That should be a choice IMO, because those who solely want their HDD being turned off, when it is not actually meant to be used anymore, wouldn't like their HDD spinning up and down...

But once you boot a game on another device, you won't go back to select another game for a while. Even Sony does this, when you boot a disc-based game from the CD/DVD drive from the HDD browser. If you're a developer, why would you have HDD set to AUTO and not boot games from it?

However... If it's not implemented that way in IGR but solely exit/poweroff and IGR-Poweroff the, there are no big backdrops.

This was an earlier patch, to put the HDD in STANDBY mode first, to prevent an emergency park by some HDDs when power is cut when DEV9 is shut down.

I firstly made the +OPL and HDL settings partitions, just after formatting the HDD. But I think it wouldn't matter (if not for some milliseconds) where the partition is anyway.

Oh no, it won't be just a few ms, since we're trying to reduce the 40s loading time observed by some people. It's not about the position of the partition on the HDD, but whether it's behind 1000 partitions or behind just 5.

​

Indeed when the HDD first spin up, the orange light is off. as soon as it lights up (just one quick blink) my game-list is displayed on screen

That is great! Yes, that is how it is now supposed to be.

I think it's better as it is at the moment. However, assuming that the file position really affect the loading speed, I think the better alternative where place the file would be on the MC (together with the other CFG files within OPL folder).
But then you'd say: "there are FHDB users without MCs!!"

Game CFGs are stored in +OPL and this will always be checked for, so it is impossible to avoid spending some time scanning for +OPL (even if it doesn't exist).

I am not so sure about it, because it would need to 'scan' through all APA-Chunks (sub-partitions or atleast the 'head'-partitions) and only would load the cache-file, when it finally found it!

That would lead to a higher amount of seek-times and thus waiting-times.

It wouldn't need to 'scan' for the Game-ID tho', so it would still need a bit less time...​

Yes, that is true. Game title information is only 2 sectors per game and this is read after a list of games is generated. It also exists once for each title. So if you have 500 titles, then it will seek an additional 500 times. If each game took up 2 partitions (not always the case), we can say that it might represent a third of the seek time observed.

So if @Tupakaveli always had to wait 50s, just scanning for the partitions (omiting this step) could reduce the wait by a third.. which is still quite some time. It would have been nice if this wasn't a problem though. Unless 30s is still tolerable.
​

If it is true, we can simply suggest thise users that they should create the OPL-partition in the beginning!

It's sad, but this is only a problem because the cache within HDD.IRX does not work.
Otherwise, it would have helped with reducing the time taken to scan through the whole disk, at least once.

It don't power off when IGR. I access Ulaunchelf from Apps in OPL and the HDD spindown.

Now to think about it, this might need some tweaking because we're back at this question: what happens when a user boots a program that uses the HDD unit?
In the case of FHDB, I think I chose to just leave the HDD unit on.

We can also choose to implement something like the HDDUNITPOWER setting that Sony uses, which is used to determine whether the HDD is put into IDLE mode (because only the NIC is required), the HDD & NIC are left on or if DEV9 should be shut down entirely.

Make a IGR, go to Ule (HDD will spin down), press square (my shortcut from Ule to OPL) and I bet that it would solve the "after IGR problems" in FC and SPPS

But it isn't a solution. Now that we will block further accesses to ATA before IGR, I'm not sure why you will have problems under specific conditions...

However, just an option in OPL main settings page to make the HDD spindown, would be good too.

But if we can avoid it, the less options there are, the less choices the user must make.