OPL with prototype game list cache

Discussion in 'Open PS2 Loader (OPL)' started by sp193, Jan 6, 2019.

  1. 709
    1,253
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    709
    Likes Received:
    1,253
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    Related Github issue: https://github.com/ifcaro/Open-PS2-Loader/issues/158

    Summary

    OPL seems to have issues with extremely long loading times at boot, when there are many games installed. This happens when:
    • There are many games on the HDD unit.
    • Or for USB devices/SMB share: there are many games in the CD and/or DVD directories, with no boot filename in their filenames.

    For the HDD unit:
    the cause is the design of APA, which is the partitioning scheme that we use on the PS2. For forward and backward-compatibility with Sony software, as well as other software.
    Due to the design of the APA driver and the IOP only having 2MB, it becomes impossible to cache the existence of all partitions on the disk and hence things slow down when there are many games in existence. Also because of the design of APA, a game can consist of multiple partitions.
    There is also no partition table in this format, meaning that all partitions have to be scanned through, which results in long loading times (probably because of seeking, since SSDs do not have this problem). It seems to still take quite a while to list the hundreds of partitions on a sizable disk (perhaps 400~800 partitions on a 2TB disk).

    For USB devices & SMB:
    USB is generally slow, but I think the actual problem might come from users who do not name their ISO images in the old format:
    Code:
    SXXX_YYY.ZZ.Game Name.iso
    
    This would cause OPL to mount the ISO and parse SYSTEM.CNF, to obtain the boot filename, which used to be specified within the filename of the file (SXXX_YYY.ZZ).
    If the performance of iterating through (without mounting) 400 game titles is actually acceptable, then it probably becomes quite easy to automatically maintain a games list: it just has to iterate through all ISO files to identify what has changed, before updating the game list.

    ***

    So now I have wrote up a prototype that will cache the game list.
    For USB & SMB shares, the list is automatically maintained by OPL and it will cache ISO files that do not have the boot filename in their names (i.e. mygametitle.iso).

    For the HDD unit, the game cache is used by default and will only be refreshed if you press the refresh button. Automatic refresh of the HDD unit is disabled because it does not make much sense; the HDD cannot be hot-plugged.

    If you have a large number of games, please consider helping us by trying this out and letting us know how the user experience is like. There is nearly no risk, given that the game list cache is a new file.

    We should aim to make the user experience comfortable. But with hardware restrictions, a line has to be drawn to make the requirements realistic.
    Given a 2TB disk, users with a "high number" of games could have about 500x 4GB titles. On the PS2 HDD unit, this could translate to something like 800 partitions, due to the design of APA (i.e. if the maximum partition size is 4GB, then a 4.1GB game would consist of 2 partitions).

    Testing on my end has been minimal so far.

    Download: https://www.sendspace.com/file/h1wjyl
    Github branch: https://github.com/sp193/Open-PS2-Loader/tree/gamelistcache
     
    Last edited: Jan 6, 2019
    Anonamous, Algol, Krah and 8 others like this.
  2. 246
    332
    122
    Tupakaveli

    Tupakaveli VIP PSX-Place Supporter

    Joined:
    Sep 19, 2017
    Messages:
    246
    Likes Received:
    332
    Trophy Points:
    122
    Gender:
    Male
    Location:
    Australia
    Wow.

    My load time with a 2Tb went from ~50 seconds to ~12 seconds.

    However when I try to run a game the console freezes on 'Saving config'.
     
    TnA and jolek like this.
  3. 1,062
    549
    222
    TnA

    TnA Senior Member

    Joined:
    Jul 1, 2018
    Messages:
    1,062
    Likes Received:
    549
    Trophy Points:
    222
    Gender:
    Male
    Location:
    Germany --> Saxony
    You are just incredible... You are delivering non-pause-action on the PS2!

    Great work!


    I will try it later with an HDD, because my SSD only needs around 3seconds, once i start the device (manually) in OPL's GUI.
     
  4. 709
    1,253
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    709
    Likes Received:
    1,253
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    Thank you. Right now, I just happened to have some time. But I'll fade away again sooner or later. :D

    Thanks, but then this won't help your case because you already got something much, much better. o_O

    I realized that there were problems with ETH and HDD shutting down DEV9, caused by their deinit() functions being always called by the UI regardless of whether these modes were used or not.
    Please use the fixed file: https://www.sendspace.com/file/pnlgxg
     
    Algol, TnA, ShaolinAssassin and 3 others like this.
  5. 1,452
    1,138
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,452
    Likes Received:
    1,138
    Trophy Points:
    347
    Gender:
    Male
    Thanks for new version.
    I've tried it unfortunately just couple times (limited time today) only through HDD.
    At first "load-out"... hmm... maybe I do not pay enough attention, but the time was the same as with OPL 1229.
    Currently when I measure time to load and shown cover art:
    OPNPS2LD-GAMELIST-2.ELF needs 9s.
    OPL 1229 needs 13s.

    I've got "only" ~140 Games on 500 GB drive, but the time to load has been reduced.
    I hope that people with 2TB disks will notice much more noticeable difference.

    For me it is a very handful feature, thanks once again.
    BTW games now should launch normally, without a problem.
     
    Algol, TnA and Tupakaveli like this.
  6. 684
    316
    72
    Peppe90

    Peppe90 Member

    Joined:
    Dec 7, 2018
    Messages:
    684
    Likes Received:
    316
    Trophy Points:
    72
    Gender:
    Male
    It's really a healthy touch for beta testing purposes :D
     
    TnA and jolek like this.
  7. 684
    316
    72
    Peppe90

    Peppe90 Member

    Joined:
    Dec 7, 2018
    Messages:
    684
    Likes Received:
    316
    Trophy Points:
    72
    Gender:
    Male
    For HDD users, where the file it is saved??
     
  8. 1,452
    1,138
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,452
    Likes Received:
    1,138
    Trophy Points:
    347
    Gender:
    Male
    For HDD, it should be stored in "hdd0:/+OPL/games.bin".
     
    Algol, TnA and Peppe90 like this.
  9. 709
    1,253
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    709
    Likes Received:
    1,253
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    Thanks all. I realized that it might not always update the game list correctly, so please use this new file: https://www.sendspace.com/file/h1wjyl

    As for where the game list cache is stored:
    • HDD: hdd:/+OPL/games.bin
    • Others: CD/games.bin & DVD/games.bin
     
    Algol, TnA, Peppe90 and 2 others like this.
  10. 1,452
    1,138
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,452
    Likes Received:
    1,138
    Trophy Points:
    347
    Gender:
    Male
    BTW does the game list "scheme" was changed during 3 rev?
    I mean, it is advised to leave it or delete old "games.bin"?
     
  11. 684
    316
    72
    Peppe90

    Peppe90 Member

    Joined:
    Dec 7, 2018
    Messages:
    684
    Likes Received:
    316
    Trophy Points:
    72
    Gender:
    Male
    It's simply another world… It takes 6 secs for me on a cold boot, then, less than 5 secs!

    And there's the surprise too!! The HDD spindown when leaving OPL, I have no words… Just thank you :D



    Yes I noticed it, here the reports:

    1. one time Half Life had some strange sobing at the starting music, then in the menu after selecting Half life mode the music loop (one second of music Always looping). Couldn't reproduce the issue though.

    2. One time after IGR from Ferrari Challenge, OPL couldn't load the list (it remains on the loading icon forever).

    Btw I tried a lot of games (among the most problematic), aside those 2 bugs all is perfect
     
    Last edited: Jan 6, 2019
    TnA likes this.
  12. 684
    316
    72
    Peppe90

    Peppe90 Member

    Joined:
    Dec 7, 2018
    Messages:
    684
    Likes Received:
    316
    Trophy Points:
    72
    Gender:
    Male
    Just refreshing would make OPL making a new one, right??
     
  13. 684
    316
    72
    Peppe90

    Peppe90 Member

    Joined:
    Dec 7, 2018
    Messages:
    684
    Likes Received:
    316
    Trophy Points:
    72
    Gender:
    Male
    About what @jolek is saying: you think it matters how the HDD modules have been loaded at first??

    I mean, if I have the HDD already turned on by another program (Ulaunchelf, FHDB, another old OPL version, etc...) it could give problems to this new OPL to correctly create the cache file (or correctly read it)??
     
    Last edited: Jan 6, 2019
  14. 1,062
    549
    222
    TnA

    TnA Senior Member

    Joined:
    Jul 1, 2018
    Messages:
    1,062
    Likes Received:
    549
    Trophy Points:
    222
    Gender:
    Male
    Location:
    Germany --> Saxony
    If you earn enough knowledge and have fun while doing it, you will become 100 years! :D

    We need an SP193-AI (or like a Star Trek: TNG 'Data'), lol! But it can't match a human being (not only you), so this solely would be a half-arsed copy...

    Anyway... A real life-Data from Doctor Soong ... erm... Yung would be insane!

    Yeah! That's true for the SSD, but I want to revert to the 1TB-HDD and solely am doing some tests with the SSD.

    I am very much interested, how fast it will scroll in the HDD-OSD and how fast it will load in OPL, once the SSD is FULL! ^^

    However... The SSD is intended to be used in one of my PCs or in my upcoming Netbook.

    THIS new feature we are talking about here, actually just contributes to that (because there is one less reason [OPL], to have a storage-device with no seek-times).

    That's what testers are there for! ;)

    Indeed! For those who have a lot of games installed AND are testing these games AND newer OPL-revisions, it is perfect!
    It will save hours of their life-time! :)

    The best thing of it all [this feature] (IMO) however is, that it also could increase the life-time of the HDD WHILE increasing performance on boot-up! :D

    I suppose the physical position of the +OPL-Partition, COULD have quite an effect and that having it at the beginning of the HDD would probably yield a smaller amount of loadup-time...

    I suppose if the +OPL-Partition is the last on the disc, that there wouldn't be any (or just a small) speed-improvement, but I am not 100% certain.

    Please check, where your +OPL-Partition is located on the disc!

    @sp193: Would you mind trying to change the path (just for 1 test) to a system-partition instead? These are (most often) in the beginning and those who have the partition quite at the end of the disc, are the perfect candidates for this kind of test! ;)

    That should be a choice IMO, because those who solely want their HDD being turned off, when it is not actually meant to be used anymore, wouldn't like their HDD spinning up and down...

    However... If it's not implemented that way in IGR but solely exit/poweroff and IGR-Poweroff the, there are no big backdrops.
     
    Last edited: Jan 6, 2019
    Algol likes this.
  15. 684
    316
    72
    Peppe90

    Peppe90 Member

    Joined:
    Dec 7, 2018
    Messages:
    684
    Likes Received:
    316
    Trophy Points:
    72
    Gender:
    Male
    Exactly. I think it was an important thing to do as soon as possible, 'cause it could speed up subsequent tests/improvements.



    I firstly made the +OPL and HDL settings partitions, just after formatting the HDD. But I think it wouldn't matter (if not for some milliseconds) where the partition is anyway.

    The HDD takes some seconds to spin-up, it seems just the normal HDD times the Ps2 need to correctly enable it. I think it couldn't be faster than that.

    Indeed when the HDD first spin up, the orange light is off. as soon as it lights up (just one quick blink) my game-list is displayed on screen :D

    It would be a custom build just for tests anyway, 'cause I (and many other people) don't have those partitions into the HDD.
    Plus one can make those partitions too subsequently having them on the middle/end of the HDD.

    I think it's better as it is at the moment. However, assuming that the file position really affect the loading speed, I think the better alternative where place the file would be on the MC (together with the other CFG files within OPL folder).
    But then you'd say: "there are FHDB users without MCs!!"

    ...Indeed the best thing is to leave the thing as it is, or make 2 versions of this OPL rev. One that save into +OPL and the other in your favorite partition :D
     
  16. 684
    316
    72
    Peppe90

    Peppe90 Member

    Joined:
    Dec 7, 2018
    Messages:
    684
    Likes Received:
    316
    Trophy Points:
    72
    Gender:
    Male
    I made a quick comparison :D

     
    TnA likes this.
  17. 1,062
    549
    222
    TnA

    TnA Senior Member

    Joined:
    Jul 1, 2018
    Messages:
    1,062
    Likes Received:
    549
    Trophy Points:
    222
    Gender:
    Male
    Location:
    Germany --> Saxony
    'As soon as possible', after ~9 years! :D

    But I agree, that this feature will be VERY useful for testers especially, but also to John Doe(s)!


    I am not so sure about it, because it would need to 'scan' through all APA-Chunks (sub-partitions or atleast the 'head'-partitions) and only would load the cache-file, when it finally found it!

    That would lead to a higher amount of seek-times and thus waiting-times.

    It wouldn't need to 'scan' for the Game-ID tho', so it would still need a bit less time...

    Indeed... On your setup (sounds familiar? :D)!

    But your partition is already in the beginning of the disc!


    Well, it is only 1-2 seconds faster on my SSD, so this feature seems to accelerate the aewk-up of the games quite fast!​

    Indeed!
    I think it is worth a test, trying it!

    What are you referring to with this?

    That could be a solution (at least for the game-cache), but would exclude FHDB and DEV.2-Users (if they don't use an MC as well)! ;)

    You got me! :D
    But I am merely stating facts!
    Fortunately you are capable of understanding and finding them on your own!

    The OPL-Partition is alright, but I really think this is worth a test!

    If it is true, we can simply suggest thise users that they should create the OPL-partition in the beginning!

    Great! That's almost as fast as my SSD! Very cool indeed!

    Now the only 'bigger' thing left to explore is, if it does load faster/slower, depending on the position the partition is located at! ;)
     
  18. 684
    316
    72
    Peppe90

    Peppe90 Member

    Joined:
    Dec 7, 2018
    Messages:
    684
    Likes Received:
    316
    Trophy Points:
    72
    Gender:
    Male
    I meant since this: http://www.ps2-home.com/forum/viewtopic.php?f=90&t=6207 ;)

    Anyway there wasn't this need before, to have very huge game-lists is a quite new trend after all.


    I'm curious to know the @sp193 opinion about it. If it really makes a noticeable difference in loading time it would be good to add an option in settings to specify where the file will be saved, i.e. these 3 options: hdd0:/+OPL, hdd0:/system-partition, mc?:/OPL.


    I've the "absolute" setup :D


    Yes, I'm developing mind-reading powers


    Well, to have the +OPL partition at the beginning would be Always the best (you have, cfg, vmc, into it, to not talking about cover-art and screenshot!).
    About it, I know people that enlarged the OPL partition when having the HDD almost full, and it don't seems to affect in a noticeable way the loading time of game CFGs and cover arts…

    It don't power off when IGR. I access Ulaunchelf from Apps in OPL and the HDD spindown.

    For me it's a very good thing. For 2 resons:

    1. If I'm leaving OPL to do something not related to the iHDD (like playing from USB with PS2ESDL, playing a DVD, etc...)

    2. Even if I'm leaving i.e. for switching to another OPL version/HDLoader (or other applications that use the iHDD) it is better to restart the HDD modules with the new program (as we know it could make many differences).

    Anyway I must thank you for this input, I'm going to try this:

    Make a IGR, go to Ule (HDD will spin down), press square (my shortcut from Ule to OPL) and I bet that it would solve the "after IGR problems" in FC and SPPS :D

    However, just an option in OPL main settings page to make the HDD spindown, would be good too.
     
  19. 246
    332
    122
    Tupakaveli

    Tupakaveli VIP PSX-Place Supporter

    Joined:
    Sep 19, 2017
    Messages:
    246
    Likes Received:
    332
    Trophy Points:
    122
    Gender:
    Male
    Location:
    Australia
    Tested OPNPS2LD-GAMELIST-3.ELF

    I deleted the previous games.bin and generated a new one.

    Everything seems good, ~12 second load time is amazing, thank you!

    This 2Tb HDD is almost full, the partitions are set up as:

    __mbr
    __net
    __system
    __sysconf
    __common
    +OPL

    Then the games. 1100 games, ~104 of which are larger than 4Gb so all up ~1210 partitions.

    I sincerely hope it's the latter :-p
     
    Last edited: Jan 6, 2019
    TnA likes this.
  20. 709
    1,253
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    709
    Likes Received:
    1,253
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    No, it was not changed. I changed the way it compares recorded game entries, as I realized that not every part of each game entry is initialized (i.e. the unused parts contained undefined content).


    Are all of you including the time taken for the HDD to spin up?

    Yes, and that was not mentioned, but it was done because I think it is necessary.

    It's to allow the PS2 to be switched off properly (cannot shut down when DEV9 is active) and also as a safety mechanism to protect some modern 2.5" HDDs (in USB enclosures or connected directly to the PS2).

    You once had a question regarding the Idle command. The Idle command was meant for power-management. We have this option to configure the Standby Timer of this function because it used to be that some people complained that the HDD would go to sleep and the game would crash.
    But IMO, it should not have been a problem in the first place.

    No, I was referring to a possibility that it might reject the game list and spend more time to generate a new one.

    But this game was known to have issues. It might have some problem related to uninitialized hardware, since letting the PS2LOGO program run first would magically solve things.
    We might have to either patch the game or to initialize the SPU2 in the way it expects, since the game would be normally booted after PS2LOGO.

    This might not be related to this feature, as I only cache the existence of games. As it is quite simple, it should either always work or not at all.
    There are too many existing problems with OPL and the PS2SDK, unfortunately.

    Yes.

    No, the capabilities of a program are determined by the program itself. Usually, we don't share modules between programs, due to possible compatibility problems.

    I doubt it'll change for the HDD OSD's case. Once you exceed the limits of the HDD.IRX cache, the cache becomes totally useless and cannot become any more useless.

    Now that you mentioned it, this might be the problem. Well, we cannot solve all problems.

    I'm sure it's a plausible reason, because of how APA (unfortunately) works. If it's the 1100th partition, the poor thing must go through 1100 partitions to find the +OPL partition (or not). Anyway, since game settings are stored in +OPL, it needs to be accessed sooner or later.

    But once you boot a game on another device, you won't go back to select another game for a while. Even Sony does this, when you boot a disc-based game from the CD/DVD drive from the HDD browser. If you're a developer, why would you have HDD set to AUTO and not boot games from it?

    This was an earlier patch, to put the HDD in STANDBY mode first, to prevent an emergency park by some HDDs when power is cut when DEV9 is shut down.



    Oh no, it won't be just a few ms, since we're trying to reduce the 40s loading time observed by some people. It's not about the position of the partition on the HDD, but whether it's behind 1000 partitions or behind just 5.




    That is great! Yes, that is how it is now supposed to be.


    Game CFGs are stored in +OPL and this will always be checked for, so it is impossible to avoid spending some time scanning for +OPL (even if it doesn't exist).



    Yes, that is true. Game title information is only 2 sectors per game and this is read after a list of games is generated. It also exists once for each title. So if you have 500 titles, then it will seek an additional 500 times. If each game took up 2 partitions (not always the case), we can say that it might represent a third of the seek time observed.

    So if @Tupakaveli always had to wait 50s, just scanning for the partitions (omiting this step) could reduce the wait by a third.. which is still quite some time. It would have been nice if this wasn't a problem though. Unless 30s is still tolerable.
    It's sad, but this is only a problem because the cache within HDD.IRX does not work. :eek:
    Otherwise, it would have helped with reducing the time taken to scan through the whole disk, at least once.

    Now to think about it, this might need some tweaking because we're back at this question: what happens when a user boots a program that uses the HDD unit?
    In the case of FHDB, I think I chose to just leave the HDD unit on.

    We can also choose to implement something like the HDDUNITPOWER setting that Sony uses, which is used to determine whether the HDD is put into IDLE mode (because only the NIC is required), the HDD & NIC are left on or if DEV9 should be shut down entirely.

    But it isn't a solution. Now that we will block further accesses to ATA before IGR, I'm not sure why you will have problems under specific conditions...

    But if we can avoid it, the less options there are, the less choices the user must make.
     
    Algol, jolek, uyjulian and 5 others like this.

Share This Page