PS2 HDLGameInstaller

Discussion in 'PS2 Homebrew' started by sp193, Dec 16, 2015.

  1. 821
    1,494
    247
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    821
    Likes Received:
    1,494
    Trophy Points:
    247
    Location:
    Singapore
    Home Page:
    Originally posted on psx-scene:
    Thank you for your support!
    Even though I do hate the problems, I wish to see it completed. After all, this console has been ignored for quite a long while...

    Ah okay. Thanks for the clarification.

    No, it is fine. This is just a small thing in life, so please take your time!

    Take your time! We can also continue our discussion here.

    I'm hoping that people can enjoy it too.
     
    Last edited: May 31, 2018
  2. 1,981
    1,557
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,981
    Likes Received:
    1,557
    Trophy Points:
    347
    Gender:
    Male
    @sp193 I've tried:
    • HDLGameInstaller-180530-ctrl
    • HDLGameInstaller-180530-RxWM
    • HDLGameInstaller-180530-RxWM-100FDX-FC
    Mostly average speed was ~4700-5000 KB/s:
    [​IMG]
    I really can't tell which version was the fastest.
    All versions have some transfer drops (even to 2000 KB/s).

    I still have the problem that when I reach 99%, HDL "hangs".
    Only this time after ~30 sec, I've this message:
    [​IMG]

    Tried with PC client from v0.814B.

    EDIT:
    I've also checked [PS2] HDLGameInstaller-0814B (29/05/2018)
    with the same PC client and the game was installed successful,
    but the transfer speed was poor (~2300 KB/s).
     
    Last edited: May 30, 2018
  3. 821
    1,494
    247
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    821
    Likes Received:
    1,494
    Trophy Points:
    247
    Location:
    Singapore
    Home Page:
    HDLGameInstaller v0.814C Released

    Changelog for v0.814C:

    • Updated with latest PS2SDK fixes.
    • Adjusted font rendering to advance the cursor based on input from FreeType.
    • Organized deinitialization code for cleaner shut-downs.
    • Replaced network status screen code.
    • Added flow control status to network status screen.
    • Converted hardcoded labels on the network status screen into translatable labels.
    • Updated translation template.

    I think the program is getting very unstable now and I do not want to work on it anymore.
    For reasons, now I am getting crashes within FreeType. Either it is bugged or I have corrupted the memory it uses.

    I've gone as far as giving data transfers its own socket connection, causing absolutely zero overhead. Putting frames into LWIP on the EE-side is now zero-copy as well. But despite all that, I could only get about 6.1MB/s, despite what I could get during simple tests (~8.6MB/s).
    It may be caused by frame losses, or maybe the overhead from everything else (e.g. writing to the HDD, which also involves the SIF and DEV9 DMA channels).

    Pinging normally results in very low RTT, but specifying a longer payload (e.g. 1490, to give the maximum frame size of 1514 per ping) gives high RTT of about 8ms each. Since we no longer have one extra memcpy() step (other than the one used at the socket layer), the limitation must be with the DMA channels... unfortunately.

    But that being said, thank you all, for your support and patience.

    Downloads/links
    HDLGameInstaller project page (downloads at bottom): https://sites.google.com/view/ysai187/home/projects/hdlgameinstaller

    ***

    Really? That's... :(
    Then I know of nothing I can do about this hardware problem, sorry.
    There is just no way to even tell where the frame losses are occurring at, but I suspect it happens in the EMAC3 Rx FIFO - just that the hardware function for indicating an overrun does not work.

    By the way, I made a mistake. It was a RTL8102E that was installed in my laptop.

    If it's an occasional thing, then it has got to do with the great amounts of frame losses you are experiencing. I can work around it by adding code that will re-connect, but that is very silly because I would be trying to make TCP reliable - when TCP should be making IP & Ethernet work reliably. It's just so unreliable that even TCP cannot fix it!

    I think we're just all very unfortunate to have PCs that have Realtek adaptors - that somehow always trigger the hardware bug in the PlayStation 2 SMAP. :(
    If it doesn't even occur with all PlayStation 2 SMAP models, then we were just unfortunate to have the ones that are affected.

    But thank you for standing with me, to try to make the PlayStation 2 experience great.

    Although it's mixed with problems with the IOP being unable to accept frames quickly (due to a configuration problem), it shows that when we have very slow transfers, then the hardware bug isn't really an issue...

    As of the fixed version I sent yesterday, the code within HDLGameInstaller was exactly the same lol. Just these 2 settings were changed, within the PS2SDK!
     
    Last edited by a moderator: Apr 19, 2019
  4. 1,981
    1,557
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,981
    Likes Received:
    1,557
    Trophy Points:
    347
    Gender:
    Male
    Even so, thanks for attempts.

    I don't know if this might help, but I have RTL8111B.

    Hey, no problem.

    I think that we will have v0.814D. [​IMG]

    I'm still having a problem at:
    [​IMG]
    After taking a while (about ~30 sec) this ugly error pops out:
    [​IMG]

    I'll try to do my best, even when we gonna have to test "Z" version. [​IMG]

    Best regards.

    EDIT: I forgot to add that, new font looks cool on LCD TV:
    [​IMG]
    I hope that people with CRT TV will not complains that it's too small. ;)
     
    Last edited: May 31, 2018
  5. 821
    1,494
    247
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    821
    Likes Received:
    1,494
    Trophy Points:
    247
    Location:
    Singapore
    Home Page:
    Yeah, that's in the same family as the one in my desktop (RTL8168B). But I cannot connect it to my TOOL console because it's too far away. :X

    At least, they're both PCIe Gigabit Ethernet adaptors.

    But at this point, I've got no idea how to solve this because I did check on LWIP and it did not drop any frames.
    It has to be a hardware thing... :D


    Yeah, nothing was changed there. Sorry.
    As of now, if your network has this problem, then...

    I will wait for Maximus32 to write his findings. If nothing good can be discovered, then maybe some workable compromise can be reached...

    haha. [​IMG]

    It doesn't happen here, so I think it really depends on whether the connection gets disrupted for long enough.

    [​IMG]
    [​IMG]

    That is the MAGIC! The font is exactly the same as before, just that I let FreeType decide on the spacing.
    I did not do that earlier because I was wondering if the spacing is too small. But then I remembered that OPL was also like that, so.... nevermind! The weird spacing was also driving me [​IMG]

    Oh oh. Your SMAP negotiated to have flow control disabled too?
    I wonder if all Realtek adaptors cause this same problem lol. [​IMG]

    With flow control disabled, I have to agree with Maximus32 that it may have frame losses. At least because of the stupidly small Rx FIFO within EMAC3.

    What's your network adaptor model? Best if you can tell us the SPEED and PHY models too.
    The one that I was using in my TOOL is a SCPH-10190, with a CXD9624GG and a National Semiconductor DP83846A4 PHY.
    PS2Ident can give this information, so you do not need to open it up.
     
    Last edited: May 31, 2018
  6. 1,981
    1,557
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,981
    Likes Received:
    1,557
    Trophy Points:
    347
    Gender:
    Male
    Maybe I wrote something wrong.
    This error pops out after 99% has been reached, for ~30 sec nothing is going on.
    And out of the sudden:
    [​IMG]


    Maybe something else will drive you...
    When I want to add space (" ") in FlatOut2, to have FlatOut 2
    The marker is in strange position (way beyond wanted):
    [​IMG]

    When I enter Network status in HDL this option is Disabled.
    [​IMG]

    On PC side in my advanced Realtek settings this, option is on (flow control):
    [​IMG]
    "Sterowanie przepływem" means Flow control.
    "Włącz" means ON.

    My NA is SCPH-10350
    Speed rev: 0x0011 CXD9624GG ES
    Speed capabilities: 0003. 0002 SMAP, ATA.
    PHY vendor: National Semiconductor
    PHY model: 0x02 DP83846A
    PHY rev: 0x03

    More info has been sended in PM.
     
    Last edited: May 31, 2018
  7. 821
    1,494
    247
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    821
    Likes Received:
    1,494
    Trophy Points:
    247
    Location:
    Singapore
    Home Page:
    No, I understood what you meant.

    How the system works, is that it'll send commands across the network. For every bit of data sent, it must be acknowledged.
    So if your network is unreliable enough, then the command would be lost.

    The tool will try to reconnect (which is why there is a ~30 second pause). However, if the network is unreliable enough, then it would fail again...

    Right now, I cannot replicate this issue, but the client on my laptop is not the latest one.
    If anything changes, I will let you know again.

    EDIT: Yes, I cannot replicate it. Sorry.
    When communicating with my PC that has the RTL8168B (over all the network nodes), there are zero frame drops at about 5.5MB/s. So I am pretty convinced (or in denial) that it is a compatibility problem with Realtek PHYs.

    It's caused by the old code working with the new font-drawing ways. I haven't figured out how I would handle this.
    Previously, each character would take up the same amount of space. Now each character may take up any amount of space, but it is usually half of the old width.

    We can enable it on the PC, but to the DP83846A:
    Code:
    dsidb S> ow 0xB000205C 0x10250000
    dsidb S> iw 0xB000205C      
     0xb000205c: 0x802541e1
    
    This was an access to the PHY Automatic-Negotiation Link Partner Ability Register (ANLPAR). The PAUSE bit is not set, so the Realtek RTL8102E is not advertising support for the flow control...
    So it cannot be enabled because the Realtek does not advertise support for it.

    It is a good question as to whether this is a compatibility problem with the DP83846A or if flow control never worked! lol

    It's worth noting that the Sony driver does check this register for the PAUSE bit (bit 10), while the datasheet for the DP83846A indicates that it is a reserved bit. So either the DP83846A we have for the PS2 is a custom device or if it is an undocumented feature.

    Thank you. I've also received your PM.
    So your network adaptor has the same parts as mine. What a coincidence!

    EDIT: this also happens with my SCPH-10281, which has the CXD9731GP and the STE100S PHY. Must be a thing with Realteks...

    I think the R/W buffer length may be too long, allowing the HDD to spend a long time writing (maybe even attributing to any overrun conditions), so I tried to lower it until I found some zen point: out of 4 buffers, 2 will be available. I'm getting 6.4MB/s, slightly higher than the usual 6.2MB/s.
    Maybe the speed is more regular for me, even though my Realtek causes flow control to be disabled. If you can, please let us know if this actually helps with : https://www.sendspace.com/file/5tiagr
     
    Last edited: Jun 5, 2018
  8. 1,981
    1,557
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,981
    Likes Received:
    1,557
    Trophy Points:
    347
    Gender:
    Male
    @sp193,
    I still have a problem with HDLGameInstaller-180601-48K.elf when I reach 99%:
    [​IMG]

    But look at the speed:
    [​IMG]
    [​IMG]

    Transfer drop sometimes to ~2400 KB/s, but for most of the time it stays at 4500-6700 KB/s,
    PS2HDD OLED is for most of the time light up solid (sometimes for 1-2 sec OLED stops shines).
    Probably when the transfer speed drops.
    Progress has been made!

    BTW I also tried to switch off flow control in Win, but the results where the same.

    [​IMG]

    EDIT:
    I also tried HDLGameInstaller-180601-48K-32x256.elf.
    OMG! It was very fast commit:
    [​IMG]
    [​IMG]

    Sometimes speed drops, but overall THIS version is the fastest that I tried.

    Even so, when I reach 99%... you probably know what's gonna happen. [​IMG]
     
    Last edited: Jun 1, 2018
  9. 821
    1,494
    247
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    821
    Likes Received:
    1,494
    Trophy Points:
    247
    Location:
    Singapore
    Home Page:
    That's very great news! Thanks!

    Special thanks for the adjustment, goes to wisi. I wouldn't have thought of trying this.

    Have you been trying to install the game, over and over again? Or have you been installing different games, but they all end with the same problem at 99%?
    If you've been installing the same game over and over again, what game is this? It would be good if you can mention the game ID as well.
     
  10. 1,981
    1,557
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,981
    Likes Received:
    1,557
    Trophy Points:
    347
    Gender:
    Male
    I've been installing different games, but they all end with the same problem at 99%.
    The last version that don't have problems with it was HDLGameInstaller-0814B (29/05/2018)
    with the same PC client (from 29/05/2018), but the transfer speed was poor (~2300 KB/s).

    No, I haven't tried to install the same game over and over again.
    I've been trying to install mostly CD games, because transferring them to PS2 takes less time (obviously).
    • SLES_501.26.Quake III - Revolution
    • SLES_505.04.Half-Life
    • SLES_520.95.Gradius V
    • SLES_535.08.Ultimate Pro-Pinball

    I've tried also DVD games:
    • SLES_518.24.Colin McRae Rally 04
    • SLES_528.59.Project Snowblind

      These games works through SMB, I've also checked MD5 and it matches the ones from redump.
     
    sp193 likes this.
  11. 821
    1,494
    247
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    821
    Likes Received:
    1,494
    Trophy Points:
    247
    Location:
    Singapore
    Home Page:
    Changelog for v0.815
    • Updated font-rendering for the soft keyboard and various menus, to work properly with the variable-width fonts.
    • Updated network protocol for faster game-list transmission.
    • Changed I/O strategy: 64KB R/W buffer, with 32x512KB ring buffer.
    • Added support for DHCP.
    • Added support for title-sorting (controlled from the PlayStation 2).
    • Added options screen.
    • Network settings are now stored in the HDLGameInstaller save, on the HDD unit.
    • Added IP address indicator to main menu.
    • Added free disk space indicator to the main menu.
    • Network Settings can now be changed from the options screen.
    • Changed internal game list format to store titles in UTF-8 instead, for memory savings.
    • Fixed booting from the HDD unit.
    • Added key-repetition functionality to the soft keyboard and menus.

    Changelog for the PC client
    • Changed I/O strategy to match the PlayStation 2 side: 32x512KB ring buffer.
    • Adjusted control sizes and placement on install dialog box. DVD is now the default disc type.
    • Adjusted network protocol for quicker transfer of game list.
    • Removed old TCP_NODELAY setting for better utilization of the command socket.
    • Fixed division by zero glitch during installation and game copying.

    Settings are now saved into the HDLGameInstaller save data folder:
    Code:
    hdd0:__common/Your Saves/HDLGAMEINSTALLER
    
    If you wish to delete the save, you may do so with the HDD Browser or compatible homebrew (e.g. LaunchELF).

    Downloads/Links
    HDLGameInstaller project page (downloads at bottom): https://sites.google.com/view/ysai187/home/projects/hdlgameinstaller

    ***

    So as to follow up, I think it might have been due to a lack of flow control. But Maximus32 doesn't seem to think so.
    For now, I have no further advice for you, other than trying to get flow control working, putting a switch/router between the PS2 and PC, or even using another PC. I don't actually know what's even happening with the SMAP... :'(

    Thank you for your help and support during this period of time.
     
    Last edited by a moderator: Apr 19, 2019
    jolek and bguerville like this.
  12. 1,981
    1,557
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,981
    Likes Received:
    1,557
    Trophy Points:
    347
    Gender:
    Male
    @sp193 thanks for new version, unfortunately now I've another problem.
    When I launch "HDLGameInstaller.elf" from latest "[180604]HDLGameInstaller-0815-bin",
    I can't pass this screen:
    [​IMG]
    After ~30-40 sec Loading stops.

    I've tried different flash drives.
    I even formatted my PS2HDD.
    None of these steeps helps.
     
  13. 821
    1,494
    247
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    821
    Likes Received:
    1,494
    Trophy Points:
    247
    Location:
    Singapore
    Home Page:
    Have you tried waiting for about 1 minute? Now that it has support for DHCP, it will default to and try to get an IP address on its own - unless it successfully found your old IP address settings. In that case, then it would self-configure for static IP address use.

    But if it defaults to using DHCP, then it will not get a response when connected directly to your PC. The only way out, is to wait for it to time-out.
    Once you get to the main menu, change the IP address setting to static and save (select the OK button).

    I had it hard-freeze on me once, but only once though.
     
  14. 1,981
    1,557
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,981
    Likes Received:
    1,557
    Trophy Points:
    347
    Gender:
    Male
    I've waited even longer (~5 min).
    I think that app hangs always in the same point, when Loading have two dots (Loading..).
    Even picture in previous post show it.
    In Windows, I've configured my network to have static IP with PS2.

    I've even tried to load HDL without ETH cable plug into NA,
    unfortunately this method also fails.
     
    sp193 likes this.
  15. 821
    1,494
    247
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    821
    Likes Received:
    1,494
    Trophy Points:
    247
    Location:
    Singapore
    Home Page:
    HDLGameInstaller v0.816 released!

    Changelog for v0.816:

    • Improved rendering of button labels between half-width and full-width characters.
    • Fixed freeze when network connection is unavailable.
    • Updated PFS to correct creation of "Your Saves" folder.
    • Fixed save-file generation.
    • Settings will be automatically saved once the defaults are loaded, if the user never saved the settings before.
    • Improved network performance.
    Downloads/Links
    HDLGameInstaller project page (downloads at bottom): https://sites.google.com/view/ysai187/home/projects/hdlgameinstaller
     
    Last edited by a moderator: Apr 19, 2019
  16. 1,981
    1,557
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,981
    Likes Received:
    1,557
    Trophy Points:
    347
    Gender:
    Male
    1. With new version I can pass this screen (Connecting to network. Please wait.):
    [​IMG]
    I'm getting error that (Error: there is no network connection)
    [​IMG]

    But no matter if the static IP is set (Use DHCP - Disabled):
    [​IMG]

    I will still get the same error, even when I launch HDL one more time:
    [​IMG]
    [​IMG]

    Even with "no network connection", I can establish connection
    using PC Client ("HDLGManClient.exe").

    My transfer was:
    [​IMG]
    Fastest speed
    [​IMG]
    Slowest speed

    So I'm not complaining. ;)

    2. You will not believe, but... [​IMG]
    When I reach 99%, I'm getting:
    [​IMG]
     
  17. 821
    1,494
    247
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    821
    Likes Received:
    1,494
    Trophy Points:
    247
    Location:
    Singapore
    Home Page:
    I have replaced the PS2 server. Please use the new file instead.

    7.5MB/s is amazing, really! Thanks.
    Now if only Realtek will fix the flow-control problem for us.

    This (including the slow speeds) can only be fixed by Realtek, unfortunately. Since we can both experience the bug, it has to be an actual bug. [​IMG]

    My RTL8102E will not advertise support for flow control, even when connected to my desktop PC, which has a RTL8168B. The RTL8168B has no problem with flow control.
    My newer laptop has a RTL8168GU. It has no issues with flow control either.

    I am still waiting for Realtek to get back to me, since our last message. Hopefully, they will be able to replicate the glitch and take action. That would be, IMO, the best-possible ending.
     
    jolek likes this.
  18. 1,981
    1,557
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,981
    Likes Received:
    1,557
    Trophy Points:
    347
    Gender:
    Male
    Where can I found it?

    I hope they will get in touch with you soon.
    Thanks, best regards.
     
  19. 821
    1,494
    247
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    821
    Likes Received:
    1,494
    Trophy Points:
    247
    Location:
    Singapore
    Home Page:
    Download from the same link again. The filename is exactly the same, but it'll work.
    The only difference, is a small correction that affects that error message, which was a cosmetic problem (there is a network connection).
     
  20. 1,981
    1,557
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,981
    Likes Received:
    1,557
    Trophy Points:
    347
    Gender:
    Male
    Okay, I was asking because my browser highlight this link from HDLGameInstaller project page:
    [​IMG]
    like I already use it.

    Normally unused links in my browser have this color:
    [​IMG]

    EDIT: MD5Checksum is different.
     
    Last edited: Jun 9, 2018

Share This Page