PS3 Multithreaded hashing

Discussion in 'Homebrew Development' started by sandungas, Nov 25, 2018.

  1. 2,904
    2,428
    372
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    2,904
    Likes Received:
    2,428
    Trophy Points:
    372
    Location:
    Babylon 20xxE series
    /!\ WARNING /!\
    This thread has been created by moving some post that was written originally here (post 38)
    It started as a talk about IRD verification checks... later IRD format... and later high performance MD5 checks by using multiple SPUs of the CELL processor
     
    Last edited: Nov 28, 2018
  2. 2,904
    2,428
    372
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    2,904
    Likes Received:
    2,428
    Trophy Points:
    372
    Location:
    Babylon 20xxE series
    Im guessing if made in PS3 the IRD files generated should be very relliable :)
    Not sure how hard could be to achieve this, but is "only" about dealing with keys + generating the MD5 of all files

    That MD5 of all files could take lot of time in PS3 though, in managunz checking the MD5 of a PS2 ISO takes around 5 minutes... so is like 1gb per minute
    But this was only because an ISO is a single file, if you do the same with a game like arcania the complete tale (100.000+ files) it could take all the evening

    So generating an IRD in PS3 could be annoying, but imo is afordable, and could be a very nice feature :)


    *Edit:
    Managunz have functions related with IRD, it can check the IRD of a PS3 game (when you use the feature it downloads the IRD from the online IRD databases)... but managunz cant create IRD files
     
    Last edited: Nov 26, 2018
  3. 7,571
    6,184
    622
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    7,571
    Likes Received:
    6,184
    Trophy Points:
    622
    Location:
    Earth
    That's what I meant.
    The homebrew code to write should be a piece of cake really, there are plenty of ways to calculate hashes on the PS3, from the hash library in the sdk to existing exports & even by hijacking the kernel functions using Cobra sc15. We might even be able to calculate hashes on various SPUs to improve performance, am not sure how Zar implemented md5 hashing in MGZ.
    Such a feature could be added to xai_plugin (XMB CFW Setting) for instance, alongside the disc id dump feature...
     
    sandungas and jcorrea like this.
  4. 2,904
    2,428
    372
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    2,904
    Likes Received:
    2,428
    Trophy Points:
    372
    Location:
    Babylon 20xxE series
    Multithreading could be really nice to make the times shorter, i guess managunz does the IRD MD5 checks in single thread, not sure though
    And in latest versions of openPS3FTP i remember it was rewritten for multithreading, but as far i know this did not increased speeds
    So... dunno, but multithreading looks like something hard to achieve

    Maybe is needed to write some code especifically for the SPUs following some requirements
     
  5. 7,571
    6,184
    622
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    7,571
    Likes Received:
    6,184
    Trophy Points:
    622
    Location:
    Earth
    Ftp & md5 calculations are 2 different beasts though.
    I cannot recall how multithreading was applied to the ftp server but ftp is file ops & network streams mainly whereas the ird creation is file ops + calculations.
    If it's possible to hash simultaneously on various SPU threads, performance should be improved on calculation times. Afaik file ops perf (using syscalls, not slow libc) however are unlikely to be improved much.
    Still, overall performance improvement would of course depend on which of the 2 is currently the biggest bottleneck, file ops or calculations?
    So basically you are totally right, using multithreading wouldn't guarantee de facto a huge performance boost, the perf should improve but by how much, it is hard to tell. Some research & testing is needed.. The usual. Lol
    With a couple of quick edits, the Managunz code could be used to log file ops vs md5 calculations processing time. That would give us some idea of what to expect.
     
    Last edited: Nov 26, 2018
  6. 2,904
    2,428
    372
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    2,904
    Likes Received:
    2,428
    Trophy Points:
    372
    Location:
    Babylon 20xxE series
    Just found an interesting pdf while doing an small google search
    http://joppebos.com/files/PARA2008.pdf

    It seems to be part of the documentation of eBACS
    https://bench.cr.yp.to/

    @Zar ^
     
    jcorrea likes this.
  7. 7,571
    6,184
    622
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    7,571
    Likes Received:
    6,184
    Trophy Points:
    622
    Location:
    Earth
    Nice find, very interesting stuff actually.
    So it's not only possible to use SPUs but it's recommended if you have many calculations to perform, I was right. ;)
    And as you can read in that paper, the SPE is highly suited to this kind of work. The more calculations to perform, the more overall improvement.
    But the file ops processing time will probably remain the same no matter what.
     
    Last edited: Nov 26, 2018
    jcorrea and sandungas like this.
  8. 2,904
    2,428
    372
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    2,904
    Likes Received:
    2,428
    Trophy Points:
    372
    Location:
    Babylon 20xxE series
    I expected it but was nice to see the PS3 mentioned in that pdf as "high performance" for MD5 calculations :)
    It seems they ripped the MD5 algo to the bones to achieve the best performance, in the table at bottom of the document it looks his implementation have more performance than the official from IBM, lol

    Now im wondering if the people that wrote the pdf uploaded his code somewhere, they explained the algorithm anyway but well... is much better to have the code they was using
    I advised zar because i consider he is very skilled in repurposing code and fixing and patching problems, and managunz could be a good project for this experiments

    And... is needed to "send" the binary files to the SPEs in assembler especifically for them, not sure how hard could be to achieve this from a homebrew running in gameOS, right now i dont remember any homebrew doing it... maybe that game ported to PS3 by kakaroto lot of years ago that used lot of physics (and had move controller support by emulating the move controller as a USB mouse)
     
    Last edited: Nov 26, 2018
    jcorrea likes this.
  9. 7,571
    6,184
    622
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    7,571
    Likes Received:
    6,184
    Trophy Points:
    622
    Location:
    Earth
    They used their own optimised compression algo code written in spu ppc, that might be tedious to replicate especially at the level of perf mentioned in this paper.
    But still, decent md5 hashing algorithm are easy to find, compiling one for spu might be feasible.
    As to comms between ppe & spe, passing binaries & data from one to the other, I don't think that will be a problem. You could probably use arguments to pass the binaries offsets (binaries loaded into shared ppu/spu memory) then get the calculated hash back to the ppu through MFC/DMA or something along those lines.

    Check this paper.
    https://www.researchgate.net/public..._and_MD5_and_AES_on_the_Cell_Broadband_Engine
     
    Last edited: Nov 26, 2018
    jcorrea and sandungas like this.
  10. 2,904
    2,428
    372
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    2,904
    Likes Received:
    2,428
    Trophy Points:
    372
    Location:
    Babylon 20xxE series
    Thats the part that looks more challenging

    Looks much better explained :)
    Something that called my attetion... (in page 5)
    This looks convenient when checking MD5 of a ISO (a big file in a single data stream)
    Also, is making me think in what could be better... if "pairing SPUs" like that or not

    So... PS3 CELL have 8 SPUs, but one is disabled at factory, and other is used by hypervisor... so for hash calculations can be used 6
    And that method explained in the document uses 2 * SPUs for every data stream
    So i guess by using the implementation of that document CELL can handle 3 data streams in total runing in paralell

    Nevermind, is explained later in the document (page 15: "Application Overview")... there are several hashing modes availables, and are not using "just 2 SPUs" as i mentioned, it seems all the modes uses more than 2 SPUs

    The next thing im wondering is if the code they published have hardcoded the number of SPUs availables... or if there are parts of it dependant of the fact that they was using CELL processors with 8 SPUs... because they was working with CELL blade servers (same processor than the PS3 but with all SPUs unlocked)


    Edit:
    Sorry for the offtopic, the staff of the forum reads a lot of offtopic dailly all around the forum, and we are a bit permisive with it (incase are not cummulatesd too much offtopic posts), but give us a break, we go a bit offtopic too eventually ;)
    At this point i bet all we agree the amount of offtopic about this matter of IRD files and hash checks cummulated too much, so in the next days (after zar or other people interested takes a read at what we was talking) we should split the last post to a new thread
     
    Last edited: Nov 27, 2018
    bguerville and jcorrea like this.
  11. 45
    18
    7
    jcorrea

    jcorrea PSX-Place Supporter

    Joined:
    Sep 3, 2018
    Messages:
    45
    Likes Received:
    18
    Trophy Points:
    7
    Gender:
    Male
    Occupation:
    Developer
    Location:
    Curitiba/Brasil
    If you guys need, I can create a repo with 3K3y Ripp3r 1.2 src. It's not the last, but I'm sure it have source that can be used to guide us to develop a new feature to generate IRD on PS3 file managers. I even can try to help you, the only problem is I don't have a PS3 "CFW compatible".
     
  12. 2,904
    2,428
    372
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    2,904
    Likes Received:
    2,428
    Trophy Points:
    372
    Location:
    Babylon 20xxE series
    /!\ WARNING /!\
    Since this post consider the talk a new thread, the name of this new thread is just the last things we was talking about in the original posts, but if someone wants to talk about related things (research, development or other ideas) feel free to do it
     
    Last edited: Nov 28, 2018
  13. 45
    18
    7
    jcorrea

    jcorrea PSX-Place Supporter

    Joined:
    Sep 3, 2018
    Messages:
    45
    Likes Received:
    18
    Trophy Points:
    7
    Gender:
    Male
    Occupation:
    Developer
    Location:
    Curitiba/Brasil
    I created a repo to 3k3y Ripp3r. It has a feature to create IRD's. It can help us to develop something to use directly from PS3.

    https://github.com/jhonathanc/3k3y-Ripp3r
     
    Last edited: Nov 28, 2018

Share This Page