Various PS2 projects

Discussion in 'General PS2 Discussion' started by uyjulian, Oct 11, 2018.

  1. 53
    12
    37
    Anonamous

    Anonamous Member

    Joined:
    Aug 26, 2018
    Messages:
    53
    Likes Received:
    12
    Trophy Points:
    37
    Occupation:
    Electrical Engineering Student
    You just need to generate the yara signatures. If you have the libraries there is a python script to generate them. They also have a public set or signatures only their github page.

    It would probably be better to use ida with the retdec plugin if you have the pro version of ida.
     
  2. 148
    251
    122
    uyjulian

    uyjulian Developer

    Joined:
    May 27, 2017
    Messages:
    148
    Likes Received:
    251
    Trophy Points:
    122
    Gender:
    Male
    I think I figured out the reason why retdec isn't reading the debug information that I saw in the file.
    The executable is using an old "STABS" format of debugging information, not the newer "DWARF" format is used commonly.

    Well, it's not really a problem, as objdump can read the "STABS" format and export nice type information with -g flag.
     
    Anonamous likes this.
  3. 148
    251
    122
    uyjulian

    uyjulian Developer

    Joined:
    May 27, 2017
    Messages:
    148
    Likes Received:
    251
    Trophy Points:
    122
    Gender:
    Male
    I decided to look at a FUSE example. It appears that iomanX and FUSE functions are quite different.

    Here are some FUSE functions
    Code:
    .getattr       
    .readlink       
    .mknod         
    .mkdir         
    .unlink         
    .rmdir         
    .symlink       
    .rename         
    .link           
    .chmod         
    .chown         
    .truncate       
    .open           
    .read           
    .write         
    .statfs         
    .flush         
    .release
    .fsync         
    .setxattr       
    .getxattr       
    .listxattr     
    .removexattr   
    .opendir       
    .readdir       
    .releasedir     
    .fsyncdir       
    .init
    .destroy
    .access         
    .create         
    .ftruncate     
    .fgetattr       
    .lock           
    .utimens       
    .bmap           
    
    I can do ls, mkdir, rmdir, read, write, readdir, and rename pretty easily. I'm probably going to implement those first for the first release of my FUSE driver for PFS.
     
    svotib, Berion, Anonamous and 4 others like this.
  4. 148
    251
    122
    uyjulian

    uyjulian Developer

    Joined:
    May 27, 2017
    Messages:
    148
    Likes Received:
    251
    Trophy Points:
    122
    Gender:
    Male
    Let's talk about image formats for Open PS2 Loader.
    Currently, the formats for the following devices are supported:
    USB: Defragmented USBExtreme or ISO format on FAT32 filesystem (on MBR partition scheme)
    Network (SMB): ISO file
    HDD: 0x1337 HDLoader format on APA partition scheme

    I suggest a unified format for storing games on any block device.
    A file would have a list which would describe the game ID, LBA offset, and size which would be stored wherever it is convenient, such as the +OPL partition on the HDD, a file stored on the remote networked computer's filesystem, or the first FAT32 partition on a USB device.
    ISO data would be stored on the block device with a simple block-by-block copy.
    When sending block data over the network, the LBA offset can determine which ISO data on the remote filesystem is to be used.
    Reserved partitions on MBR and APA would prevent destruction of the ISO data.

    This would solve the 2TB limit, partition limits, file limits, fragmentation, and speed issues.

    I calculated the size of the redump.org PS2 set from the dat using perl, paste, and bc. It's 23045577394591 bytes, or around 23 TB. I'm curious if there are any limitations on the size supported by the PS2 ATA port? (But who is actually going to play 9239 games?)
     
  5. 710
    1,256
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    710
    Likes Received:
    1,256
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    We also support the USBExtreme format on SMB shares.

    We need different formats to better fit the designs of the filesystems of the various devices we support, and to also gain maximum flexibility.

    For example, we're constrained to 4GB files (maximum) on FAT32 (used on USB devices). Without some non-standard functions, we're also incapable of supporting files larger than 2GB. That's why we need the USBExtreme format. But yet, people also love to be able to just drag 'n drop ISO files, as how things are for the PSP.

    On the PS2 HDD unit, we have no such restriction, but it is also desirable to cut out PFS, as we can ensure that the ISO exists across a known number of segments, with a custom format. This isn't possible for the PC filesystems that we support.

    Then we have SMB shares: since the filesystem is managed by the SMB server, we are free from any restrictions. If you use a NAS, then it also becomes very simple to use.
    We are, however, still constrained by the 4GB limitation if the user uses FAT32. That's why we also support the USBExtreme format.

    If you design a system that will cater to the most restrictive format (in order to be universal), you will lose the flexibility of the more flexible format.

    IMO, you're re-inventing the filesystem. But the reason why we cannot just do this, is because we do not have enough memory to support a filesystem. Even if we used the absolute minimum amount (without reading in full clusters, no caching etc), performance can be very mediocre.
    It's why we cannot just write a micro filesystem driver (for any format), without any problems.

    But that being said, if you can somehow solve this problem, you will win the Internet.

    By supporting 48-bit LBA, we can (in theory) support up to 1.44PB. Since we're intending to keep APA around, we are limited to 2TB. But that format is also horrible for large disks, as you can probably tell.
     
    Last edited: Jan 6, 2019
    svotib, Berion and TnA like this.
  6. 2,263
    2,186
    372
    Berion

    Berion Developer

    Joined:
    Feb 3, 2015
    Messages:
    2,263
    Likes Received:
    2,186
    Trophy Points:
    372
    Gender:
    Male
    Location:
    rom0:/
    About internal fs: what if we exchange HDD modules (if possible, I don't know if whole firmware goes to RAM and if it is possible to replace anything from there on the fly) to support something normal, and different than APA and PFS like i.e GPT and i.e F2FS. Isn't this get rid of all PFS limitations? Of course it is titanic work but it is just an idea.

    Similar to external storage. Forgetting of MBR with FATs and focusing on i.e exFAT or F2FS (without partition table). There was some sort of experiment in OPL few years ago with EXT2.

    You are the experts, but for me, from raw point of view, this could free us from ancient limitations.
     
    Last edited: Jan 6, 2019
  7. 710
    1,256
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    710
    Likes Received:
    1,256
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    Such modules are provided by the software itself, hence the capabilities of any software is determined by its own design. The ROM of the PS2 is mostly just for booting up.

    Yes, it is possible to drop APA & PFS entirely, but that will be at the cost of no backward compatibility. Neither with existing homebrew software nor with Sony games.

    PFS is likely good to use. It's a 64-bit filesystem, similar to the types used with *nix.
    The problem is APA. But I must say that it's still somewhat meeting our needs. It's only going to be truly left behind, should we (somehow) desire to move beyond 2TB.

    MBR is okay IMO. It's because APA is different from MBR, in very bad ways.

    In APA, each application will have a parent partition, and may have a child partition. Each of them may have 1 main partition, as well as up to 64 sub-partitions. The maximum partition size on a disk is 1/32 of the disk's capacity, limited to 32GB.
    So if you have 500 titles, for example, you could have something like 1000 partitions. We also try to save space by mixing and matching partitions (i.e. for a 4.1GB game, it might consume 2 partitions, one 4GB and one 128MB).

    It's poor for large disks because it has no partition table, requiring the driver to scan through all partitions. Each partition is also very spread out, so the HDD's seek time gets ramped up (if there are 1000 partitions, 5ms will become 5000ms). It's likely why using a SSD will magically solve all its problems.
    Also because there's no partition table, the cache system of APA involves storing the partition's data in memory. That's 1KB per partition, which results in high memory usage. Since that's not possible, so the cache remains small and hence becomes insufficient for large disks.

    When we comes to MBR, you can have up to 4 partitions, but nothing really forces you to have a specific number of partitions. You can have any number of applications on a partition, up to the limits of the filesystem.

    I was skeptical about the capabilities of E2OPL (if I remember right) because I did try to make a micro FAT32 driver, and failed to make it work very well (the old commits are there).
    But because I had a lot to deal with (in real life) during that period of time, I did not look at what its author did and did not involve myself with his progress. So I don't know for myself, whether it really worked as well as claimed, which is why I still have my doubts.
     
    Last edited: Jan 6, 2019
  8. 2,263
    2,186
    372
    Berion

    Berion Developer

    Joined:
    Feb 3, 2015
    Messages:
    2,263
    Likes Received:
    2,186
    Trophy Points:
    372
    Gender:
    Male
    Location:
    rom0:/
    Oh, right, I forgot that PS2 doesn't have OS in background providing support for higher layers. :(

    Could You elaborate more about APA limitations?
     
    Last edited: Jan 6, 2019
  9. 1,076
    556
    222
    TnA

    TnA Senior Member

    Joined:
    Jul 1, 2018
    Messages:
    1,076
    Likes Received:
    556
    Trophy Points:
    222
    Gender:
    Male
    Location:
    Germany --> Saxony
    @Berion: You are mixing up APA and PFS... Both have their own limitations...

    Implementing a GPT-Based partition-structure to exceed the 2TB-Boundary would render EVERY other Homebrew and Sony-Software which accesses the HDD (even the OSDSYS) incompatible with it.

    Filesystems already can be implemented on the HDD (and also USB), but their benefits would be marginal and the addition of another rather complex FS would atleast not be a wise choice for OPL and games...
    That would require a more complex 'driver', which would at least require more IOP-RAM and CPU-Cycles + Fragmentation could be a real pain in the a** (especially for games)...
    That PROBABLY would lead to some more games being incompatible and POSSIBLY can let FMVs even stutter from the internal HDD (at least on a quite heavily fragmented HDD).

    Edit: Arrrrr... 2 new replies, while I wrote... :D
     
    Berion likes this.
  10. 2,263
    2,186
    372
    Berion

    Berion Developer

    Joined:
    Feb 3, 2015
    Messages:
    2,263
    Likes Received:
    2,186
    Trophy Points:
    372
    Gender:
    Male
    Location:
    rom0:/
    I don't mixing them, I know both are two different things. ;) PFS is a file system, and APA is... if not partition table (because there is MBR as I remember, right?), then what exactly? I don't fully understand concept behind it. But at the end there is user which cannot management data outside PS2 environment because of this two evil twins (APA+PFS, and here such fuse module could come in very handy).
     
  11. 710
    1,256
    222
    sp193

    sp193 Developer

    Joined:
    Oct 13, 2014
    Messages:
    710
    Likes Received:
    1,256
    Trophy Points:
    222
    Location:
    Singapore
    Home Page:
    Yes, APA is the scheme for allocating space for individual filesystems, much like MBR. But unlike the PC where we keep many applications in one filesystem, they just decided that each application will have its own filesystem. And unlike MBR (which can have only 4 partitions at most), there is virtually no limit to the number of partitions because APA uses a linked list format... across the HDD's sectors.

    Back then, the official HDD was only 40GB, so there were only so many partitions for APA to manage. I don't think the folks at Sony ever envisioned anybody owning the whole PS2 collection on a single HDD, back in 1999. :D

    The use of APA allowed Sony licensees to restrict other licensees from accessing their data on the user's HDD (through setting passwords), so it does have its own use as well. Maybe you can also consider the fact that the system remains functional as long as the __system partition is intact, as a feature.
     
    Berion and svotib like this.
  12. 148
    251
    122
    uyjulian

    uyjulian Developer

    Joined:
    May 27, 2017
    Messages:
    148
    Likes Received:
    251
    Trophy Points:
    122
    Gender:
    Male
    I was thinking about just passing the LBA offset to the in-game driver, without any additional filesystems.
     
  13. 53
    12
    37
    Anonamous

    Anonamous Member

    Joined:
    Aug 26, 2018
    Messages:
    53
    Likes Received:
    12
    Trophy Points:
    37
    Occupation:
    Electrical Engineering Student
    Forgive me as I mainly program in python for high level programming so I have a tendency to think of things in terms of python. Would it be possible, for example, create a dictionary of a game's files and their corresponding LBA's and then overload the game's sceCdSearchFile to instead use the dictionary and put the corresponding information into the sceCdlFILE object?
     
    Last edited: Jan 7, 2019
  14. 148
    251
    122
    uyjulian

    uyjulian Developer

    Joined:
    May 27, 2017
    Messages:
    148
    Likes Received:
    251
    Trophy Points:
    122
    Gender:
    Male
    It is possible to create a mapping of files to LBA.
    However, some games like Star Ocean 3 use a custom filesystem. The LBA can be mapped to files if the custom filesystem also aligns entries to LBA boundaries.
     
  15. 53
    12
    37
    Anonamous

    Anonamous Member

    Joined:
    Aug 26, 2018
    Messages:
    53
    Likes Received:
    12
    Trophy Points:
    37
    Occupation:
    Electrical Engineering Student
    Not sure if you meant can or can't. I don't see how it wouldn't work though as everything should work in theory as long as it takes the same inputs and returns the same outputs. Even if it has a custom filesystem it still has to find a file on disc using what was provided in the IORP img yes?
     
  16. 148
    251
    122
    uyjulian

    uyjulian Developer

    Joined:
    May 27, 2017
    Messages:
    148
    Likes Received:
    251
    Trophy Points:
    122
    Gender:
    Male
    I misread your post. The LBA offset is the starting point for where the files are read by various CDVDMAN functions.
     
  17. 2,263
    2,186
    372
    Berion

    Berion Developer

    Joined:
    Feb 3, 2015
    Messages:
    2,263
    Likes Received:
    2,186
    Trophy Points:
    372
    Gender:
    Male
    Location:
    rom0:/
    @sp193 So the APA is a MBR/GPT/etc. "replacement"?
     
  18. 1,076
    556
    222
    TnA

    TnA Senior Member

    Joined:
    Jul 1, 2018
    Messages:
    1,076
    Likes Received:
    556
    Trophy Points:
    222
    Gender:
    Male
    Location:
    Germany --> Saxony
    @Berion: In Layman-terms, yes! Technically, not quite!

    But it is a 'partitioning-scheme', so MBR/GPT is indeed the closest PC-Example similar to it (at least the closest quite known thing).


    But if someone has a better example to compare it with, I'd like him/her to post it as well!
     
    Berion likes this.
  19. 148
    251
    122
    uyjulian

    uyjulian Developer

    Joined:
    May 27, 2017
    Messages:
    148
    Likes Received:
    251
    Trophy Points:
    122
    Gender:
    Male
    Progress on reversing dsnet: it's just a copy-paste-edit job now.
    Ditto for the ps2sdk Makefile update.
    I'm going to see if some of my friends are interested in doing it.
    I have more consolidated time these 6 months, so hopefully I'll be able to do more things.

    I have a Patreon: https://www.patreon.com/uyjulian
     
  20. 292
    148
    97
    jcorrea

    jcorrea PSX-Place Supporter

    Joined:
    Sep 3, 2018
    Messages:
    292
    Likes Received:
    148
    Trophy Points:
    97
    Gender:
    Male
    Occupation:
    Developer
    Location:
    Curitiba/Brasil
    Do someone know how do I compile PS2SDK? I'm just trying to compile Snes9x to PS2, but I'm completely newbie on this scene. I tried with ps2toolchain script, but I'm not sure if I did it "the right way", because I'm getting the following error: ee-g++ not found (not exactly with these words, because I'm not on front of my computer right now, but with the same meaning).
    I intend to upgrade an old source that I've found on github:
    https://github.com/jhonathanc/snes9x
     

Share This Page