PS3 Apollo save tool (development thread)

[Berion]

That's interesting. Especially default size because minimum partition size is 128MiB, so the empty image should have around 1GiB if size from PS2 was kept or 640MiB if it's reduced to minimum (__mbr, __net, __sysconf, __content, __common). Until it's compressed of course.

Could You send me empty and filled such image? I'm working with someone currently on a tool which reads i.e PS2 HDD or it's PFS partitions images and this should put some light on it. Oh, and to not hijack Apollo thread, we can continue this on private messaging if You want. ^^

PS: You can sign any ELF to "BOOT2 KELF" by SCEDoormat NoME R3.

 

[sandungas]

That's interesting. Especially default size because minimum partition size is 128MiB, so the empty image should have around 1GiB if size from PS2 was kept or 640MiB if it's reduced to minimum (__mbr, __net, __sysconf, __content, __common). Until it's compressed of course.

Could You send me empty and filled such image? I'm working with someone currently on a tool which reads i.e PS2 HDD or it's PFS partitions images and this should put some light on it. Oh, and to not hijack Apollo thread, we can continue this on private messaging if You want. ^^

PS: You can sign any ELF to "BOOT2 KELF" by SCEDoormat NoME R3.

As far i remember in the file structure of IMAGE.DAT there is not a minimal size restriction, so i was modifying some fields to reduce it to half (64mb) and later to another half (32mb), because i considered that was enought for homebrew
The IMAGE.DAT is a file and is not going to be written in a real hdd, so i was trying to "fake" some fields

The original (and valid) IMAGE.DAT was several gb, is something i made years ago so i had many files and many notes.txt in dirty (not good to publish), my intention was to document it in psdevwiki but it was too much and lost the motivation, not sure if i still have the files in other PC other hdd or dunno... maybe i deleted them, i cant check them right now

About the kelf, i was talking with l_oliveira (at that time he was trying to crack the encryption for the PSX hdd) but there was no tools available to do it, or well... dont remember but i was not able to do it

Long story short, i dont have anything to share sorry, the best you can do is to ask someone with a BC PS3 model to perform that PS2 hdd installation and send the IMAGE.DAT to you to take a look at it

 

[Berion]

If You find those notes, I'm still interesting. ^^

 

[bucanero]

now that I have the PSV resigning working, I was thinking I could add:

- convert PS1 .mcs file to .PSV (+resign)
- convert PS2 save to .PSV (+resign)

I just need to research a bit about which PS2 save format to use, as I see there are a lot (armax, ems, sharkport, cbs, etc).
From what I saw, I think that using EMS (.psu) as input format would be a good choice, since tools like myMC support it...
another option would be the AR max format... I will compare and decide which one is the easiest /faster to implement.

btw, I tested my theory:
- I created a PS2 memory card using the PS3 XMB called "APOLLO0". (ps3 created savedata/vmc/APOLLO0.VM2)
- Apollo decrypted a ps2classics .VME , overwriting vmc/APOLLO0.VM2
- back on the XMB, since all the memory card stuff is cached on the internal DB, nothing was shown (empty card)
- to force the update, I went to my USB, selected a dummy PS2 PSV save to copy/import, selected the APOLLO0 card
- XMB copied the PSV save, and updated the whole MC content
- now all the saves stored in my VME from "PS2 classics placeholder" games where showing up
- PROFIT

from there, you're free to export your ps2 classic saves as regular PSV, or import saves to the VM2 card, etc... and finally, if you return to Apollo and encrypt APOLLO.VM2 back to VME, your ps2 classic games will be loading your modified memcard.

(I was a bit disappointed to find that the XMB also caches the internal memcard contents on the DB, but at least doing a save import forces a whole scan/rebuild without issues )

 

[sandungas]

Thats pretty cool, individual PS2 save management and conversions back and forth from the official PS2 formats, i guess this completes all the kind of conversions that could be made with the official PS2 save formats ?

We are lucky the importing of a single save works to update the XMB database, lol, nice trick, previouslly i used to rebuild database from recovery menu
What happens when that dummy PS2 PSV is imported btw ?, is stored as data and labeled as a valid save ? (so is needed to delete it manually later)... im wondering if there is some way to prevent the save to be created

Personally i liked AR max but im a bit biassed, it would be better to use something more generic without any security, btw there is a tool named "omniconverter" that seems to eat all the unnofficial PS2 save formats gracefully... not sure if the source code was published though

 

[bucanero]

Thats pretty cool, individual PS2 save management back and forth from the official PS2 formats, i guess this completes all the kind of conversions that could be made with the official PS2 save formats ?

We are lucky the importing of a single save works to update the XMB database, lol, nice trick, previouslly i used to rebuild database from recovery menu

yes I was hoping to avoid the whole rebuild database stuff, as it was totally overkill for only showing up stuff that was already there (but invisible)

What happens with that dummy PS2 PSV btw ?, is stored as data and labeled as a valid save ? (so is needed to delete manually later)... im wondering if there is some way to prevent the save to be created

Personally i liked AR max but im a bit biassed, it would be better to use something more generic without any security, btw there is a tool named "omniconverter" that seems to eat all this PS2 save formats gracefully... not sure if the source code was published though

the dummy PS2 PSV gets stored properly and labeled as valid save (it looks like the import-copy function was developed properly and doesn't write stuff blindly, so it actually reads the .VM2 file content before applying stuff)
e.g. If you already have a save with the same name, it will ask if you want to overwrite or not, etc...
After the update you can delete the dummy safely without issues.

I'd suggest keeping a dummy on USB for these tricks (maybe someone wants to craft a tiny hacked PS2 "apollo" save just for this purpose)

I've seen arMax format source code in a few tools (myMC, psv-exporter, and others)... it also uses a kind of LZH compression, but I need to review it further.

 

[sandungas]

the dummy PS2 PSV gets stored properly and labeled as valid save (it looks like the import-copy function was developed properly and doesn't write stuff blindly, so it actually reads the .VM2 file content before applying stuff)
e.g. If you already have a save with the same name, it will ask if you want to overwrite or not, etc...
After the update you can delete the dummy safely without issues.

I'd suggest keeping a dummy on USB for these tricks (maybe someone wants to craft a tiny hacked PS2 "apollo" save just for this purpose)

Good enought by now, but im wondering if we could craft a "malformed" PSV to create an error, and as result:
-the save is not stored
-but the database is updated

The idea is to "break" some info from the PSV header
https://www.psdevwiki.com/ps3/PS1_Savedata#PS1_Single_Save_.3F_.28.PSV.29

First candidate imo could be the field at offset 0x44 (indicates where the "raw" save data starts), if we set a value bigger than the PSV file size then is imposible to store any data when you import it because the PS3 is not able to find the data
The idea is... maybe the importing functions are going to start the importing procedures normally, and when they tryes to locate the data they are going to return an error, the XMB is going to do a "poof" and the database could be updated at that point

Something similar happens when swapping PARAM.SFO files and you try to boot weird things, the XMB "poofs" and the new PARAM.SFO is loaded instantly (at the same time the database is being updated at the background)

Dunno, is just an idea, and there are other candidates in the PSV header we could try to "break" to achieve this effect

 

[Berion]

No one using *.max. It is commercial container. I strongly recommending PSU as everyone keep saves in this format. ;p

Also remember to keep folder and files time stamps. It is used in some games as security layer (games refuse to load save if TT on mcfs and inside saves are different).

I made for You dummy PS2SD (if needed). Do not change PSV file name, it is used as validating data (for unknown reason, because the same is written in header...). It is made by application, not PS3 so I don't know if it is ok, in case of fail sig, I attached also *.psu and *.max variants.

 

[sandungas]

Well, the most generic and simple PS2 save format is the one created by uLaunchELF when you export a save (as far i remember by using the option named "McCopy")
The save is exported as folder/files, and yeah, as a general rule is needed to preserve his original timestamps and attributes (otherway some games refuses to load the save)

The problem is... to add support for this save format i guess it would be needed to add some kind of "PFS" filesystem driver to apollo (as far i know is the filesystem of the PS2 MC)
I mean... apollo should be able to create an empty PFS partition or at least to read/write his contents (but i guess this is not so easy to achieve)

 

[Berion]

No. PFS is filesystem on HDD on Sony partitions, wrapped in APA header. On PS2 Memory Cards there is "mcfs". Something similar to FAT16. ;p No partition on MC and no partition table, just whole device is fs space.

On PSX mc is even no file system. Just simple index and raw access to next blocks of data (by 8KiB).

 

[bucanero]

I made for You dummy PS2SD (if needed). Do not change PSV file name, it is used as validating data (for unknown reason, because the same is written in header...). It is made by application, not PS3 so I don't know if it is ok, in case of fail sig, I attached also *.psu and *.max variants.

thanks a lot!!! I'll test the saves I think this dummy Apollo ps2 save will be enough to trigger the XMB update.

Alright, to keep everyone happy (@Berion @sandungas ) , I implemented both .psu and .max PS2 conversions.

credits to PSV Explorer and Cheat Device PS2, for some code related to ps2 save formats:
https://github.com/PMStanley/PSV-Exporter
https://github.com/root670/CheatDevicePS2

(btw, if someone really needs to have the CodeBreaker format, the source from CheatDevicePS2 should be enough to add it)

Also remember to keep folder and files time stamps. It is used in some games as security layer (games refuse to load save if TT on mcfs and inside saves are different).

Yes I learned it the hard way, while I was testing my results. I found this additional "security" in the .PSV format that I'm not sure if it's mentioned anywhere. The filename of the .PSV file must follow an exact pattern, or else the PS3 will reject the savegame.

Let's take for example a FF12 save (US region):

the save folder is: BASLUS-20963FF1200

you take the product code: "BASLUS-20963"
then, "FF1200" converted in ASCII Hex >> "464631323030"
and the ".PSV" extension

so the PSV filename must be named:
BASLUS-20963464631323030.PSV

or else, even with the proper resigned header, the PS3 will fail with "data is corrupted" when trying to copy from XMB.

what I forgot to test, is if this "miss-named" file is enough to refresh the memcard without even saving the changes.

 

[Berion]

This hex conversion was implemented because some PSX titles using in "sub-id" symbols not allowed for FAT32. In example "Tales of Desitiny 2" ("Tales of Eternia" in Japan, but in US it is named ToD2 but do not mislead with real ToD2 for PS2 ;]). It has some slashes in filename. FYI.

Do You need also PSX dummy save? ;]

BTW: You have quoted something not related to answer. Time stamps have nothing to do with PSV filename.

 

[pinky]

This hex conversion was implemented because some PSX titles using in "sub-id" symbols not allowed for FAT32. In example "Tales of Desitiny 2" ("Tales of Eternia" in Japan, but in US it is named ToD2 but do not mislead with real ToD2 for PS2 ;]). It has some slashes in filename. FYI.

Do You need also PSX dummy save? ;]

BTW: You have quoted something not related to answer. Time stamps have nothing to do with PSV filename.

oh, yeah, it creates a log with the file name in it. I own that game. here's the log it created:

 

[bucanero]

BTW: You have quoted something not related to answer. Time stamps have nothing to do with PSV filename.

For some reason I was reading "Also remember to keep folder *names* and files time stamps."
I guess my mind added the word because I was already thinking about the filename issue that got me puzzled for a bit... I was like "why the hell the ps3 is rejecting the save if everything is resigned properly!"

Do You need also PSX dummy save? ;]

if I add something like raw PS1 memcard copy/import on Apollo, then I think a ps1 dummy might be needed to "refresh" the xmb. But first I need to finish all this ps2 stuff

I'll post the source and release a command line tool to generate+resign ps2 PSV files from .psu /.max saves, because I already did it for testing, and it might help somebody if they don't want to use Apollo (or don't have a ps3 around)

(for ps1 saves there's a tool already: https://bitbucket.org/SilicaAndPina/ps12psv/src/master/ )

 

[sandungas]

No. PFS is filesystem on HDD on Sony partitions, wrapped in APA header. On PS2 Memory Cards there is "mcfs". Something similar to FAT16. ;p

Ops, my mistake, is not PFS but something based on FAT, then the challenge to implement it is smaller than i thought
This could be mostly a feature for a filemanager though, because it would allow to browse the PS2 MC contents with a file explorer. I always liked it because is the most raw way
Im not telling that i always like to do things in raw, but incase of adding support for multiple stuff the most raw is always interesting

PSU and MAX could be more handy (btw great work @bucanero ), but is mostly because are a single file and are popular

No partition on MC and no partition table, just whole device is fs space.

I used the word "partition" because i consider a filesystem needs to be created inside a partition (or region) of a storage device (either real or virtual device)
But im not using the term in a technical way, is just it makes sense to me to think in it that way

On PSX mc is even no file system. Just simple index and raw access to next blocks of data (by 8KiB).

I know, it happens like in old consoles, the most we rewind back to previous console gens his saves was more simple, for the PS1 MC there is not much we can do, for the exporting/importing the saves needs to be treated as "slots" of a specific size
Most games uses 1 slot, and the others that exceeds the predefined size of an slot uses 2 slots or 3 etc... (and as far i remember there is some kind of flag in them that indicates how many consecutive slots composes the save), this is the only tricky thing for the matter we are discussing about exporting/importing them

 

[pinky]

each block is exactly 8,192 bytes. for games that use multiple blocks, it will be a multiple of that number. the first block is the file name list, with the 2nd through 16th blocks (15 blocks total) being the saves. the save name and game id don't appear to be in the saves themselves. I think that's what the first block is for.

 

[Berion]

@bucanero It works in the same way so high probably You will need. ;]

 

[bucanero]

here's my command line tool to convert PS1 (.mcs) and PS2 (.max, .psu) saves to .PSV , including proper resigning for PS3:

https://github.com/bucanero/psv-save-converter

Windows and macOS binaries can be found here:
https://github.com/bucanero/psv-save-converter/releases/tag/v1.0.0

 

[bucanero]

btw, about the .max vs .psu format, one drawback I found about the ArMax file, is that there's no record of the file's timestamps.

The .max headers don't keep track of that information at all, so the only thing I could do was take the "file.max" timestamps and use it on the savegame files.
In that sense, the .psu format is way more accurate, as it keeps the memcard entries intact, and you can restore that information when building up the .PSV.

I also reviewed the codebreaker format, and they do keep the file timestamp information in the CBS headers. I might end up adding Codebreaker to Apollo after all, since I already did most of the heavy stuff... I just need to add the RC4 encryption handling.

next, I need to add all this stuff back to Apollo, run some tests, and release a new version.

 

[sandungas]

.max format doesnt stores timestamps ?, thats an important flaw, i was not aware of it so i think i just become a .psu believer instantly