PS1 Lib Enigma - A PlayStation 1/2 CD Image C Library For Patching And Identification v1.0.4

[alexfree]

| GitHub |
Enigma: a person or thing that is mysterious, puzzling, or difficult to understand.
Paradox: a seemingly absurd or self-contradictory statement or proposition that when investigated or explained may prove to be well founded or true.

This is a C library that can be used in your program to:

• Ensure validity of a PlayStation 1 or 2 CD disc image in MODE2/2352 format.
• Get the boot file name (i.e. SLUS_XXX.XX) of the disc image. PSX.EXE games are supported (some still need to be identified by the library but many already are). When a PSX.EXE game is identified, the serial/product code on the original CD case of the game is returned in the same manner as a proper boot file game. See example 1 and example 2.
• Identify different disc images that share the same boot file name by checking revision differences. See example 2.
• Identify if a bin file will fit on a 71 minute, 74 minute, or 80 minute CD-R.
• Apply PARADOX PPF v1, PPF v2, and PPF v3 patches from unsigned char arrays. You can convert your PPF patch file into an unsigned char array, use the library to apply the patch, and create a standalone single executable patcher for your project/tool that requires no external files (see example 2).
• Find and replace method for byte patterns, with advanced options. Traditional find and replace with i.e. a hex editor DO NOT WORK for BIN/CUE PSX CD images because of how raw sectors are laid out:

Full Sector (0x930/2532 bytes) layout: Sync header - 0x18 bytes - tells the psx what sector/where in the CD the laser is reading. User data - 0x800 bytes - actual data for the game or software. EDC - 0x04 bytes - a checksum used to check if a data read by the CD drive was successful. ECC - 0x114 bytes - repair data, if the EDC checksum mismatches with what is read, this data can be used to 'fix' what was read, unless it's really off.
So keep in mind sectors are chained together. So you could have a byte pattern that starts at the very end of the user data on say sector 40. The data will be cut off at offset 0x818 in the sector (a total of 0x800 bytes are in the user sector, then the header itself was 0x18 bytes). So then you have your 0x04 bytes of EDC, 0x114 bytes of ECC, ANOTHER header of 0x18 bytes for sector 41, then finally you arrive at the user data of sector 41. This is where the REST of the byte pattern ends in this example. Because half of the byte pattern was cut off 0x136 bytes before the rest of it started, simple hex editor find n replace will FAIL if you encounter something like this.

Another thing about this feature, is you can specify offsets that you want to ignore in your byte pattern. So for example if you are looking for a byte pattern of 08, 0F, 16, 24, and after 24 you know there is going to be 6 bytes you don't care about, but then there is 1C, 37, 4A that is supported! You can tell the library to ignore certain offsets of your pattern, and it will do so! This increases the strength of finding your unique pattern, and enables more advanced find and replace. See example 3.

3 examples are available, compiled for multiple operating systems w/source code:
Example 1: Prints out boot file of psx game bin file, with support for early PSX.EXE games. Example 2: Single executable that uses no external files, applies english translation patch w/save fix for the game King's Field (Japan). Example 3: Patches LibCrypt v1 if it detects the game MediEvil (Europe) using 'lossy' patching without hardcoded offsets. If any other game is given as input it will patch APv2 protection if it is detected. This patch for APv2 protection allows the game to work with a soft-mod correctly, if their is not a non-stealth mod-chip in the console.

 

[alexfree]

I'll be honest, I didn't test on Windows XP because I KNEW it would work. But here's photo evidence, on a real computer (not a VM!):

In fact, this comment to the thread as well as screenshot was taken on XP as I finally have a real XP machine again. Writing proper portable C99 libraries/software and cross compiling with modern mingw will always be magical to me. For literally no extra work you get Windows 95 OSR 2.5+ on Pentium (ONE) CPU or better compatibility. This is why C is the best lang. The same library? Yea it runs on modern Linux and ancient to modern Mac OS too.

 

[alexfree]

Wow, the library I designed for my PSX patcher programs is letting me delete a bunch of worse code that does what the library does now in a better way for LibCrypt patcher . I'll post the commit changes for the developers out there designing ps1 patcher programs to show this. Oh wait, that's basically just me. The funny thing is many people will still benefit from release binaries so still 100% worth it!

https://github.com/alex-free/libcrypt-patcher/blob/master/lcp.c

https://github.com/alex-free/libcry...67ce76ef82eb8cf8472b1b0d8b5bc07f3451ee8e74f5e here's the implementation for the new LibCrypt patcher v1.1.0 update.

 

[Ronnie Sahlberg]

Very nice. I think I will find use for this.
Disc identification is a difficult task.
How do you check it? Do basically just parse the "system.cnf" file in the filesystem on the cd/iso?
For the boot line?
"BOOT = cdrom:\SLUS_008.84;1"

Or do you read the disc label or something else?


In pop-fe I found that parsing system.cnf worked well for ~99% of NTSC-U commercial releases but much less for demo discs.
PAL discs were worse, and NTSC-J discs did not really follow the cdrom:\mangled-disc-id pattern much at all.

Quite a lot of demo disks did not store any indetifier at all. And at least one commercial game did contain the WRONG disc id !!!!!


So I had to revert to fingerprint the more difficult disks.
The fingerprint is just the md5sum of the first 1M of the iso in track 1. Not the bin file but the iso extracted from the bin file.

In this file you have my current database:
https://raw.githubusercontent.com/sahlberg/pop-fe/refs/heads/master/gamedb.py

The mapping is stored in gameid_by_md5sum[]

There are currently 484 entries in this map.
Most of them are for games, commercial or demo, that could not be identified any other way.
Some of them are for different revisions of disks. Here I only add them to the database if I need to know the exact revision in order to apply a ppf. So not many different revisions tracked here.
And also there are a bunch of fan-translations and homebrew where I want to be able to detect.
For the homebrew disks I just made up my own ABCD12345 name.

Please feel free to grab that table for your library if you find it useful.

Note, the 'id' in these maps are not the disc id of the game. It is the entry in the "disc database" games[].

So of you find a match in the fingerprint map, for example this entry :


'7c1b3596b4483b9420b843cbbfbb59b6': {
'id': 'SLPS01940_ENG_101',
},

Then you have to lookup games['SLPS01940_ENG_101'] :
and you will find :



'SLPS01940_ENG_101': {
'url': "games/J/S/SLPS-01940.html",
'id': 'SLPS01940',
'title': "SEPTENTRION - OUT OF THE BLUE (ENGLISH 1.0.1)",

Which then tells you the actual game is 'SLPS01940' (and that this is the english fan translation of this game, revision 1.0.1)

 

[GuilloteTesla]

So I had to revert to fingerprint the more difficult disks.
The fingerprint is just the md5sum of the first 1M of the iso in track 1. Not the bin file but the iso extracted from the bin file.

One question: what values do you get from different revisions of a same game?. I.e. Mortal Kombat Trilogy has 2 revisions: 1.0 and 1.1.

I'm suspecting that both discs return the same MD5 for the first 1M. Just curious.

Great job!, your work is amazing!.

 

[alexfree]

Very nice. I think I will find use for this.
Disc identification is a difficult task.
How do you check it? Do basically just parse the "system.cnf" file in the filesystem on the cd/iso?
For the boot line?
"BOOT = cdrom:\SLUS_008.84;1"

Or do you read the disc label or something else?


In pop-fe I found that parsing system.cnf worked well for ~99% of NTSC-U commercial releases but much less for demo discs.
PAL discs were worse, and NTSC-J discs did not really follow the cdrom:\mangled-disc-id pattern much at all.

Quite a lot of demo disks did not store any indetifier at all. And at least one commercial game did contain the WRONG disc id !!!!!


So I had to revert to fingerprint the more difficult disks.
The fingerprint is just the md5sum of the first 1M of the iso in track 1. Not the bin file but the iso extracted from the bin file.

In this file you have my current database:
https://raw.githubusercontent.com/sahlberg/pop-fe/refs/heads/master/gamedb.py

The mapping is stored in gameid_by_md5sum[]

There are currently 484 entries in this map.
Most of them are for games, commercial or demo, that could not be identified any other way.
Some of them are for different revisions of disks. Here I only add them to the database if I need to know the exact revision in order to apply a ppf. So not many different revisions tracked here.
And also there are a bunch of fan-translations and homebrew where I want to be able to detect.
For the homebrew disks I just made up my own ABCD12345 name.

Please feel free to grab that table for your library if you find it useful.

Note, the 'id' in these maps are not the disc id of the game. It is the entry in the "disc database" games[].

So of you find a match in the fingerprint map, for example this entry :


'7c1b3596b4483b9420b843cbbfbb59b6': {
'id': 'SLPS01940_ENG_101',
},

Then you have to lookup games['SLPS01940_ENG_101'] :
and you will find :



'SLPS01940_ENG_101': {
'url': "games/J/S/SLPS-01940.html",
'id': 'SLPS01940',
'title': "SEPTENTRION - OUT OF THE BLUE (ENGLISH 1.0.1)",

Which then tells you the actual game is 'SLPS01940' (and that this is the english fan translation of this game, revision 1.0.1)

I do both. Basically the logic is in psuedo code:

1) Check if a file named SYSTEM.CNF exists in the root dir of the disc image. If it does not, check if a file named PSX.EXE does.

2) If SYSTEM.CNF exists, go to that LBA and start reading the content which comes after the : or \ on the line containing cdrom:, cdrom:\\, or cdrom:\ (all are valid) until you hit the ;1 (which signifies v1 of the file, always will be there and is good to signal then end of reading).

3) If the value just read from SYSTEM.CNF is PSX.EXE (yes this exists, especially in some older Japanese games) then flag it as a PSX.EXE game as well.

4) If flagged as PSX.EXE game, check the volume creation timestamp (16 unique bytes of when the game's original ISO was generated by tools). Check that against an internal database, if it matches I know what the serial of the game is, and return that instead of the bootfile. So for example we know that King's Field 1 (Japan) is a PSX.EXE game with the serial SLPS_00017 (http://redump.org/disc/7072/.) so I just return the bootfile as SLPS_000.17 when it matches the volume creation timestamp 1994111013000000 which is YYYYMMDDHHMMSSFF.

So far no conflicts have been found using volumecreationtimestamp. I also made an example that will print out what the volume creation timestamp is if it's not in the database already (example 1 psx bootfile printer source: https://github.com/alex-free/lib-enigma/tree/master/example-1-boot-file-printer, compiled win32 binary: https://github.com/alex-free/lib-en...oot-file-printer-v1.0-windows-i686-static.zip, other binaries available for this example here: https://github.com/alex-free/lib-enigma/releases). If there were to ever be a conflict, I could use the id_rev() function which you just give it an offset and 2 different bytes. If byte xx is at offset y instead of byte zz, then you know what you have. I just find the first different byte between the disc images in question and check that (my other tool aprip automates figuring out what this is using the diff functionality, but I'll probably make a more streamlined approach eventually but it's good enough for now).

The offset byte diff id_rev is also how I know for example if a different disc revision needs to be detected, or if a PPF patch has already been applied. See example 2 king's Field Japan English Translation patcher: https://github.com/alex-free/lib-en...field-1-english-translation-patcher/example.c .

Relevent code to see is (note these are 'permalinks' so If I update this it will still be links to the exact lines these funcs start at for Lib Enigma v1.0):

get_boot_file_name(): https://github.com/alex-free/lib-en...0d1236cb813aa3159931251ac84/lib-enigma.c#L770

get_volume_creation_timestamp(): https://github.com/alex-free/lib-en...0d1236cb813aa3159931251ac84/lib-enigma.c#L711

get_psx_exe_gameid() https://github.com/alex-free/lib-en...a0d1236cb813aa3159931251ac84/lib-enigma.c#L10

The volume creation timestamp func is written a bit weird because it's pretty much directly lifted from Tonyhax International. It's how I identify PSX.EXE games for MemCardPro devices. Someone here also used the same method for their PS2 project to identify them giving me credit.

 

[Ronnie Sahlberg]

Volume creation timestamp. That is clever.
The timestamp does fall inside the first 1MB so I guess we check the "same" thing, just differently
(I should have thought about the timestamp. It would have been easier.)

My scope is slightly different though. I also want to identify "discs" that were never officially released, such as prototypes/beta, cancelled games, fan-translations, homebrew etc.
And other games that were never released officially. For example all the Russian games that I guess were "released and sold" to only play on consoles with modchips etc?

I am also thinking about harvesting identification data from redump.org. They have several useful pieces of metadata for the games,
such as the disc-id, region and revision. And they have things like "md5 over the first track" which can be used to identify the disc.
They don't have many demo discs :-( and they don't have homebrew or fan-translations or unreleased stuff, but it is a start.


Would you be interested in working with me on a database for this that we can can both use?


@GuilloteTesla The 1MB check I do can tell them apart. The volume creation timestamp for example, and a bunch of other disc-unique metadata is in this region so collisions are very unlikely.
These are my md5 fingerprints for MKT NTSC-U:

Beta: e543a2dc3413d3b459a0a46f06945aed
v1.0: 92d8e70c652f3e5938f8c9fddaa74534
v1.1: b828aac80855cfc0e452ebf161f2b365

 

[GuilloteTesla]

@GuilloteTesla The 1MB check I do can tell them apart. The volume creation timestamp for example, and a bunch of other disc-unique metadata is in this region so collisions are very unlikely.
These are my md5 fingerprints for MKT NTSC-U:

Beta: e543a2dc3413d3b459a0a46f06945aed
v1.0: 92d8e70c652f3e5938f8c9fddaa74534
v1.1: b828aac80855cfc0e452ebf161f2b365

Awesome, thank you for the explanation!.

 

[alexfree]

Volume creation timestamp. That is clever.
The timestamp does fall inside the first 1MB so I guess we check the "same" thing, just differently
(I should have thought about the timestamp. It would have been easier.)

My scope is slightly different though. I also want to identify "discs" that were never officially released, such as prototypes/beta, cancelled games, fan-translations, homebrew etc.
And other games that were never released officially. For example all the Russian games that I guess were "released and sold" to only play on consoles with modchips etc?

I am also thinking about harvesting identification data from redump.org. They have several useful pieces of metadata for the games,
such as the disc-id, region and revision. And they have things like "md5 over the first track" which can be used to identify the disc.
They don't have many demo discs :-( and they don't have homebrew or fan-translations or unreleased stuff, but it is a start.


Would you be interested in working with me on a database for this that we can can both use?


@GuilloteTesla The 1MB check I do can tell them apart. The volume creation timestamp for example, and a bunch of other disc-unique metadata is in this region so collisions are very unlikely.
These are my md5 fingerprints for MKT NTSC-U:

Beta: e543a2dc3413d3b459a0a46f06945aed
v1.0: 92d8e70c652f3e5938f8c9fddaa74534
v1.1: b828aac80855cfc0e452ebf161f2b365

Thanks! I kept starring at the no $ PSX spx docs trying to figure out a way to identify PSX.EXE that wouldn't involve a checksum or a ton of space. In Tonyhax International the executable is less then 50KB and for performance reasons I didn't want to do the checksum/it seemed harder to write then just checking nicely sized 16 bytes

Fun fact, there was Y2K bug with (older?) sony tools that would make the YYYY in volume creation timestamp 1900 IIRC.

The scope could be similar/expanded. I just started with PSX.EXE games I already identified in Tonyhax International, but would like to expand to betas/protos/things like Resident Evil 1.5 vanilla/fan mod/etc could be useful too. Really all the functions are there even for that already in the library like I said. Basically volume creation timestamp + id_rev(), or if it's some russian game with a SYSTEM.CNF saying like MAIN.EXE coud check for that and go off volume creation time stamp as well. That's what's great about libraries. And I'm sure you know many betas just went with PSX.EXE bootfile even into the late 90s, like the DOA 1 beta and stuff.

I would definitely like to do a database. I'm always pressed for time now but I managed to finally ship this library which I've been "getting around to" since I first wrote LibCrypt patcher in 2023. Feel free to use any of the volume creation timestamps I've already found. And I intentionally made that example 'bootfile printer' print unknown volume creation timestamps to make it easier then opening up a hex editor.

 

[Ronnie Sahlberg]

I have started working on the redump database (you can download a "dat" file for PS1, unfortunately it does not contain all the useful metadata but enough to create a mapping between md5sum of the first track and the game name.
Using that I have a small program that can then cross-reference my database and discover the disc-id.
I also read the volume creation timestamp off the disc,

Currently it looks lsomething like this :

'0d44c34f036763c1c1433af0aa0740b9': {'id': 'SLUS01090', 'title': 'Arthur! Ready to Race (USA)', 'vct': b'0000101316561100$'},

There are so far quite a few disks, including release games, that have a vcf that does not follow the starndard.
A little surprising as I would have thought that the tooling would automatically populate this.

Both mine and redumps database contains almost 10900 discs. There are some 300 or so that differ between them so I will need to resolve those diffs. Looks mostly like they have entries I don't have and I have entries they don't have.
(I add fan translations and homebrew, they don't)

This will take a fair amount of time to complete but I think it will be worth it. It would allow tools to not just identify discs but also implicitely verifying that the dump is correct.
Are there any particular other fields you think would be useful to add ?
Right now I have title, id and vct but can easily add more.

It will take a fair amount of time. There are almost 11.000 entries and at least 300 that needs to be handled manually.


A fun thing I noticed in the redump data is that there are a handful of games that had multiple DIFFERENT releases but where Track 1 was identical.
For example the game : Risk (Europe)
has two different entries at redump where Track 1 are identical (md5 40fb457e1694698fa7a57aae053d4332)
but they have different cdda tracks.

I did not expect that.

 

[alexfree]

I have started working on the redump database (you can download a "dat" file for PS1, unfortunately it does not contain all the useful metadata but enough to create a mapping between md5sum of the first track and the game name.
Using that I have a small program that can then cross-reference my database and discover the disc-id.
I also read the volume creation timestamp off the disc,

Currently it looks lsomething like this :

'0d44c34f036763c1c1433af0aa0740b9': {'id': 'SLUS01090', 'title': 'Arthur! Ready to Race (USA)', 'vct': b'0000101316561100$'},

There are so far quite a few disks, including release games, that have a vcf that does not follow the starndard.
A little surprising as I would have thought that the tooling would automatically populate this.

Both mine and redumps database contains almost 10900 discs. There are some 300 or so that differ between them so I will need to resolve those diffs. Looks mostly like they have entries I don't have and I have entries they don't have.
(I add fan translations and homebrew, they don't)

This will take a fair amount of time to complete but I think it will be worth it. It would allow tools to not just identify discs but also implicitely verifying that the dump is correct.
Are there any particular other fields you think would be useful to add ?
Right now I have title, id and vct but can easily add more.

It will take a fair amount of time. There are almost 11.000 entries and at least 300 that needs to be handled manually.


A fun thing I noticed in the redump data is that there are a handful of games that had multiple DIFFERENT releases but where Track 1 was identical.
For example the game : Risk (Europe)
has two different entries at redump where Track 1 are identical (md5 40fb457e1694698fa7a57aae053d4332)
but they have different cdda tracks.

I did not expect that.

Let me blow your mind, a retail PSX game with 2 data tracks: http://redump.org/disc/508/ and it reads the 2nd data track too during boot up.

great job. I can't think of anything else at this time.

 

[alexfree]

@Ronnie Sahlberg this is 1) very old code from me and 2) was what I used as well (like you) before moving to a curated collection of patches for LibCrypt patcher. Just funny thinking you did a python implementation and I did a C one before we figured out better ways: https://github.com/alex-free/aprip/blob/767fa1ded63076e2380822986120170272420443/aprip.c#L574

my new lib-enigma supports this generalized method in a much better way… but really since we have a good amount of the protected games with specific patches it only makes sense now if we had some random beta pop up from someone who bought a 30 year old dev computer with a rip of some substantial game or something. But I used the same exact docs as you back in the day and just thought you'd might enjoy this read

 

[Ronnie Sahlberg]

@Ronnie Sahlberg this is 1) very old code from me and 2) was what I used as well (like you) before moving to a curated collection of patches for LibCrypt patcher. Just funny thinking you did a python implementation and I did a C one before we figured out better ways: https://github.com/alex-free/aprip/blob/767fa1ded63076e2380822986120170272420443/aprip.c#L574

my new lib-enigma supports this generalized method in a much better way… but really since we have a good amount of the protected games with specific patches it only makes sense now if we had some random beta pop up from someone who bought a 30 year old dev computer with a rip of some substantial game or something. But I used the same exact docs as you back in the day and just thought you'd might enjoy this read

Yeah, a set of curated PPF patches is always going to be superior.
While the process in the doc seems to work fine, I was always a little surprised that they worked so well.
There is so little context to match on for the automatic patching an the files are always several hundred mb in size
so assumed there would be a lot of false positives.
I guess most of any false positives just happene in the ecc data of just iside media data and not in executable code ...

Better than nothing but better ways exist now. And 100% carefully curated patches (or SBI files) is the way to go, unless like on SONYs own emulators you can just tell it "please do what needs to be done".

I am making progress on a database based on using the md5sum for the first track and mapping it to disc id and game name, and the volume creation timesstamp.

You can have a look here:
https://github.com/sahlberg/pop-fe/blob/master/db.py

There is still about ~1000 entries that I need to manually reconcile but the idea is to have a database that consists of every disc image.
All the redump ones, that are the golden standard, for disks that exist in physical form but also every fan translation, every homebrew, every unreleased games, ...
But right now it has ~10.000 entries populated.

It is slow progress but I am getting there, at some stage, when it is more complete, I will start using it from pop-fe as the primary way to identify disks, falling back to the old/current detection if it fails.
This will also allow me to do things like showing a dialog when a game image has the "wrong" md5sum.

Like "You are trying to convert SLUS12345 but the disc image does not match the expected md5. Bad dump?"


Entries with a title of UNKN00000 means the pop-fe detection could not figure out the name.
Mostly because the the executable is PSX.EXE or MAIN.EXE or something and the disc volume name didn't provide any clues either.
I need to manually go and populate these with correct disck ids later on.

I also need to decide how/what naming convention I should use for "discs" that never existed in a released version. I have been pretty inconsistent before but think I should be more stringent now.
For fan-translations I now keep the original disc id in the translation but add a "(T) to the title.

For example :
'36a96b8e813c1f902446f7fed7d88dd3': {'id': 'SLPS01783', 'title': 'Mizzurna Falls (Japan)', 'vct': b'1998111518205900$'},
'b376bf1ecd1524f4ca5ef97855c1656f': {'id': 'SLPS01783', 'title': 'Mizzurna Falls (English) (T)', 'vct': b'1998111518205900$'},


But then also things that never existed as physical in the first place, like these:
'4f6baa774fa5e5cb81d9283b75326966': {'id': 'SLUS99999', 'title': 'RE1.5 Magic Zombie Door (Unreleased)', 'vct': b'1899123000000000\x
00'},
'ec8b8ec29385c0e0eb8627c3979f5c44': {'id': 'RUSS00004', 'title': 'Randevu s neznakomkoj (Russia) (En,Ru) (Disc 1) (Unl) (Alt)', 'vct
': b'2000121020202979\x0'},

For these latter cases I just made up ad-hoc ids. Not great and I need to redo this and come up with a better, consistent, way to generate an id for these.
The id need to be unique but should also clearly indicate it is not an official disk.
Any suggestions for naming convention for these type of entries?

 

[alexfree]

Yeah, a set of curated PPF patches is always going to be superior.
While the process in the doc seems to work fine, I was always a little surprised that they worked so well.
There is so little context to match on for the automatic patching an the files are always several hundred mb in size
so assumed there would be a lot of false positives.
I guess most of any false positives just happene in the ecc data of just iside media data and not in executable code ...

Better than nothing but better ways exist now. And 100% carefully curated patches (or SBI files) is the way to go, unless like on SONYs own emulators you can just tell it "please do what needs to be done".

I am making progress on a database based on using the md5sum for the first track and mapping it to disc id and game name, and the volume creation timesstamp.

You can have a look here:
https://github.com/sahlberg/pop-fe/blob/master/db.py

There is still about ~1000 entries that I need to manually reconcile but the idea is to have a database that consists of every disc image.
All the redump ones, that are the golden standard, for disks that exist in physical form but also every fan translation, every homebrew, every unreleased games, ...
But right now it has ~10.000 entries populated.

It is slow progress but I am getting there, at some stage, when it is more complete, I will start using it from pop-fe as the primary way to identify disks, falling back to the old/current detection if it fails.
This will also allow me to do things like showing a dialog when a game image has the "wrong" md5sum.

Like "You are trying to convert SLUS12345 but the disc image does not match the expected md5. Bad dump?"

That's exactly why I ignore sync header/EDC/ECC and only look at user data (executable code/real data) in find n replace sections. I also 'chain' user sector data together in case matches would spill over at the end of one sector into another. It's still possible to find things like this with lib-enigma but yea with libcrypt patcher I do the PPF database.

Keep me updated on the work your doing, great to hear updates.

 

[Ronnie Sahlberg]

Keep me updated on the work your doing, great to hear updates.

Sure.
db.py now contains 10981 entries that all map to a disc id. That should be virtually every official and unofficial disk ever released (and then some things that were never released as physical disk at all).

Discs that are known to not have an official number (such as homebrew, all the cheat discs, lots and lots of russian and taiwaneese releases...) are flagged as 'UNLDxxxxx" ~~ UNreLeaseD or UNLicenseD. The best name I could come up with

Ureleased games and homebrew and such, like these:

'aefa8f53118ce3ec2e9a10e7fba93a1e': {'id': 'UNLD00010', 'title': 'Resident Evil 0 Demake', 'vct': b'2023060117580932\x00'},

'001869ae4f620b7229fbee32fc2b77ee': {'id': 'UNLD00012', 'title': 'SOTN Editor', 'vct': b'2022031302485900\xe8'},

79 Fan translations. They are mapped to the same disc id as the original :

'10b44bffbbbc19fc43936389c65a5d40': {'id': 'SCPS10039', 'title': 'Linda^3 Again (English) (T)', 'vct': b'1997091710100000\x00'},

These are far far from all fan translations but should be (Almost) all English ones. I know there are a bunch more, especially Spanish and Portugeese, but I will have to add them over time, or rely on people to contribute these entries.


I have switched pop-fe over to use this new db.py to identify the discs. I will keep the old database but it will be delegated to only
map from a disc-id to game assets, such as icon, background, link to the manuals, ...

I.e. existing database now becomes just a database for the additional assets for a game, which everyone that don't care can just ignore.

And the new db.py is used for identification and possibly also additional game metadata. This is the one that others might find useful and maybe yourself will too.
This file will ONLY contain a single python dict that maps the md5sum of the first track into a disc-id, the title, and what other metadata might be useful.
Sure, it is a python dict but that can just be considered a fancy type of markup. For other consumers it is easy to write a small script that will extract and transform the required data into any other format.

Please have a look. I think this will be quite useful for the handful of people still out there that works with ps1 disk images as a hobby.
https://github.com/sahlberg/pop-fe/blob/master/db.py


Some interesting things I notice is how the three regions use such vastly different processes/policies for how to handle disc-ids.
In NTSC-J for example, it seems quite common that they don't really put a lot of effort into naming the executable in SYSTEM.CNF, or the disc volume id based on the serial.
On the other hand they seem to be VERY careful to always give every different disc a unique id. For example a game that was properly released but also had two different betas/demos, those three discs almost always have different id's assigned.

NTSC-U on the other hand seems to be very careful and almost always put a name based on the serial in SYSTEM.CNF but when it comes to original release, revisions, demos, beta discs they often all have the same id.
Thus you can't reliably depend on the disc id to uniquely identify an NTSC-U disc.

PAL on the other hand seems to be in the middle. Sometimes doing it the NTSC-U way, other times doing it the NTSC-J way ...

Not important at all, just found it interesting. I would have assumed that this would have been part of the process to create the master at the factory and thus it would be very consistent, but apparently it is not.


Oh, I found one NTSC-U game where it looks like they have the wrong id in SYSTEM.CNF, lol
# urban chaos has wrong discid in system.cnf
'573d7c4f54b9b934e6741d5733c9e611': {
'id': 'SLUS01091',
},

EDIT: I consider db.py probably in a state now that it is viable and ready to be used by external projects if they need an identification database.

 

[Ronnie Sahlberg]

A simple idea of a useful utility one could build with this.
The database contains pretty much every single disc ever released for PS1, officially or unofficially (and then some more).

Maybe a small utility to check if a dump is good or not.
Example: you rip a diskto cue/bin then you run this utility to check if it is good or bad.
If the md5 of the first track is found in the database then the rip is good. If it is not found then the rip is bad and you should clean the disk and try ripping it again.

 

[alexfree]

A simple idea of a useful utility one could build with this.
The database contains pretty much every single disc ever released for PS1, officially or unofficially (and then some more).

Maybe a small utility to check if a dump is good or not.
Example: you rip a diskto cue/bin then you run this utility to check if it is good or bad.
If the md5 of the first track is found in the database then the rip is good. If it is not found then the rip is bad and you should clean the disk and try ripping it again.

With your database I'm sure you could generate a list of official PSX.EXE games which don't have multiple revisions and have serials starting with SLPS or SCPS. My thinking is I can speed run adding support for more japanese PSX.EXE titles to lib-enigma like this:

1) Get all lines of all games containing PSX.EXE, i.e.

'c2b8b1652407c6c8107b0c93e20624a6': {'id': 'SLPS00017', 'title': "King's Field (Japan)", 'vct': b'1994111013000000$'},
'13223ebd9b33c48aa0620e5863de071f': {'id': 'SCPS10008', 'title': 'Arc the Lad (Japan)'},
'8c8ad4e1c182fc986ab6ff691e6f89d8': {'id': 'SLPS00001', 'title': 'Ridge Racer (Japan)', 'vct': b'1994111009000000$'},

2) Use each line to create output in the format of (including the empty lines, 4 space tab for the (strcmp line) then 8 space tab from 0 pos for the return "XXX.XX" line, then 4 space tabe for else if line, love C liberal formatting):

(strcmp( (char *)volume_creation_timestamp, "1994111013000000") == 0) {
return "000.17";
} else if

(strcmp( (char *)volume_creation_timestamp, "1995052420065100") == 0) {
is_scps = true;
return "100.08";
} else if

(strcmp( (char *)volume_creation_timestamp, "1994111009000000") == 0) {
return "000.01";
} else if

(This site nerfs formatting, I am looking for https://github.com/alex-free/lib-en...0d1236cb813aa3159931251ac84/lib-enigma.c#L122)

I know it's possible, and it would pretty much save me from running all games with my volumecreationtimestamp identifier. Would need to manually go through any games with revisions later...

Besides some variable replacement, you'd need to know if you need to add the is_scps = true; line or not depending on if the bootfile is scps or slps. Again why this is so weird is because it's exactly the code for PSX.EXE Memcardpro gameid support that I wrote in Tonyhax International, and trust me having a bool flag set if you need to add the text "scps" or "slps" to the serial saves a TON of space on a real PS1 which is targeting sub 64KB entire executable size for fake chinese memory cards that have half the space of real ones. And If I can I think it's just easier to keep both code bases the same weird format, so if I need add some game I already figured out in lib-enigma to tonyhax international (which I'd only do if it had save support, to again keep exe size as small as possible for so many reasons) or vice versa it's just copy paste. Plus I want to keep my library as small as possible on PC anyways.

I genuinely hate asking for things, but since you know python and I don't maybe this would be really easy for you so worth a shot anyways. If not I will try with bash/sed/find foo I've done things like this before. I'm. not opposed to non-official PSX.EXE games either, but the library really is just designed for PSX.EXE japanese SCPS or SLPS currently (the reason SLPS is the default is because way more games were released without SYSTEM.CNF as SonyLicensedProductSoftware, and again memory reasons) due to the reason I wrote it in the first place so it can't really yet handle more then just a XXX.XX and bool line or no bool atm.

 

[Ronnie Sahlberg]

With your database I'm sure you could generate a list of official PSX.EXE games which don't have multiple revisions and have serials starting with SLPS or SCPS. My thinking is I can speed run adding support for more japanese PSX.EXE titles to lib-enigma like this:

1) Get all lines of all games containing PSX.EXE, i.e.

'c2b8b1652407c6c8107b0c93e20624a6': {'id': 'SLPS00017', 'title': "King's Field (Japan)", 'vct': b'1994111013000000$'},
'13223ebd9b33c48aa0620e5863de071f': {'id': 'SCPS10008', 'title': 'Arc the Lad (Japan)'},
'8c8ad4e1c182fc986ab6ff691e6f89d8': {'id': 'SLPS00001', 'title': 'Ridge Racer (Japan)', 'vct': b'1994111009000000$'},

2) Use each line to create output in the format of (including the empty lines, 4 space tab for the (strcmp line) then 8 space tab from 0 pos for the return "XXX.XX" line, then 4 space tabe for else if line, love C liberal formatting):

(strcmp( (char *)volume_creation_timestamp, "1994111013000000") == 0) {
return "000.17";
} else if

(strcmp( (char *)volume_creation_timestamp, "1995052420065100") == 0) {
is_scps = true;
return "100.08";
} else if

(strcmp( (char *)volume_creation_timestamp, "1994111009000000") == 0) {
return "000.01";
} else if

(This site nerfs formatting, I am looking for https://github.com/alex-free/lib-en...0d1236cb813aa3159931251ac84/lib-enigma.c#L122)

I know it's possible, and it would pretty much save me from running all games with my volumecreationtimestamp identifier. Would need to manually go through any games with revisions later...

Besides some variable replacement, you'd need to know if you need to add the is_scps = true; line or not depending on if the bootfile is scps or slps. Again why this is so weird is because it's exactly the code for PSX.EXE Memcardpro gameid support that I wrote in Tonyhax International, and trust me having a bool flag set if you need to add the text "scps" or "slps" to the serial saves a TON of space on a real PS1 which is targeting sub 64KB entire executable size for fake chinese memory cards that have half the space of real ones. And If I can I think it's just easier to keep both code bases the same weird format, so if I need add some game I already figured out in lib-enigma to tonyhax international (which I'd only do if it had save support, to again keep exe size as small as possible for so many reasons) or vice versa it's just copy paste. Plus I want to keep my library as small as possible on PC anyways.

I genuinely hate asking for things, but since you know python and I don't maybe this would be really easy for you so worth a shot anyways. If not I will try with bash/sed/find foo I've done things like this before. I'm. not opposed to non-official PSX.EXE games either, but the library really is just designed for PSX.EXE japanese SCPS or SLPS currently (the reason SLPS is the default is because way more games were released without SYSTEM.CNF as SonyLicensedProductSoftware, and again memory reasons) due to the reason I wrote it in the first place so it can't really yet handle more then just a XXX.XX and bool line or no bool atm.

Let me try, is this what you want :
#!/usr/bin/env python
# coding: utf-8
#
import db
for k in db.disc_by_md5:
id = db.disc_by_md5[k]['id']
if id[:4] not in ['SLPS', 'SCPS']:
continue
if 'vct' not in db.disc_by_md5[k]:
continue
print(' (strcmp((char *)volume_creation_timestamp, "%s") == 0) {' % db.disc_by_md5[k]['vct'][:16].decode('utf-8'))
if db.disc_by_md5[k]['id'][:4] == 'SCPS':
print(' is_scps = true;')
print(' return "%s.%s";' % (id[4:7], id[7:9]))
print(' } else if')
print()

(Uploaded it as a.txt, just rename to a.py)

This prints output like this :
(strcmp((char *)volume_creation_timestamp, "2001070400000000") == 0) {
return "032.37";
} else if
(strcmp((char *)volume_creation_timestamp, "1999110110424304") == 0) {
is_scps = true;
return "100.93";
} else if
(strcmp((char *)volume_creation_timestamp, "2001083101445600") == 0) {
return "032.68";
} else if

For every entry with a name SLPSxxxxx or SCPSxxxxx
if I know the volume creation timestamp. I do not know it for every disk so please send me the ones you want me to add and I will add them.

I uploaded the output as out.txt. It contains 3349 entries. I guess you don;t need all of them but can just cut'n'paste the ones you need ?


Let me know what else I can do to help. Or if there are other things you think I should add to the database.

 

[Ronnie Sahlberg]

There are 68 SLPS / SCPS disks I do not have the volume creation timestamp for

I have attached the list as a file

I don't know. Maybe these lists are not too useful?
Because I don't know which games use PSX.EXE and which ones have a proper name for the executable.
And reading your post again you are asking for just the PSX.EXE ones, not all of them ?

Do you have a list of which disks/games name the binary PSX.EXE ?
Otherwise I will have to try to figure something out.

 

[Ronnie Sahlberg]

Ok, I redid it and added to the database a tag IF psx.exe / PSX.EXE is referenced.It is probably not 100% accurate
but likely accurate enough.
190 entries for SCPL/SLPS, has PSX.EXE and also a known volume creation timestamp/

a.txt is the python program to extract it, out.txt is the 190 entries found.