PS3 Thumbs.db files in game disc, mistake or copy protection?

[AlexRhine]

*edit* Seems like its windows is causing these Thumb.db files to go missing when building iso on pc, which then causes the game data install to fail.

Oh, so there is a correlation! Just to be sure, could you check if the folder game you copied has those files? (just want to make sure it's not a problem with makeps3iso or iso mounting). If possible (if they are in the folder game you're using), try deleting them, delete the game data, and boot up the game. Does it fail?

Now I wonder (assuming the ISO isn't the problem), were those Thumbs.db repurposed for the game, or is it a copy protection? If they are the same as the Thumbs.db files from windows, you might be able to extract the image they contain (though, of you're using a Windows machine, you might need to check the md5sum of the file to make sure it isn't overwritten). There is a tutorial online on how to show them in file manager (though, show hidden files might do the trick).

 

[DonKeballs]

Oh, so there is a correlation! Just to be sure, could you check if the folder game you copied has those files? (just want to make sure it's not a problem with makeps3iso or iso mounting). If possible (if they are in the folder game you're using), try deleting them, delete the game data, and boot up the game. Does it fail?

Now I wonder (assuming the ISO isn't the problem), were those Thumbs.db repurposed for the game, or is it a copy protection? If they are the same as the Thumbs.db files from windows, you might be able to extract the image they contain (though, of you're using a Windows machine, you might need to check the md5sum of the file to make sure it isn't overwritten). There is a tutorial online on how to show them in file manager (though, show hidden files might do the trick).

Ok so I've deleted the game data and deleted those Thumb.db files and then tried to reinstall and it's getting stuck at 53% again.

I agree with you, It's difficult to know whether it's windows or makeps3iso at fault.

Not quite sure exactly what you wanted me to do with regards to extraction the image those files contain but I have attached them to this post so maybe you can take a look at them.

 

[sandungas]

Please @DonKeballs if you install the gamedata again (successfully) try to see if the Thumb.db files has been copyed to the gamedata installation path
This would explain why the gamedata installation fails without them... because when we perform a gamedata installation the game needs to read a "list of files" that needs to be copyed

If the Thumbs.db are in that list (but we delete them) the gamedata installation cant be completed, it stops trying to find them

It sometimes happening like i.e executables with debug symbols or readme from sdk. In example on Halo 2 in "media" folder there is file called "command line.txt" and inside:

cd C:\Program Files\Microsoft Xbox SDK\xbox\bin

bundler C:\halo2\bin\Downloader\media\dlcustom.rdf -o C:\halo2\bin\Downloader\media\dlcustom.xpr
bundler C:\halo2\bin\Downloader\media\dlres.rdf -o C:\halo2\bin\Downloader\media\dlres.xpr

Dlcustom holds the resources for the custom backgrounds. Dlres is the resources for all the buttons and controls. If dlres.xpr is absent then downloader will simply reference the default resources embedded in the .xbe. You must have the sdk installed in the default location for the command lines to work.

Well, this is a bit more funny because i guess the tools used to create the master discs does some checks for specific file names that are a frequent problem and Thumbs.db should be on top of that list, lol

I mean... if we create file named "notes.txt" maybe is going to bypass that check because the name is random so the tools could consider is a game file, but other names like "Thumbs.db" or "New Text Document.txt" are easy catchs

 

[AlexRhine]

If the Thumbs.db are in that list (but we delete them) the gamedata installation cant be completed, it stops trying to find them

I wonder if someone in the development team created the game installation list with those Thumbs.db files included, and then during debugging someone else found out the game wouldn't install properly unless they manually included the files on the game disc. That would be pretty funny, actually.

Anyway, I guess the next question is how we could fix ISO creation for all games with a similar problem on Windows.

We could probably force makeps3iso to read Windows system files, but this would also mean that other game ISOs will get junk data added to them by default.

We could include a manual patch for this game in the source code of makeps3iso, but then other games with a similar problem would have to be fixed on a case by case basis.

The best solution would probably be to add a parameter to makeps3iso to allow the user to choose to include Windows system/hidden files. That way, knowledge of the problem would have to be learned online, like the problem with Assassin's Creed games and their empty ASASSIN_ISO folder.

The next question would be how to make this patch. Makeps3iso uses the dirent.h file to read files while building the iso. I'm guessing the Windows port of that library ignores system files by default. I wonder if there's a way to force these files to be read.

BTW, *nix users shouldn't have a problem building the ISO of the game, since Thumbs.db files mean nothing special to those systems.

 

[sandungas]

I wonder if someone in the development team created the game installation list with those Thumbs.db files included, and then during debugging someone else found out the game wouldn't install properly unless they manually included the files on the game disc. That would be pretty funny, actually.

Thats my bet at this point, waiting for the final confirmation from @DonKeballs if the thumb.db files are copyed to the gamedata installation path it would explain why they passed all that controls, lol
Should be considered "game files"... but just because his names are included in the gamedata installation list, the game only copies them but probably doesnt loads them ever

I mean... im not sure the exact steps in the production of this blurays, but im assuming there are at least 3 steps in the chain, from game company to sony for review to the manufacturer plants
And in every one of that steps probably there are going to be several persons taking a look at the game files

Anyway, I guess the next question is how we could fix ISO creation for all games with a similar problem on Windows.

We could probably force makeps3iso to read Windows system files, but this would also mean that other game ISOs will get junk data added to them by default.

We could include a manual patch for this game in the source code of makeps3iso, but then other games with a similar problem would have to be fixed on a case by case basis.

The best solution would probably be to add a parameter to makeps3iso to allow the user to choose to include Windows system/hidden files. That way, knowledge of the problem would have to be learned online, like the problem with Assassin's Creed games and their empty ASASSIN_ISO folder.

The next question would be how to make this patch. Makeps3iso uses the dirent.h file to read files while building the iso. I'm guessing the Windows port of that library ignores system files by default. I wonder if there's a way to force these files to be read.

BTW, *nix users shouldn't have a problem building the ISO of the game, since Thumbs.db files mean nothing special to those systems.

Hmm, i think adding a whitelist with the paths could work

PS3_GAME\USRDIR\Data_SCCT\System\Resources\deu\Thumbs.db
PS3_GAME\USRDIR\Data_SCCT\System\Resources\esp\Thumbs.db
PS3_GAME\USRDIR\Data_SCCT\System\Resources\fra\Thumbs.db
PS3_GAME\USRDIR\Data_SCCT\System\Resources\int\Thumbs.db
PS3_GAME\USRDIR\Data_SCCT\System\Resources\Thumbs.db

I mean... instead of using his hashes we can use his paths, this way is a check specific for this game but made in a easy way
His hashes doesnt matters btw... i bet you can replace them by 0kb dummies in the ISO and the gamedata installation will accept them gracefully because the gamedata installation doesnt uses to do hash checks. I personally made this in tenths of games

 

[DonKeballs]

Thats my bet at this point, waiting for the final confirmation from @DonKeballs if the thumb.db files are copyed to the gamedata installation path it would explain why they passed all that controls, lol
Should be considered "game files"... but just because his names are included in the gamedata installation list, the game only copies them but probably doesnt loads them ever

I mean... im not sure the exact steps in the production of this blurays, but im assuming there are at least 3 steps in the chain, from game company to sony for review to the manufacturer plants
And in every one of that steps probably there are going to be several persons taking a look at the game files


Hmm, i think adding a whitelist with the paths could work

PS3_GAME\USRDIR\Data_SCCT\System\Resources\deu\Thumbs.db
PS3_GAME\USRDIR\Data_SCCT\System\Resources\esp\Thumbs.db
PS3_GAME\USRDIR\Data_SCCT\System\Resources\fra\Thumbs.db
PS3_GAME\USRDIR\Data_SCCT\System\Resources\int\Thumbs.db
PS3_GAME\USRDIR\Data_SCCT\System\Resources\Thumbs.db

I mean... instead of using his hashes we can use his paths, this way is a check specific for this game but made in a easy way
His hashes doesnt matters btw... i bet you can replace them by 0kb dummies in the ISO and the gamedata installation will accept them gracefully because the gamedata installation doesnt uses to do hash checks. I personally made this in tenths of games

Currently away @sandungas Will check the game data installation path for these Thumbs.db files and report back this week sometime.

 

[AlexRhine]

Hmm, i think adding a whitelist with the paths could work

Yes, a separate whitelist/patch_file which you could input into the program sound like a good idea. Like a list of all games which are known to have problematic "unneeded" files and empty folder. I would suggest something like:

[BLUS-30761] {
FILE {
PS3_GAME/USRDIR/Data_SCCT/System/Resources/deu/Thumbs.db
PS3_GAME/USRDIR/Data_SCCT/System/Resources/esp/Thumbs.db
PS3_GAME/USRDIR/Data_SCCT/System/Resources/fra/Thumbs.db
PS3_GAME/USRDIR/Data_SCCT/System/Resources/int/Thumbs.db
PS3_GAME/USRDIR/Data_SCCT/System/Resources/Thumbs.db
}
FOLDER {}
}

[BLUS-30089] {
FILE {}
FOLDER {
PS3_GAME/USRDIR/ASSASSIN_ISO
}
}

Or similar, where each file/folder would be empty (for files, 0kb size). I want to include empty folders because I know that older versions of PS3 Disc Dumper did not include these empty folders (in fact, I think the version that is posted on the resource page is one of those older versions).

 

[DonKeballs]

Thats my bet at this point, waiting for the final confirmation from @DonKeballs if the thumb.db files are copyed to the gamedata installation path it would explain why they passed all that controls, lol
Should be considered "game files"... but just because his names are included in the gamedata installation list, the game only copies them but probably doesnt loads them ever

I mean... im not sure the exact steps in the production of this blurays, but im assuming there are at least 3 steps in the chain, from game company to sony for review to the manufacturer plants
And in every one of that steps probably there are going to be several persons taking a look at the game files


Hmm, i think adding a whitelist with the paths could work

PS3_GAME\USRDIR\Data_SCCT\System\Resources\deu\Thumbs.db
PS3_GAME\USRDIR\Data_SCCT\System\Resources\esp\Thumbs.db
PS3_GAME\USRDIR\Data_SCCT\System\Resources\fra\Thumbs.db
PS3_GAME\USRDIR\Data_SCCT\System\Resources\int\Thumbs.db
PS3_GAME\USRDIR\Data_SCCT\System\Resources\Thumbs.db

I mean... instead of using his hashes we can use his paths, this way is a check specific for this game but made in a easy way
His hashes doesnt matters btw... i bet you can replace them by 0kb dummies in the ISO and the gamedata installation will accept them gracefully because the gamedata installation doesnt uses to do hash checks. I personally made this in tenths of games

@sandungas Just checked and can confirm that the thumb.db files have been copied to the game data installation path.

 

[sandungas]

@sandungas Just checked and can confirm that the thumb.db files have been copied to the game data installation path.

Nice, just incase someone had a doubt about it, can be considered "game files", his presence is critical for the gamedata installation... and yeah the IRD was legit, btw i dont think we are going to find more games with this problem, this is a rarity, maybe unique (and thats why i suggested that makeps3iso could use a whitelist, because probably is going to be very short)

Yes, a separate whitelist/patch_file which you could input into the program sound like a good idea. Like a list of all games which are known to have problematic "unneeded" files and empty folder. I would suggest something like:

[BLUS-30761] {
FILE {
PS3_GAME/USRDIR/Data_SCCT/System/Resources/deu/Thumbs.db
PS3_GAME/USRDIR/Data_SCCT/System/Resources/esp/Thumbs.db
PS3_GAME/USRDIR/Data_SCCT/System/Resources/fra/Thumbs.db
PS3_GAME/USRDIR/Data_SCCT/System/Resources/int/Thumbs.db
PS3_GAME/USRDIR/Data_SCCT/System/Resources/Thumbs.db
}
FOLDER {}
}

[BLUS-30089] {
FILE {}
FOLDER {
PS3_GAME/USRDIR/ASSASSIN_ISO
}
}

Or similar, where each file/folder would be empty (for files, 0kb size). I want to include empty folders because I know that older versions of PS3 Disc Dumper did not include these empty folders (in fact, I think the version that is posted on the resource page is one of those older versions).

When i suggested the whitelist i was thinking that the TITLE_ID could be innacurate (incase someone try to create an ISO with a modifyed EBOOT.BIN and PARAM.SFO to make the game compatible with a DLC or a savegame from other region, etc...), and i was also thinking in the ASSSASSIN_ISO but i was not sure the best way to do this kind of checks

Btw @bucanero the makeps3iso in your github (or the original version) are doing some check to delete thumbs.db or any other problematic file ?

 

[AlexRhine]

Btw @bucanero the makeps3iso in your github (or the original version) are doing some check to delete thumbs.db or any other problematic file ?

I've looked at the source code, and there doesn't seem to be any patches for specific games. Btw, if we're to modify makeps3iso, it would probably be better to make those changes with @Zar's fork of makeps3iso, since it includes some major patches for ODE consoles: https://github.com/Zarh/ps3iso-utils (Ignore the v1.49 in the readme; I meant to write v1.9)

When i suggested the whitelist i was thinking that the TITLE_ID could be innacurate (incase someone try to create an ISO with a modifyed EBOOT.BIN and PARAM.SFO to make the game compatible with a DLC or a savegame from other region, etc...)

Hmm, I think people with modified eboots/title_ids will have to edit the whitelist manually to support their modified game. I have no idea how else a game could be recognized in a fast way (makeps3iso already extracts the game-id from the PARAM.SFO, so I was thinking on just using that).

 

[sandungas]

I've looked at the source code, and there doesn't seem to be any patches for specific games. Btw, if we're to modify makeps3iso, it would probably be better to make those changes with @Zar's fork of makeps3iso, since it includes some major patches for ODE consoles: https://github.com/Zarh/ps3iso-utils (Ignore the v1.49 in the readme; I meant to write v1.9)

Ohh i missed that fork made by zar, yep i guess this kind of checks fits better in it, actually it seems it already have a function to scan filenames PS3GAME.*** from multiman and skips them

What we was discussing is a bit the other way around, we need to scan the path/file names and based in his presence either:
1) create empty folder
...or...
2) create dummy file (only if is missing)

Hmm, I think people with modified eboots/title_ids will have to edit the whitelist manually to support their modified game. I have no idea how else a game could be recognized in a fast way (makeps3iso already extracts the game-id from the PARAM.SFO, so I was thinking on just using that).

Not sure, i was just thinking loud trying to do it in a failproof way
For the splinter cell trilogy using the path seems to be failproof because the path names seems to be unique
For the ASSASSIN_ISO folder the path in worthless though... is too much generic, by reading the path we dont know if the game is an assassin's creed

 

[AlexRhine]

Not sure, i was just thinking loud trying to do it in a failproof way
For the splinter cell trilogy using the path seems to be failproof because the names seems very specific
For the ASSASSIN_ISO folder the path in worthless though... is too much generic, by reading the path we dont know if the game is an assassin's creed

SPA:
Ah, ya entiendo lo que dices. Quieres usar la presencia de esos archivos en el juego para detectar que estamos lidiando con el juego en cuestion. Pero seria un problema si los archivos/folders no existen o estan oculto. Lo que yo proponia era un tanto diferente: Hacer que si makeps3iso detecta que esos archivos no estan, o estan ocultos por el sistema operativo, que cree unos vacios para que la instalacion/boot-up no falle; lo mismo para los juegos de Assassin's Creed. Como supongo que esos archivos/folders no van a existit, quiero usar algo como el game-id en el PARAM.SFO para que el programa pueda "estar al tanto" de crear esos archivos si no existen.
EN:
Ah, I understand what you say. You want to use the presence of those files in the game to detect that we are dealing with the game in question. What I proposed was a bit different: If makeps3iso detects that these files are not, or are hidden by the operating system, that it creates some gaps so that the installation / boot-up does not fail; same for Assassin's Creed games. Since I assume that those files are not going to be, I want to use something like the game-id in the PARAM.SFO so that the program can "be aware" of creating those files if they do not exist. (Sorry for the somewhat odd English translation, I didn't want translate by hand).

 

[sandungas]

SPA:
Ah, ya entiendo lo que dices. Quieres usar la presencia de esos archivos en el juego para detectar que estamos lidiando con el juego en cuestion. Pero seria un problema si los archivos/folders no existen o estan oculto. Lo que yo proponia era un tanto diferente: Hacer que si makeps3iso detecta que esos archivos no estan, o estan ocultos por el sistema operativo, que cree unos vacios para que la instalacion/boot-up no falle; lo mismo para los juegos de Assassin's Creed. Como supongo que esos archivos/folders no van a existit, quiero usar algo como el game-id en el PARAM.SFO para que el programa pueda "estar al tanto" de crear esos archivos si no existen.
EN:
Ah, I understand what you say. You want to use the presence of those files in the game to detect that we are dealing with the game in question. What I proposed was a bit different: If makeps3iso detects that these files are not, or are hidden by the operating system, that it creates some gaps so that the installation / boot-up does not fail; same for Assassin's Creed games. Since I assume that those files are not going to be, I want to use something like the game-id in the PARAM.SFO so that the program can "be aware" of creating those files if they do not exist. (Sorry for the somewhat odd English translation, I didn't want translate by hand).

Nice to know you speak spanish hehe, yes the first idea that came to mind was to identify the game based in a path, for the splinter cell should work because the path where the thumbs.db are located probably are unique (only this game have them)... but... it would be needed to check if there are other versions of this game with the same paths but without the thumb.db files (maybe it was re-released few years later again and the publisher removed them)

But for the assassin's creeds... checking the path of the ASSASSIN_ISO folder is pointless
But there could be different ways of how to implement this, right now (as something conceptual) im thinkinig in being composed by 2 values... a "condition" and a "result" and both using paths

For the assassin's creed the condition could be the path of any other of the game files... i dont remember right now the exact names used in them, but i remember are packed in a similar way (same game engine), and there is a directory with veeeeery explicit filenames, so basically:

If condition is satisfyed (one of the paths from whitelist for the assassing creed) check if the ASSASSIN_ISO folder exists (at a different path than the condition path)... and if it doesnt exists create it
And for the splinter cell with the thumbs.db pretty muc the same... but using the same path in both, the "condition" and the "result" and creating thumbs.db dummy files of 0kb size

Not sure, maybe is too much weird, im trying to avoid checking the TITLE_ID but maybe is better to do it

 

[AlexRhine]

the condition could be the path of any other of the game files

I was initally thinking of using a variation of one of Zar's signatures for IRDs (I think it's called "IRD_FILES_SIG") which uses the md5sums of all files to get a value. Though, I discarded the idea because it would take too long to generate, and it would be too hard to decide which files to ignore.

It's a really good idea to pick a supposedly unchanging file path to detect a game, but perhaps a bit too complicated for the problem at hand? If any other games with problems appear, there would have to be a discussion as to what file path to use for detection. I think a comment in the whitelist like "Modified game ids in PARAM.SFO may need to set the game id manually" would work. Plus, I'm thinking of the whitelist being independent of the program, so anyone who doesn't want to deal with it for any reason can choose to not use it.

 

[Zar]

FYI, the issue with "ASSASSIN_ISO" folder missing come from genps3iso, it has several bug/issue, I do not trust it at all anymore, that's why I made a mod of makeps3iso.

 

[bucanero]

Btw @bucanero the makeps3iso in your github (or the original version) are doing some check to delete thumbs.db or any other problematic file ?

no, I didn't add any filter or check to the makeps3iso tool. The only thing that my build removes should be the update PUP file to save some space, that's all.

from the original code, I remember that Estwald merged the .666 split files back into a single one, but I think that's all.

btw, about weird files like Thumbs.db , I usually check the file timestamp. Original files from the disc should have the same date/time., and usually foreign files (like the ones from multiman) have a totally different date

 

[sandungas]

The bigfiles can be identifyed by his paths too, when we split a file we delete the original... so his presence (or better said, the absence ?) can be used indirectly to know if the game have splitted files

There are only a few games that are an exception to this rule (uncharteds, the last of us) that uses PSARC format and the game works if you replace the PSARC by his contents extracted... but other than that i think the rule applyes to all other games, when we create the ISO is needed to restore the "bigfiles" in 99,9% of the games

Im thinking in a relativelly easy way to do it, but is a bit overkill
1) Write an small PC utility to read and calculate all the file sizes from all the IRD's from jonny/zar databases (the whole PS3 game library)
2) Make a list with all the paths/names of the files bigger than 4GB (candidates to be splitted by the user)
3) Use that list to check if the path/filename of the bigfile exists... if it exists thats fine... otherway it means is splitted (or missing)

But... i guess this needs to check the TITLE_ID because there are going to be a lot of bigfiles located at root of USRDIR (sharing path... and eventually some could be sharing the file name too incase are 2 games from the same game company)
Or... well... it could be identifyed by using an alternative path of other file