Hi there,
I found this thread while looking for ways to do something similar to what you folks are talking about. In particular, I'd like to add more songs to the game (the ones from my PS2 discs and maybe others).
While I have reverse engineering experience I have zero experience with PS3 specifics, so I was hoping somebody can point me in the right direction.
I've checked what
@Berion said about the files being EDAT and unless I'm mistaken I don't think they are. According to this page
https://www.psdevwiki.com/ps3/EDAT_files said files have a header with a lot of information about the encrypted file, but those `pkg.drm` are encrypted from the very first byte (I ran a quick analysis of the entropy of the file and I'm mostly certain it's just encrypted/compressed from the beginning). MSELF files are executables as far as I can see, so they should also be ruled out because of the lack of header (again, no experience on this system, so please correct me if I'm wrong).
My suspicion is that they might be encrypted using some key unique to the game (and maybe even found inside the game itself) but one missing piece of information is: are those files "generic", unique to each user, to user + console or something else? I thought that one quick way of finding out is by getting the same file (or just a hash, should be enough) from different people that belong to the same song, so please let me know if anybody can help with that, unless somebody already knows this is useless

. This will actually only tell us if files are unique or not (but not in what way) but it's a start

.
After decription the PKG format itself is easy to parse, I'm not sure if it's the same as the PS2 but even if it's not, I doubt it will be that complicated.