PS3 [Research] Modifying the Gaia Visualization (custom_render_plugin/earth.qrc)

Discussion in 'XMB Modifications' started by DeViL303, Jan 9, 2020.

  1. 1,820
    1,971
    297
    pink1

    pink1 Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    1,820
    Likes Received:
    1,971
    Trophy Points:
    297
    Gender:
    Male
    For the cubemap generator I think we can just add the files and a web browser pointed to it. It's kind of a cheap way of doing it for now anyway.

    I'm happy to try and help get it figured out if you wanna play along. I think for a test we should try to find something close to the end & edit it's size that way we don't have to change a lot of the address, less chance of messing it up lol
     
    DeViL303 likes this.
  2. 8,066
    9,294
    797
    DeViL303

    DeViL303 Developer PSX-Place Supporter

    Joined:
    Jan 23, 2016
    Messages:
    8,066
    Likes Received:
    9,294
    Trophy Points:
    797
    Yeah the cube map generator is not so important really, The site is so easy to use.

    Yeah for sure.The last jpg is at :

    upload_2020-1-22_0-11-37.png

    This would be a candidate.

    This 18.1KB file is there normally in OFW. In my template there is a 1KB black image instead

    24.jpg



    Or there is a text file at the very end of the end if that would be easier.

    I gotta go..back tomorrow.
     
    Last edited: Jan 21, 2020
  3. 1,820
    1,971
    297
    pink1

    pink1 Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    1,820
    Likes Received:
    1,971
    Trophy Points:
    297
    Gender:
    Male
    So for the last jpg we would take 295BA0 - E5F0 = 2875B0. We search for that in the file table and the next bytes are the size.
    So if we make it bigger we would need to edit the address of the rest by adding the difference of the size of the jpg.

    We can also find the addresses we need to edit with the python tool @sandungas posted.
     
    DeViL303 likes this.
  4. 6,746
    6,379
    622
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    6,746
    Likes Received:
    6,379
    Trophy Points:
    622
    Location:
    Babylon 20xxE series
    Btw, there is an easy way to find the start offsets of all the files, it just needs to be made a simple sum math operation, and a trick in hexeditor

    1) Extract the contents of the .qrc with the python CXML decompiler tool

    2) Open the xml file generated by CXML decompiler, it looks like this:
    Code:
    <?xml version="1.0" encoding="utf-8"?>
    <qrc>
    	<file-table>
    		<file src="Offset=0x00000000, Length=0x0000025A.bin" id="earth/ATM.mnu" />
    		...
    		...
    		...
    	</file-table>
    </qrc>
    
    Thats the first file, located in the first position of the file table, this is why it have an Offset=0x00000000
    Is an offset relative to the start offset of the filetable
    In other words... the start offset of the first file is the same than the start offset of the filetable :)

    5) So... now we need to find the absolute offset of the filetable
    Open in the hexeditor earth.qrc and the file named "Offset=0x00000000, Length=0x0000025A.bin" (because we know this is the first file)
    In the file... press keys CTRL + A (to select all), then CTRL + C (to copy)
    In the .qrc... press CTRL + F (to search)... it opens a settings window, click in the tab "search for hex vaues" and press CTRL + V (to paste)... then click "OK"

    6) So... now you have highlighted the first file in the file table entirelly
    If your goal is to replace it select the tab [EDIT] -> [FILL SELECTION] and fill the selected area with zeroes
    This cleans the garbage bytes... also it changes the color of the edited bytes to red (is very easy to identify where the file started and ended)

    7) Ok, so now is time to use the math formula :D
    Scroll up to the first byte marked in red... in this case for earth.qrc 4.85 the first file was located at offset 0xE5F0
    And as mentioned before... the first file matches exactly with the start offset of the filetable... so now by looking at the xml generated by the CXML decompiler pythong script we can calculate the absolute offset of all the files, this way
    Code:
    <?xml version="1.0" encoding="utf-8"?>
    <qrc>
    	<file-table>
    		<file src="Offset=0x00000000, Length=0x0000025A.bin" id="earth/ATM.mnu" />
    		...
    		<file src="Offset=0x00044F20, Length=0x00000880.bin" id="lib/gaia/Sun.fpo" />
    		...
    		<file src="Offset=0x0019E0B0, Length=0x00002B38.bin" id="earth/flashrom/disk.dds" />
    		...
    		<file src="Offset=0x002B8A30, Length=0x0000057F.bin" id="earth/camera/views.txt" />
    	</file-table>
    </qrc>
    
    The formula is..
    File absolute offset = filetable absolute offset + file relative offset

    Some examples
    earth/ATM.mnu absolute offset = 0xE5F0 + 0x00000000 = 0xE5F0 (yeah, genius, something + 0 = something)
    lib/gaia/Sun.fpo absolute offset = 0xE5F0 + 0x00044F20 = 0x53510
    earth/flashrom/disk.dds absolute offset = 0xE5F0 + 0x0019E0B0 = 0x1AC6A0
    earth/camera/views.txt absolute offset = 0xE5F0 + 0x002B8A30 = 0x2C7020


    -----------------
    And remember, hexeditors like HxD have an option named "go to" (CTRL + G), you can use it with the values for the absolute offsets of all the files to "jump" into them easilly
    As example... in earth.qrc press CTRL + G and paste 1AC6A0 ... this is going to take you to the absolute start offset of disk.dds
     
    Last edited: Jan 21, 2020
    Berion, DeViL303 and pink1 like this.
  5. 6,746
    6,379
    622
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    6,746
    Likes Received:
    6,379
    Trophy Points:
    622
    Location:
    Babylon 20xxE series
    Yes, is very convenient, actually we have been doing it before in other frankentests (inside lines.qrc trying to replace a couple of files inside it named spline.elf and particles.elf)
    That 2 boys was specially picky in the experiments and was refusing to work, so in one of the tests i had to adjust well the file length values in the index "just incase"... but not luck, the test failed for other reasons

    Anyway... What we do to remap files inside the .qrc structure is to change that values of the index, is very straightforward, in the index you can see 2 values for every file for the "offset" and "length", with 4 bytes each, and consecutivelly
    This values are what you can see in the xml generated by the CXML decompiler, as example...
    Code:
    <?xml version="1.0" encoding="utf-8"?>
    <qrc>
    	<file-table>
    		<file src="Offset=0x00001860, Length=0x00000121.bin" id="earth/presets/0/earth/CLOUDS_4M.mnu" />
    	</file-table>
    </qrc>
    
    Copy together the 2 hex values that appears in the xml
    So the... Offset=0x00001860, Length=0x00000121.bin ... becomes ... 0x0000186000000121

    If you search for that in the .qrc this is going to highlight it in the index... the firmware reads that value and then it "jumps" to load the file ... by changing that values we cheat the firmware into loading a different file... or a file in the same offset but with a different size

    Loading different files is something we have been doing many times in the experiments with other .qrc files
    Changing the size works... but we have made it only to convert the .MNU files to dummies (reducing his size to 9 bytes in the index), to corrupt .MNU files (reducing them to zero), or the experiment i mentioned about spline.elf and particles.elf (because i was using .elf files from older firmwares and had smaller sizes)

    But the point is we have full freedom to change all that values of the index, and by doing this we can reorganize the contents of the whole filetable
    The restriction we have is the total size of the filetable, it can be calculated this way:
    Total size of the file table = end offset of the last file - start offset of the first file

    The reason why we cant change the total size of the filetable is because if we do it will be needed to rebuild other parts of the .QRC structure that are out of the filetable... and that could be tricky
    The good thing is the filetable is located at the end of the .QRC ...so incase of making it bigger there is not going to be other data located after the filetable displaced by it


    --------------
    And yeah... i mentioned it before... for this kind of mods replacing the textures of the earth most of the times the custom files are going to be smaller,, so is better to adjust his size in the index
    As example, with the value i mentioned before... 0000186000000121
    You need to search for that... and change the 121 (original size) by the real size of the custom file (that usually is going to be smaller)
     
    Last edited: Jan 21, 2020
    DeViL303 and Berion like this.
  6. 6,746
    6,379
    622
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    6,746
    Likes Received:
    6,379
    Trophy Points:
    622
    Location:
    Babylon 20xxE series
    The tools you are using to create the cubemaps are not confirmed to work for what we need btw
    When talking about this is needed to difference the polygonal object from the textures, with this i mean...
    The "quadsphere" is a primitive 3D object, usually the primitive 3D objects are the ones that can be made by 1 mouse click (in other words, by applying a single action), the quadsphere needs 2 though, first you need to create a cube, and then is needed to "inflate" it like a balloon, this results in a very special distribution of the vertex and edges, there is actually a algorithm made by someone called "cat something" and as far i did learn in the last days most/all of the 3D tools uses it

    The point is... at the time that polygonal 3D object is created the priority is to have the edges and vertex spreaded in the most convenient way to generate a surface the most smooth posible with the smallest number of polygons posible
    But the textures doesnt matters at all... there are many ways to apply textures to that polygons with cubemap textures but evryone of them generates a different distortion

    In the .pdf document uploaded the other day are explained 7 or 8 different ones... really take a look at the pdf, there is no need to read it, just scroll down and dedicate some minutes to look at the images
    There are some that adjusts the texture at the poles of the spehere like a pyramid (obviouslly this ones are not used by the PS3)
    And a bunch others with different deformation patterns
    The shockign thing is from that pdf (that looks very serious, technical and standard) none of them is used by the PS3, and to me this looks like a problem because it means the distortion patterns used by the PS3 are not very common

    As a consequence of that... the mod you made @DeViL303 with the moon texture had wrapping in the sphere poles
    I dont know exactly why happened in that case, but is either because the image you used was already distorted with a different pattern ... of because the tool you used added the incorrect distortion pattern

    --------------
    So... long story short...
    We are in the good road, but we are having problems with the distortion patterns
    Incase of using images that already have distortion we dont know which ones are compatibles with the PS3
    And incase of using a tool to generate the cubemaps we dont know which tools are generating the distortion patterns compatibles with the PS3
     
    Last edited: Jan 21, 2020
    Berion likes this.
  7. 6,746
    6,379
    622
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    6,746
    Likes Received:
    6,379
    Trophy Points:
    622
    Location:
    Babylon 20xxE series
    Can you make me a favour ?, i dont have any 3D app installed but i would like to add a 3D image in the photoshop template i been doing as reference, see the last version to comment it...
    [​IMG]
    My goal by adding the 4 images with the spheres at bottom right is because i would like to show several things, it could be awesome to represent everything i have in mind in a single sphere, probably is not going to be posible, anyway i will talk about it later, im mentioning it just to get an idea of what im trying to do

    Displaying an sphere with the same number of polygons than the PS3 is imposible at this point because we dont know how many there are in every face of the cube (there is a screenshot made by devil some days ago where can be seen some though, but sadly that screenshot was not accurate enought to count them)
    Anyway... incase there are a lot i dont have much rooom, it needs to be an image of 1024x1024 or so so maybe is going to be imposible to have a good representation of the real mesh

    The most important thing we need is to represent the texture locations ... with 2x2 polygons per face
    Is pretty much this image im using... but the one at left have 1x1 polygon per face ... and the one at right have 4x4 polygons per face :crybaby:
    [​IMG]

    But i had another idea... is long to explain... the point is i want to have also subdivisions that are going to help understanding the distortion patterns
    For that i need to have a cube sphere with 6x6 polygons per face, something like this
    Figure 2 have 4x4 and figure 3 have 8x8 but i need 6x6 :crybaby:
    [​IMG]

    I found another image for reference, just because i guess this is how is going to look (i dont care about the style though)
    The most important thing is it needs to have 6x6 polygons per face, and please use a perspective camera angle that allows to see the 4 corners of the cube face
    [​IMG]
     
    Last edited: Jan 22, 2020
  8. 6,746
    6,379
    622
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    6,746
    Likes Received:
    6,379
    Trophy Points:
    622
    Location:
    Babylon 20xxE series
    Sorry for the rambling, just to be explicit... i would like to have an image at 1024x1024 of a cubesphere using 6x6 polygons per face, displayed like this rought sketchup
    The sphere have 6 faces and every face have 6x6 so in total the shere should have 6x6x6 = 216 polygons
    [​IMG]
     
  9. 3,353
    1,811
    297
    Cypher_CG89

    Cypher_CG89 Senior Member

    Joined:
    Jul 30, 2018
    Messages:
    3,353
    Likes Received:
    1,811
    Trophy Points:
    297
    Gender:
    Male
    Occupation:
    Lead Graphic Artist/Dev, VENOM ELITE GAMING
    Location:
    North East, England, UK
    Home Page:
    I will have to use Blender, I just removed 3DS Max as it was driving me up the wall.

    In the meantime you should check out the AMD app linked in a previous post, you can apply a manipulate cubemaps in real time on a sphere > A mini 3D app of sorts. ( its not a big app the installer is like 13.1 MB, and its free from AMD )

    CubeMap_manipulation.png

    Make a dds cube map in Photoshop with Nvidia's DDS plugins, save it, open CubeMap gen, in "Output Cube size" select 1020 x 1020, then select " Filter Cub Map " below it ( this is all the blue section of the app ), then " Load Cube Map (.dds) " and load your dds cube map you made in Photoshop (light green section, top)

    Or if you have made a cubecross you can load that.

    The best thing is with this you can load an image as a " BaseMap " and apply it to the sphere before creating the cube map in PS, this is what is loaded in the screen shot and over this you can see how a GPU ( any GPU, even the RSX ) will apply the texture by LOD and perspective view this is the image in 1020 x 1020, the other image of the same app is a view at 512 x 512 faces ( set Render mode to "MP Alpha LOD ".

    Once you have loaded the proper cube map in dds you can then begin to manipulate it in real time on the sphere, edge blending, face flipping, liner pulls etc> al;so with a base map image loaded and dds you can switch between the views so you can get the image right. All you do is click Filter cubemap in the blue section to update the rendered model.

    You can then choose how to save the manipulated images, from the screenshot you can see the "cubemap cross", you get this by selecting "Save Cube Cross", save as a dds cube map (mainly used in 3D apps like Blender etc, Unreal, 3DS), or you can select "save to images" which will save each 1020 x 1020 face in individual files ( can be saved as DDS, BMP, PNG, JPEG and a few others ) this can be handy if your wanting to try this on the PS3 as you could then use the other app Linked " TileMage Image Spliter " to batch split all the saved jpg's into 4 512 x 512 sections ready to inject into the QRC file.

    Plus this app come with it documents from AMD that tells you all about its features. ( I haven't read it lol, I just started messing with the app )

    Sorry if this wander's off in places or jumps from thing to thing, its like 6.00am and I have done my usual stay up all night messing around with things lol
     
    DeViL303 likes this.
  10. 3,353
    1,811
    297
    Cypher_CG89

    Cypher_CG89 Senior Member

    Joined:
    Jul 30, 2018
    Messages:
    3,353
    Likes Received:
    1,811
    Trophy Points:
    297
    Gender:
    Male
    Occupation:
    Lead Graphic Artist/Dev, VENOM ELITE GAMING
    Location:
    North East, England, UK
    Home Page:
    I got what you meant the first time lol.
     
  11. 8,066
    9,294
    797
    DeViL303

    DeViL303 Developer PSX-Place Supporter

    Joined:
    Jan 23, 2016
    Messages:
    8,066
    Likes Received:
    9,294
    Trophy Points:
    797
    I think you might be over complicating it a bit. The way I see it if you apply 6 square flat textures to the inside of a sphere, the distortion will ALWAYS be the same. There is only one way to do it.

    The amount of polygons is not so important. The more polygons each side of the cube uses the closer to a real sphere it will be.

    I think the reason the moon was not 100% perfect is there is no complete maps of all planets. See on the site with all the planets, there is a note at the bottom about filling in missing info with nearby stuff.
     
  12. 6,746
    6,379
    622
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    6,746
    Likes Received:
    6,379
    Trophy Points:
    622
    Location:
    Babylon 20xxE series
    See the image with the diferent distorsion generated by the different cubemap discussed in that document
    Page 35 and 36 of the pdf... there are images where can be seen marked in colors the areas with bigger distortion, and samples of how the images are deformed with everyone of that cubemaps projections

    In that pdf are discussed 8 types of cubemap proyections, TSC. ASC, OSC, CSC, QSC, KSC, and HEALpix
    For overcomplication it a bit more... none of them is the distortion pattern used in the PS3
     
    Last edited: Jan 22, 2020
  13. 8,066
    9,294
    797
    DeViL303

    DeViL303 Developer PSX-Place Supporter

    Joined:
    Jan 23, 2016
    Messages:
    8,066
    Likes Received:
    9,294
    Trophy Points:
    797
    I mean, maybe they use different layouts for the textures. But for me the distortion will be always be the same.

    Imagine you have a square photo, and you press it inside of a spherical gold fish bowl, there is only ONE way it can distort. For me the NASA maps applied are good enough. There are the best maps we are going to get afaik.

    upload_2020-1-22_12-13-58.png

    If these were maps for a real space mission I say yes, we can do better , but this is only an XMB mod. I can try fix the pinch effect with a fake crater added over the top. It should be easy to do as it is in the middle of the texture.
     
    Last edited: Jan 22, 2020
  14. 6,746
    6,379
    622
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    6,746
    Likes Received:
    6,379
    Trophy Points:
    622
    Location:
    Babylon 20xxE series
    Im not so sure about that, as far i understand there are different ways to create them with different distortions, and different ways to apply them
    Also, the fact your image of the moon had problems at the poles (but you took it from a relliable source) is a proof that something went wrong, either because:
    -the image didnt had the correct distortion before processing it with the tool
    -the tool is applying the texture incorrectly

    Not sure why, but by now i think it was because one of that 2 reasons
     
    DeViL303 likes this.
  15. 6,746
    6,379
    622
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    6,746
    Likes Received:
    6,379
    Trophy Points:
    622
    Location:
    Babylon 20xxE series
    New version of the image im doing, i decreased the transparency of the clouds layer, and added the distortion patterns lines as reference on top of the real images
    [​IMG]
    I guess this explains my request to cypher to create an sphere with 6x6 polygons in each face of the cube, i want my distortion lines to match with the edges of the polygons :)

    The point is... the distortion pattern is the same in all images, only difference is his rotation
    The distortion pattern for every image have 3x3 (deformed) "squares" that are going to be displayed as straight lines when applyed to the sphere
    Displaying the distortion patterns with 3x3 is the most convenient way for a couple of reasons tricky to explain
    A circle have 360º... every face of our cube have 90º ... and every face have 6x6 polygons (in the reference image i requested to cypher) ... so every polygon is 15ºx15º

    One of the goals is that im making it match with the real earth meridians used in maps, as discussed here :)
    https://www.psx-place.com/threads/r...der_plugin-earth-qrc.27756/page-5#post-223673
     
    Last edited: Jan 22, 2020
    DeViL303 likes this.
  16. 8,066
    9,294
    797
    DeViL303

    DeViL303 Developer PSX-Place Supporter

    Joined:
    Jan 23, 2016
    Messages:
    8,066
    Likes Received:
    9,294
    Trophy Points:
    797
    Yeah, Thinking about it more, you are right. As we should have accurate maps of the whole moon by now. Here are the 2 textures that are wrong and have a pinch effect at the poles. It not that bad really considering.

    NX_from_cubemap_generator.jpg PZ_from_cubemap_generator.jpg

    As this is perfectly in the centre I could cheat and fix them, I have some PNGs of moon terrain..

    crater.png earth.png hills1.png hills2.png moon1.png moon2.png moon3.png cracks.png


    Here are some "fixed" version.... I know its not right, and if we could find the exact type of texture that will fit that is cool, but for now its better than nothing.

    NX_FIXED.jpg PZ_FIXED.jpg
     
  17. 8,066
    9,294
    797
    DeViL303

    DeViL303 Developer PSX-Place Supporter

    Joined:
    Jan 23, 2016
    Messages:
    8,066
    Likes Received:
    9,294
    Trophy Points:
    797
    So we can move files around within the container, but we can not increase the overall QRC size?... that is a shame.
    Are you sure. When I look at the end of the qrc in a hex editor I see one of the path files or INIs or something like that. It does not look like a file table anyway.

    upload_2020-1-22_12-52-46.png

    I wonder what would happen if we edited the file table to say this last INI file keeps on going, and is 1MB or more?
     
  18. 6,746
    6,379
    622
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    6,746
    Likes Received:
    6,379
    Trophy Points:
    622
    Location:
    Babylon 20xxE series
    Yep, there is always that way of fixing it, is ok but we are so close :/
    The biggest problem i see right now is we dont know exactly what we need, lol, but at this point is just a mater of trying different tools and different image sources (incase some of that sources uses different distortion standards)

    What i mentioned some post ago could help btw, using a checkered pattern to have some visual control of the distortion results
    I just made an image for that purpose, not sure but maybe helps
    [​IMG]
     
  19. 8,066
    9,294
    797
    DeViL303

    DeViL303 Developer PSX-Place Supporter

    Joined:
    Jan 23, 2016
    Messages:
    8,066
    Likes Received:
    9,294
    Trophy Points:
    797
    I will try make a cubemap of that and inject it.
     
  20. 6,746
    6,379
    622
    sandungas

    sandungas Moderator Developer

    Joined:
    Dec 31, 2014
    Messages:
    6,746
    Likes Received:
    6,379
    Trophy Points:
    622
    Location:
    Babylon 20xxE series
    [​IMG]
    Thats the last file of the filetable
    Code:
    <?xml version="1.0" encoding="utf-8"?>
    <qrc>
    	<file-table>
    		...
    		<file src="Offset=0x002B9EA0, Length=0x00000060.bin" id="earth/camera/camera9.ini" />
    	</file-table>
    </qrc>
    
    The formula is..
    File absolute offset = filetable absolute offset + file relative offset

    earth/camera/camera9.ini absolute offset = 0xE5F0 + 0x002B9EA0 = 0x2C8490
    And his length is Length=0x00000060
    So... the file end offset is at 0x2C8490 + 0x60 = 0x2C84F0 (it matches with the end of the .qrc file)

    Thats why i said that there is nothing after the file table that could be displaced (this could be an inconvenience)
    Incase there was something after the filetable then it would be imposible/insane to increase the file table size patching the file in a hexeditor manually... because that would displace whatever that could be after the file table and it would "break" the .qrc structure
    There is nothing... so we dont have this problem... it was just a note worthy to be mentioned :)

    I guess is just a matter to increase the value that indicates the total filesize table, right now im not even sure if exists, but probably, and should be located at top of the .qrc file structure (loaded by the firmware in a previous step before loading the filetable)
     
    Last edited: Jan 22, 2020
    DeViL303 likes this.

Share This Page