PS3 [Research] Modifying the Gaia Visualization (custom_render_plugin/earth.qrc)

[pink1]

Wow, adding the cubemap generator too would be great.


It would also be great if we could add more space. Like an extra 100KB per image for all 48 (ground and clouds) and extra 50KB for all Specular.. We can do a remap OR sprx patch and run any size earth.qrc from hdd if we need to. So space is not an issue really.

Also adding more space for MNU files would be nice, so we can copy and paste freely between them, and add new settings.

For the cubemap generator I think we can just add the files and a web browser pointed to it. It's kind of a cheap way of doing it for now anyway.

I'm happy to try and help get it figured out if you wanna play along. I think for a test we should try to find something close to the end & edit it's size that way we don't have to change a lot of the address, less chance of messing it up lol

 

[DeViL303]

Yeah the cube map generator is not so important really, The site is so easy to use.

I'm happy to try and help get it figured out if you wanna play along. I think for a test we should try to find something close to the end & edit it's size that way we don't have to change a lot of the address, less chance of messing it up lol

Yeah for sure.The last jpg is at :

This would be a candidate.

This 18.1KB file is there normally in OFW. In my template there is a 1KB black image instead





Or there is a text file at the very end of the end if that would be easier.

I gotta go..back tomorrow.

 

[pink1]

So for the last jpg we would take 295BA0 - E5F0 = 2875B0. We search for that in the file table and the next bytes are the size.
So if we make it bigger we would need to edit the address of the rest by adding the difference of the size of the jpg.

We can also find the addresses we need to edit with the python tool @sandungas posted.

 

[sandungas]

Btw, there is an easy way to find the start offsets of all the files, it just needs to be made a simple sum math operation, and a trick in hexeditor

1) Extract the contents of the .qrc with the python CXML decompiler tool

2) Open the xml file generated by CXML decompiler, it looks like this:

<?xml version="1.0" encoding="utf-8"?>
<qrc>
	<file-table>
		<file src="Offset=0x00000000, Length=0x0000025A.bin" id="earth/ATM.mnu" />
		...
		...
		...
	</file-table>
</qrc>

Thats the first file, located in the first position of the file table, this is why it have an Offset=0x00000000
Is an offset relative to the start offset of the filetable
In other words... the start offset of the first file is the same than the start offset of the filetable

5) So... now we need to find the absolute offset of the filetable
Open in the hexeditor earth.qrc and the file named "Offset=0x00000000, Length=0x0000025A.bin" (because we know this is the first file)
In the file... press keys CTRL + A (to select all), then CTRL + C (to copy)
In the .qrc... press CTRL + F (to search)... it opens a settings window, click in the tab "search for hex vaues" and press CTRL + V (to paste)... then click "OK"

6) So... now you have highlighted the first file in the file table entirelly
If your goal is to replace it select the tab [EDIT] -> [FILL SELECTION] and fill the selected area with zeroes
This cleans the garbage bytes... also it changes the color of the edited bytes to red (is very easy to identify where the file started and ended)

7) Ok, so now is time to use the math formula
Scroll up to the first byte marked in red... in this case for earth.qrc 4.85 the first file was located at offset 0xE5F0
And as mentioned before... the first file matches exactly with the start offset of the filetable... so now by looking at the xml generated by the CXML decompiler pythong script we can calculate the absolute offset of all the files, this way

<?xml version="1.0" encoding="utf-8"?>
<qrc>
	<file-table>
		<file src="Offset=0x00000000, Length=0x0000025A.bin" id="earth/ATM.mnu" />
		...
		<file src="Offset=0x00044F20, Length=0x00000880.bin" id="lib/gaia/Sun.fpo" />
		...
		<file src="Offset=0x0019E0B0, Length=0x00002B38.bin" id="earth/flashrom/disk.dds" />
		...
		<file src="Offset=0x002B8A30, Length=0x0000057F.bin" id="earth/camera/views.txt" />
	</file-table>
</qrc>

The formula is..
File absolute offset = filetable absolute offset + file relative offset

Some examples
earth/ATM.mnu absolute offset = 0xE5F0 + 0x00000000 = 0xE5F0 (yeah, genius, something + 0 = something)
lib/gaia/Sun.fpo absolute offset = 0xE5F0 + 0x00044F20 = 0x53510
earth/flashrom/disk.dds absolute offset = 0xE5F0 + 0x0019E0B0 = 0x1AC6A0
earth/camera/views.txt absolute offset = 0xE5F0 + 0x002B8A30 = 0x2C7020


-----------------
And remember, hexeditors like HxD have an option named "go to" (CTRL + G), you can use it with the values for the absolute offsets of all the files to "jump" into them easilly
As example... in earth.qrc press CTRL + G and paste 1AC6A0 ... this is going to take you to the absolute start offset of disk.dds

 

[sandungas]

On the qrc can we not edit the file sizes in the file table or does that cause problems?

Yes, is very convenient, actually we have been doing it before in other frankentests (inside lines.qrc trying to replace a couple of files inside it named spline.elf and particles.elf)
That 2 boys was specially picky in the experiments and was refusing to work, so in one of the tests i had to adjust well the file length values in the index "just incase"... but not luck, the test failed for other reasons

Anyway... What we do to remap files inside the .qrc structure is to change that values of the index, is very straightforward, in the index you can see 2 values for every file for the "offset" and "length", with 4 bytes each, and consecutivelly
This values are what you can see in the xml generated by the CXML decompiler, as example...

<?xml version="1.0" encoding="utf-8"?>
<qrc>
	<file-table>
		<file src="Offset=0x00001860, Length=0x00000121.bin" id="earth/presets/0/earth/CLOUDS_4M.mnu" />
	</file-table>
</qrc>

Copy together the 2 hex values that appears in the xml
So the... Offset=0x00001860, Length=0x00000121.bin ... becomes ... 0x0000186000000121

If you search for that in the .qrc this is going to highlight it in the index... the firmware reads that value and then it "jumps" to load the file ... by changing that values we cheat the firmware into loading a different file... or a file in the same offset but with a different size

Loading different files is something we have been doing many times in the experiments with other .qrc files
Changing the size works... but we have made it only to convert the .MNU files to dummies (reducing his size to 9 bytes in the index), to corrupt .MNU files (reducing them to zero), or the experiment i mentioned about spline.elf and particles.elf (because i was using .elf files from older firmwares and had smaller sizes)

But the point is we have full freedom to change all that values of the index, and by doing this we can reorganize the contents of the whole filetable
The restriction we have is the total size of the filetable, it can be calculated this way:
Total size of the file table = end offset of the last file - start offset of the first file

The reason why we cant change the total size of the filetable is because if we do it will be needed to rebuild other parts of the .QRC structure that are out of the filetable... and that could be tricky
The good thing is the filetable is located at the end of the .QRC ...so incase of making it bigger there is not going to be other data located after the filetable displaced by it


--------------
And yeah... i mentioned it before... for this kind of mods replacing the textures of the earth most of the times the custom files are going to be smaller,, so is better to adjust his size in the index
As example, with the value i mentioned before... 0000186000000121
You need to search for that... and change the 121 (original size) by the real size of the custom file (that usually is going to be smaller)

 

[sandungas]

The tools you are using to create the cubemaps are not confirmed to work for what we need btw
When talking about this is needed to difference the polygonal object from the textures, with this i mean...
The "quadsphere" is a primitive 3D object, usually the primitive 3D objects are the ones that can be made by 1 mouse click (in other words, by applying a single action), the quadsphere needs 2 though, first you need to create a cube, and then is needed to "inflate" it like a balloon, this results in a very special distribution of the vertex and edges, there is actually a algorithm made by someone called "cat something" and as far i did learn in the last days most/all of the 3D tools uses it

The point is... at the time that polygonal 3D object is created the priority is to have the edges and vertex spreaded in the most convenient way to generate a surface the most smooth posible with the smallest number of polygons posible
But the textures doesnt matters at all... there are many ways to apply textures to that polygons with cubemap textures but evryone of them generates a different distortion

In the .pdf document uploaded the other day are explained 7 or 8 different ones... really take a look at the pdf, there is no need to read it, just scroll down and dedicate some minutes to look at the images
There are some that adjusts the texture at the poles of the spehere like a pyramid (obviouslly this ones are not used by the PS3)
And a bunch others with different deformation patterns
The shockign thing is from that pdf (that looks very serious, technical and standard) none of them is used by the PS3, and to me this looks like a problem because it means the distortion patterns used by the PS3 are not very common

As a consequence of that... the mod you made @DeViL303 with the moon texture had wrapping in the sphere poles
I dont know exactly why happened in that case, but is either because the image you used was already distorted with a different pattern ... of because the tool you used added the incorrect distortion pattern

--------------
So... long story short...
We are in the good road, but we are having problems with the distortion patterns
Incase of using images that already have distortion we dont know which ones are compatibles with the PS3
And incase of using a tool to generate the cubemaps we dont know which tools are generating the distortion patterns compatibles with the PS3

 

[sandungas]

Found this on the net while messing around doing this in Blender and 3DS Max and was wondering if there was a simpler app for other's to use as Blender an 3DS MAX aren't really suited for basic users.

Can you make me a favour ?, i dont have any 3D app installed but i would like to add a 3D image in the photoshop template i been doing as reference, see the last version to comment it...

My goal by adding the 4 images with the spheres at bottom right is because i would like to show several things, it could be awesome to represent everything i have in mind in a single sphere, probably is not going to be posible, anyway i will talk about it later, im mentioning it just to get an idea of what im trying to do

Displaying an sphere with the same number of polygons than the PS3 is imposible at this point because we dont know how many there are in every face of the cube (there is a screenshot made by devil some days ago where can be seen some though, but sadly that screenshot was not accurate enought to count them)
Anyway... incase there are a lot i dont have much rooom, it needs to be an image of 1024x1024 or so so maybe is going to be imposible to have a good representation of the real mesh

The most important thing we need is to represent the texture locations ... with 2x2 polygons per face
Is pretty much this image im using... but the one at left have 1x1 polygon per face ... and the one at right have 4x4 polygons per face

But i had another idea... is long to explain... the point is i want to have also subdivisions that are going to help understanding the distortion patterns
For that i need to have a cube sphere with 6x6 polygons per face, something like this
Figure 2 have 4x4 and figure 3 have 8x8 but i need 6x6

I found another image for reference, just because i guess this is how is going to look (i dont care about the style though)
The most important thing is it needs to have 6x6 polygons per face, and please use a perspective camera angle that allows to see the 4 corners of the cube face

 

[sandungas]

Sorry for the rambling, just to be explicit... i would like to have an image at 1024x1024 of a cubesphere using 6x6 polygons per face, displayed like this rought sketchup
The sphere have 6 faces and every face have 6x6 so in total the shere should have 6x6x6 = 216 polygons

 

[Cypher_CG89]

Can you make me a favour ?, i dont have any 3D app installed but i would like to add a 3D image in the photoshop template i been doing as reference, see the last version to comment it...

My goal by adding the 4 images with the spheres at bottom right is because i would like to show several things, it could be awesome to represent everything i have in mind in a single sphere, probably is not going to be posible, anyway i will talk about it later, im mentioning it just to get an idea of what im trying to do

Displaying an sphere with the same number of polygons than the PS3 is imposible at this point because we dont know how many there are in every face of the cube (there is a screenshot made by devil some days ago where can be seen some though, but sadly that screenshot was not accurate enought to count them)
Anyway... incase there are a lot i dont have much rooom, it needs to be an image of 1024x1024 or so so maybe is going to be imposible to have a good representation of the real mesh

The most important thing we need is to represent the texture locations ... with 2x2 polygons per face
Is pretty much this image im using... but the one at left have 1x1 polygon per face ... and the one at right have 4x4 polygons per face

But i had another idea... is long to explain... the point is i want to have also subdivisions that are going to help understanding the distortion patterns
For that i need to have a cube sphere with 6x6 polygons per face, something like this
Figure 2 have 4x4 and figure 3 have 8x8 but i need 6x6

I found another image for reference, just because i guess this is how is going to look (i dont care about the style though)
The most important thing is it needs to have 6x6 polygons per face, and please use a perspective camera angle that allows to see the 4 corners of the cube face

I will have to use Blender, I just removed 3DS Max as it was driving me up the wall.

In the meantime you should check out the AMD app linked in a previous post, you can apply a manipulate cubemaps in real time on a sphere > A mini 3D app of sorts. ( its not a big app the installer is like 13.1 MB, and its free from AMD )

Make a dds cube map in Photoshop with Nvidia's DDS plugins, save it, open CubeMap gen, in "Output Cube size" select 1020 x 1020, then select " Filter Cub Map " below it ( this is all the blue section of the app ), then " Load Cube Map (.dds) " and load your dds cube map you made in Photoshop (light green section, top)

Or if you have made a cubecross you can load that.

The best thing is with this you can load an image as a " BaseMap " and apply it to the sphere before creating the cube map in PS, this is what is loaded in the screen shot and over this you can see how a GPU ( any GPU, even the RSX ) will apply the texture by LOD and perspective view this is the image in 1020 x 1020, the other image of the same app is a view at 512 x 512 faces ( set Render mode to "MP Alpha LOD ".

Once you have loaded the proper cube map in dds you can then begin to manipulate it in real time on the sphere, edge blending, face flipping, liner pulls etc> al;so with a base map image loaded and dds you can switch between the views so you can get the image right. All you do is click Filter cubemap in the blue section to update the rendered model.

You can then choose how to save the manipulated images, from the screenshot you can see the "cubemap cross", you get this by selecting "Save Cube Cross", save as a dds cube map (mainly used in 3D apps like Blender etc, Unreal, 3DS), or you can select "save to images" which will save each 1020 x 1020 face in individual files ( can be saved as DDS, BMP, PNG, JPEG and a few others ) this can be handy if your wanting to try this on the PS3 as you could then use the other app Linked " TileMage Image Spliter " to batch split all the saved jpg's into 4 512 x 512 sections ready to inject into the QRC file.

Plus this app come with it documents from AMD that tells you all about its features. ( I haven't read it lol, I just started messing with the app )

Sorry if this wander's off in places or jumps from thing to thing, its like 6.00am and I have done my usual stay up all night messing around with things lol

 

[Cypher_CG89]

Sorry for the rambling, just to be explicit... i would like to have an image at 1024x1024 of a cubesphere using 6x6 polygons per face, displayed like this rought sketchup
The sphere have 6 faces and every face have 6x6 so in total the shere should have 6x6x6 = 216 polygons

I got what you meant the first time lol.

 

[DeViL303]

As a consequence of that... the mod you made @DeViL303 with the moon texture had wrapping in the sphere poles
I dont know exactly why happened in that case, but is either because the image you used was already distorted with a different pattern ... of because the tool you used added the incorrect distortion pattern

--------------
So... long story short...
We are in the good road, but we are having problems with the distortion patterns
Incase of using images that already have distortion we dont know which ones are compatibles with the PS3
And incase of using a tool to generate the cubemaps we dont know which tools are generating the distortion patterns compatibles with the PS3

I think you might be over complicating it a bit. The way I see it if you apply 6 square flat textures to the inside of a sphere, the distortion will ALWAYS be the same. There is only one way to do it.

The amount of polygons is not so important. The more polygons each side of the cube uses the closer to a real sphere it will be.

I think the reason the moon was not 100% perfect is there is no complete maps of all planets. See on the site with all the planets, there is a note at the bottom about filling in missing info with nearby stuff.

 

[sandungas]

http://casopisi.junis.ni.ac.rs/index.php/FUMathInf/article/viewFile/871/pdf_75

See the image with the diferent distorsion generated by the different cubemap discussed in that document
Page 35 and 36 of the pdf... there are images where can be seen marked in colors the areas with bigger distortion, and samples of how the images are deformed with everyone of that cubemaps projections

In that pdf are discussed 8 types of cubemap proyections, TSC. ASC, OSC, CSC, QSC, KSC, and HEALpix
For overcomplication it a bit more... none of them is the distortion pattern used in the PS3

 

[DeViL303]

I mean, maybe they use different layouts for the textures. But for me the distortion will be always be the same.

Imagine you have a square photo, and you press it inside of a spherical gold fish bowl, there is only ONE way it can distort. For me the NASA maps applied are good enough. There are the best maps we are going to get afaik.

If these were maps for a real space mission I say yes, we can do better , but this is only an XMB mod. I can try fix the pinch effect with a fake crater added over the top. It should be easy to do as it is in the middle of the texture.

 

[sandungas]

Im not so sure about that, as far i understand there are different ways to create them with different distortions, and different ways to apply them
Also, the fact your image of the moon had problems at the poles (but you took it from a relliable source) is a proof that something went wrong, either because:
-the image didnt had the correct distortion before processing it with the tool
-the tool is applying the texture incorrectly

Not sure why, but by now i think it was because one of that 2 reasons

 

[sandungas]

New version of the image im doing, i decreased the transparency of the clouds layer, and added the distortion patterns lines as reference on top of the real images

I guess this explains my request to cypher to create an sphere with 6x6 polygons in each face of the cube, i want my distortion lines to match with the edges of the polygons

The point is... the distortion pattern is the same in all images, only difference is his rotation
The distortion pattern for every image have 3x3 (deformed) "squares" that are going to be displayed as straight lines when applyed to the sphere
Displaying the distortion patterns with 3x3 is the most convenient way for a couple of reasons tricky to explain
A circle have 360º... every face of our cube have 90º ... and every face have 6x6 polygons (in the reference image i requested to cypher) ... so every polygon is 15ºx15º

One of the goals is that im making it match with the real earth meridians used in maps, as discussed here
https://www.psx-place.com/threads/r...der_plugin-earth-qrc.27756/page-5#post-223673

 

[DeViL303]

Yeah, Thinking about it more, you are right. As we should have accurate maps of the whole moon by now. Here are the 2 textures that are wrong and have a pinch effect at the poles. It not that bad really considering.



As this is perfectly in the centre I could cheat and fix them, I have some PNGs of moon terrain..




Here are some "fixed" version.... I know its not right, and if we could find the exact type of texture that will fit that is cool, but for now its better than nothing.

 

[DeViL303]

But the point is we have full freedom to change all that values of the index, and by doing this we can reorganize the contents of the whole filetable
The restriction we have is the total size of the filetable, it can be calculated this way:
Total size of the file table = end offset of the last file - start offset of the first file

So we can move files around within the container, but we can not increase the overall QRC size?... that is a shame.

The good thing is the filetable is located at the end of the .QRC ...so incase of making it bigger there is not going to be other data located after the filetable displaced by it

Are you sure. When I look at the end of the qrc in a hex editor I see one of the path files or INIs or something like that. It does not look like a file table anyway.

I wonder what would happen if we edited the file table to say this last INI file keeps on going, and is 1MB or more?

 

[sandungas]

Yep, there is always that way of fixing it, is ok but we are so close :/
The biggest problem i see right now is we dont know exactly what we need, lol, but at this point is just a mater of trying different tools and different image sources (incase some of that sources uses different distortion standards)

What i mentioned some post ago could help btw, using a checkered pattern to have some visual control of the distortion results
I just made an image for that purpose, not sure but maybe helps

 

[DeViL303]

I will try make a cubemap of that and inject it.

 

[sandungas]

So we can move files around within the container, but we can not increase the overall QRC size?... that is a shame.

Are you sure. When I look at the end of the qrc in a hex editor I see one of the path files or INIs or something like that. It does not look like a file table anyway.

View attachment 22939

I wonder what would happen if we edited the file table to say this last INI file keeps on going, and is 1MB or more?

Thats the last file of the filetable

<?xml version="1.0" encoding="utf-8"?>
<qrc>
	<file-table>
		...
		<file src="Offset=0x002B9EA0, Length=0x00000060.bin" id="earth/camera/camera9.ini" />
	</file-table>
</qrc>

The formula is..
File absolute offset = filetable absolute offset + file relative offset

earth/camera/camera9.ini absolute offset = 0xE5F0 + 0x002B9EA0 = 0x2C8490
And his length is Length=0x00000060
So... the file end offset is at 0x2C8490 + 0x60 = 0x2C84F0 (it matches with the end of the .qrc file)

Thats why i said that there is nothing after the file table that could be displaced (this could be an inconvenience)
Incase there was something after the filetable then it would be imposible/insane to increase the file table size patching the file in a hexeditor manually... because that would displace whatever that could be after the file table and it would "break" the .qrc structure
There is nothing... so we dont have this problem... it was just a note worthy to be mentioned

I guess is just a matter to increase the value that indicates the total filesize table, right now im not even sure if exists, but probably, and should be located at top of the .qrc file structure (loaded by the firmware in a previous step before loading the filetable)