PS3 [Research] Modifying the Gaia Visualization (custom_render_plugin/earth.qrc)

DeViL303 said:
@sandungas @pink1 See the super handy little tool i found for Zlib called zdrop. Very simple and small. Seems to have slightly different compression to simplyzip, slightly different filesize, but it works perfectly for ps3 so is a great little addition.
Click to expand...
For zlib i use this a lot
https://www.psx-place.com/resources/zlib-gui.716/

The GUI is an .exe but the real work is made by a zlib.dll library
The GUI is hosted on a personal page of a russian guy... but the .dll is official from the zlib web
I took both, packed them together, and uploaded them to the forum for simplification purposes, but the .dll can be replaced by new versions at any point

Im not sure if there is some way to use it in command line though, but the .dll can be included in your program @pink1 compiled as static, right ?
In the zlib web they have source code and zlib versions compiled in all colors and flavours
 
sandungas said:
but i know the CXML structure and i knew the full rebuilding was "not so hard" in comparison with extraction ;)
The challengue is to process the TOC table, this took lot of work to reverse it, heheh, at this point you have full support for it i guess
At this point the biggest challengue is the float table, and the float array tables, the way it works is they only stores a single copy of every "float" or "float array" data types
You need to imagine it like if the "source" data was parsed from an xml, and the value you need to store is either "something=1.342" (single float) or "something=1.345, 10.345" (float array composed by 2 floats)

Also, every value is stored only a single time (for efficiency)... lets say... if the original xml had different attributes owned by different elements... but his value is the same (lets say the same float 1.234 repeated 20 times) then the table stores it only 1 time (and the structure uses references to point to it)

But as i said, this is tricky and you should ignore it by now (i would kill for a qrc rebuilder please do it)
Click to expand...

Any chance you can make an xml of what the earth qrc would be or could be structured like? or would that be too much hassle..? or another qrc if its easier...
 
sandungas said:
For zlib i use this a lot
https://www.psx-place.com/resources/zlib-gui.716/

The GUI is an .exe but the real work is made by a zlib.dll library
The GUI is hosted on a personal page of a russian guy... but the .dll is official from the zlib web
I took both, packed them together, and uploaded them to the forum for simplification purposes, but the .dll can be replaced by new versions at any point

Im not sure if there is some way to use it in command line though, but the .dll can be included in your program @pink1 compiled as static, right ?
In the zlib web they have source code and zlib versions compiled in all colors and flavours
Click to expand...
Well I have an exe that works fine from the command line, that was the point of my post.
 
Cypher_CG89 said:
Any chance you can make an xml of what the earth qrc would be or could be structured like? or would that be too much hassle..? or another qrc if its easier...
Click to expand...
I posted some in the talk pages in wiki... mostly because at that time was a bit experimental so was like dirty notes
This one from icontex.qrc is pretty straightforward https://www.psdevwiki.com/ps3/Talk:Icontex.qrc

For .qrc is like this:
Code: 
<qrc>
	<file-table>
		<file src="icontex_0x00000000.bin" id="1n" size="87508" />
		...
		..
		.
	</file-table>
</qrc>
That hierarchy of elements/atributes (represented as an xml) is exactly the same in the TOC of the .qrc
The reason why it looks so simililar to an .xml is because the "source" files (before converted into a .qrc) was an xml :D
This is why is so convenient to extract the info of the TOC as an xml (also, is convenient to use an xml to create a TOC)

The other cxml containers (such .raf and .p3t) have a lot more info in the TOC, the best example of this is coldboot.raf there is an example of the xml here:
https://www.psdevwiki.com/ps3/Talk:Coldboot.raf#Coldboot.sxml

Instead of having only a <file-table> it have a lot more tables, as example... (one of each)
Code: 
<raf>
	<scene camera_num="1" light_num="2" script_num="1" actor_num="6">
		<actor anim_speed="1.0,1.0,1.0,1.0" scale="1.28,0.64,0.64" uv_offset="0.0,0.0" color="1.0,1.0,1.0,1.0" anim_weight="1.0,0.0,0.0,0.0" material="mtrl_logo" uv_scale="1.0,-1.0" id="logo" anim_time="0.0,0.0,0.0,0.0" position="1.85,0.0,0.0" model="plane" rotation="1.570796,0.0,0.0" />
	</scene>

	<model-table num="1">
		<model id="plane">
		</model>
	</model-table>

	<material-table num="6">
		<material id="mtrl_logo" effect="sce01">
			<_texture texref="_new_logo.gtf" />
	</material-table>

	<texture-table num="6">
		<texture min_filter="0" mip_filter="0" fileindex="2" mag_filter="0" ext="1" wrap_t="1" wrap_s="1" wrap_p="1" fileref="new_logo.gtf" type="0" id="_new_logo.gtf" />
	</texture-table>

	<file-table>
		<file src="c:\coldboot\new_logo.gtf" type="4" id="new_logo.gtf" />
		...
		..
		.
	</file-table>
</raf>

By reading the TOC you can "navigate" that heirarchy, and load the names of the elements (like camera_num stored as a string), values of the attributes (that can be integers, floats, or float arrays), etc...


And btw... now that im looking at it i can show a good example of how works the float arrays... this ones are float arrays:
color="1.0,1.0,1.0,1.0"
scale="1.28,0.64,0.64"
uv_offset="0.0,0.0"

Usually are the values for colors (an array of 4 values for RGBA), or 3D coordinates (an array of 3 values for XYZ), or 2D coordinates for textures (an array of 2 values for UV)
 
Last edited:
sandungas said:
I posted some in the talk pages in wiki... mostly because at that time was a bit experimental so was like dirty notes
This one from icontex.qrc is pretty straightforward https://www.psdevwiki.com/ps3/Talk:Icontex.qrc

For .qrc is like this:
Code: 
<qrc>
    <file-table>
        <file src="icontex_0x00000000.bin" id="1n" size="87508" />
        ...
        ..
        .
    </file-table>
</qrc>
That hierarchy of elements/atributes (represented as an xml) is exactly the same in the TOC of the .qrc
The reason why it looks so simililar to an .xml is because the "source" files (before converted into a .qrc) was an xml :D
This is why is so convenient to extract the info of the TOC as an xml (also, is convenient to use an xml to create a TOC)

The other cxml containers (such .raf and .p3t) have a lot more info in the TOC, the best example of this is coldboot.raf there is an example of the xml here:
https://www.psdevwiki.com/ps3/Talk:Coldboot.raf#Coldboot.sxml

Instead of having only a <file-table> it have a lot more tables, as example... (one of each)
Code: 
<raf>
    <scene camera_num="1" light_num="2" script_num="1" actor_num="6">
        <actor anim_speed="1.0,1.0,1.0,1.0" scale="1.28,0.64,0.64" uv_offset="0.0,0.0" color="1.0,1.0,1.0,1.0" anim_weight="1.0,0.0,0.0,0.0" material="mtrl_logo" uv_scale="1.0,-1.0" id="logo" anim_time="0.0,0.0,0.0,0.0" position="1.85,0.0,0.0" model="plane" rotation="1.570796,0.0,0.0" />
    </scene>

    <model-table num="1">
        <model id="plane">
        </model>
    </model-table>

    <material-table num="6">
        <material id="mtrl_logo" effect="sce01">
            <_texture texref="_new_logo.gtf" />
    </material-table>

    <texture-table num="6">
        <texture min_filter="0" mip_filter="0" fileindex="2" mag_filter="0" ext="1" wrap_t="1" wrap_s="1" wrap_p="1" fileref="new_logo.gtf" type="0" id="_new_logo.gtf" />
    </texture-table>

    <file-table>
        <file src="c:\coldboot\new_logo.gtf" type="4" id="new_logo.gtf" />
        ...
        ..
        .
    </file-table>
</raf>

By reading the TOC you can "navigate" that heirarchy, and load the names of the elements (like camera_num stored as a string), values of the attributes (that can be integers, floats, or float arrays), etc...


And btw... now that im looking at it i can show a good example of how works the float arrays... this ones are float arrays:
color="1.0,1.0,1.0,1.0"
scale="1.28,0.64,0.64"
uv_offset="0.0,0.0"

Usually are the values for colors (and array of 4 values for RGBA), or 3D coordinates (an array of 3 values for XYZ), or 2D coordinates for textures (an array of 2 values for UV)
Click to expand...

After reading this I see exactly where I am going wrong lol. Thanks
 
Here is another layout for injecting jpgs, this bat takes a single 512x512 image and multiplies it into 2 different flat maps 4096x2048 and 8192x4096. These are perfect resolutions for putting into a cubemap generator.

upload_2020-1-27_13-5-7.png

Code: 
bin\jpegtran -crop 4096x2048+0+0 -outfile 4096x2048.jpg 512x512.jpg
bin\jpegtran -drop +512+0 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +1024+0 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +1536+0 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +2048+0 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +2560+0 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +3072+0 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +3584+0 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +0+512 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +512+512 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +1024+512 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +1536+512 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +2048+512 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +2560+512 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +3072+512 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +3584+512 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +0+1024 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +512+1024 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +1024+1024 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +1536+1024 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +2048+1024 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +2560+1024 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +3072+1024 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +3584+1024 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +0+1536 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +512+1536 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +1024+1536 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +1536+1536 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +2048+1536 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +2560+1536 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +3072+1536 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -drop +3584+1536 512x512.jpg -outfile 4096x2048.jpg 4096x2048.jpg
bin\jpegtran -crop 8192x4096+0+0 -outfile 8192x4096.jpg 4096x2048.jpg
bin\jpegtran -drop +4096+0 4096x2048.jpg -outfile 8192x4096.jpg 8192x4096.jpg
bin\jpegtran -drop +0+2048 4096x2048.jpg -outfile 8192x4096.jpg 8192x4096.jpg
bin\jpegtran -drop +4096+2048 4096x2048.jpg -outfile 8192x4096.jpg 8192x4096.jpg

So you start off with this image for example, a 512x512 PS2 Logo named 512x512.jpg.
upload_2020-1-27_11-44-50.png

This bat will tile it into a 4096x2048 image and a 8192x4096 image like these:
full.jpg 8192x4096.jpg

Then you load either of those into the cubemap generator at https://matheowis.github.io/HDRI-to-CubeMap/ and you get these.

MAKE SURE TO SET IT TO 1024x1024 PER FACE

cubemap.jpg map2.jpg

Convert that PNG to JPG and then Inject that cubemap.jpg using "Tile to QRC" bat and you have this effect.

4096x2048:
preview.jpg preview2.jpg preview3.jpg

8192x4096:
preview2.jpg preview21.jpg preview22.jpg
 

Attachments

Last edited:
@DeViL303 do you have the program CubeTheSphere installed ? (is better for this test if you dont)

Incase you dont have it... can you tell me if this works in your PC ?
http://s000.tinyupload.com/index.php?file_id=06425447159756080263

Im asking because the program CubeTheSphere have an installer (and it creates a log at installation time where it can be seen it depends of some .dll files)
The .exe i uploaded is the result of that installation, but im not including the .dll's (so im not sure if is going to work fine just like that)



-------------
Im thinking if we add cubethesphere.exe in the toolkit collection we could automate everything (included the generation of the cube faces)
As example, in the bat you just uploaded... there is a point where the user needs to pause the conversion and go to the web browser

Also, the other day i was taking a look at cubethesphere.exe in a hexeditor trying to figure how to apply a patch to align the meridian 0º with sony official files (and failed miserably btw)... but while taking a look at that i realized we can patch the strings for the filenames outputs
So... i think (not tested though) we can patch it to generate the 6 cube faces with the official names (from 0.bmp to 5.bmp)

*CubeTheSphere generates the cube faces in .bmp format though... but i guess we are not going to have much problems to do the bmp to jpg conversion with some command line tool




Edit:
Mhhh. not sure if that would be enought... to automate it well it would be needed to pass some commands to cubethesphere.exe from command line (like input image, and output size)... i doubt cubethesphere allows to take arguments from command line :/
 
Last edited:
Here is a cubemap template attached that can be used to do a little trick to avoid MOST of the distortion. :)

Write something across the centre of the template like this, it can be anything at all :)

cubemap.jpg

Then use that as the cubemap.jpg for tile_to_qrc and you get:

PREVIEW.jpg
 

Attachments

  • cubemap.jpg
    cubemap.jpg
    193.3 KB · Views: 103
sandungas said:
@DeViL303 do you have the program CubeTheSphere installed ? (is better for this test if you dont)

Incase you dont have it... can you tell me if this works in your PC ?
http://s000.tinyupload.com/index.php?file_id=06425447159756080263

Im asking because the program CubeTheSphere have an installer (and it creates a log at installation time where it can be seen it depends of some .dll files)
The .exe i uploaded is the result of that installation, but im not including the .dll's (so im not sure if is going to work fine just like that)
Click to expand...
That works with just the exe, I did not have it installed and I just used the exe alone to convert 8k jpeg flat map to 6x3MB BMPs at 1024x1024..

If it could be used from the command line or we can find the source it would be great to automate all that too.

The BMP would not be an issue, usage might be though. Have you tried calling it from command line?
 
xf505 said:
Red Snake Skin earth.qrc
Click to expand...
Nice one. :)

PREVIEW.jpg


BTW, if you are sick of trying to get a good angle for pitcures and you need to keep going in and out of it to get the one you want. Use the sprx files from post 1 and 2 that force different presets.

Here is the one attached that forces preset5, that is the angle seen in the screenshot above.

it changes to this slowly

PREVIEW2.jpg
 

Attachments

Last edited:
DeViL303 said:
That works with just the exe, I did not have it installed and I just used the exe alone to convert 8k jpeg flat map to 6x3MB BMPs at 1024x1024..

If it could be used from the command line or we can find the source it would be great to automate all that too.

The BMP would not be an issue, usage might be though. Have you tried calling it from command line?
Click to expand...
Nice that it works, but while i was writing my previous post i forgot the other day i was trying to send commands to it from command line (such the argument /help that uses to exist) but it was doing nothing :/
Evertime you call it from a bat it opens the GUI

And can you tell me if this one works for you ? (is CubeMapGen.exe made by ATI/AMD)
http://s000.tinyupload.com/index.php?file_id=09840673143710127931

Same stuff... i installed the full program from this link, and im just uploading the main .exe (is just 1.7mb)
https://gpuopen.com/archive/gamescgi/cubemapgen/

And this one really supports command line... is mentioned explicitelly in the official page :D
Command Line Interface: Allows you to use CubeMapGen as a command line tool for processing cubemaps in scripts and batch files.
Click to expand...



*It seems there was a group of programmers updating this tool unnofficially, the source code was hosted in google, and one of them had a blog where are explained the changes they was doing
https://seblagarde.wordpress.com/2012/06/10/amd-cubemapgen-for-physically-based-rendering/
https://code.google.com/archive/p/cubemapgen/downloads
 
Aight, I was gonna mess with the different presets later before doing more color variations. I also recommend the old K.I.S.S philosophy (Keep It Simple Stupid) for people otherwise most people will just not even bother with these and just wait around for other people to do all the work. Just my thought on it before people start putting in a bunch of time, of course everyone is free to do as they please but this has been made pretty easy to do already and hopefully others will join in. Thanks to all of you for the effort already put in, I for one appreciate it!!
 
sandungas said:
And can you tell me if this one works for you ? (is CubeMapGen.exe made by ATI/AMD)
http://s000.tinyupload.com/index.php?file_id=09840673143710127931

Same stuff... i installed the full program from this link, and im just uploading the main .exe (is just 1.7mb)
https://gpuopen.com/archive/gamescgi/cubemapgen/

And this one really supports command line... is mentioned explicitelly in the official page :D
Click to expand...

That works, nice one. :) It converted a cubemap to 6 images for me anyway.


upload_2020-1-27_14-59-31.png
 
Last edited:
DeViL303 said:
That works, nice one. :) It converted a cubemap to 6 images for me anyway.

View attachment 23199
Click to expand...
Nice, dont install the real app btw, just for confirmation that all the functions are working fine for you while experimenting with it

But you should take a look at this, the installer contains a .pdf where are explained the available commands, im uploading it here:
http://s000.tinyupload.com/index.php?file_id=61469881573157928652

The commands are at bottom, this ones to export the cube faces individually and to rename them
-exportFacePrefix:[string=CubeFace]
Filename prefix for series of face images. Use -exportCubeFaces to export the faces after specifying this option.

-exportCubeFaces
Export cube map as a series of face images.

-exportFormat:{BMP|JPG|PNG|DDS|DIB|HDR|PFM}
Image format used when exporting cubemap to face images or cross images.
Click to expand...

Take a look at the GUI and make some tests, when the image is a rectangle (like the test i made with the map of the earth using a spherical distortion and at 4096x2048) you need to "load" it into the GUI by clicking in the button "load basemap"
But i dont see any command line option to "import basemap" :/
 
Here is another method of applying a texture for anyone interested. Find a 3d 360 degree flattened out texture. Resize it so the width is twice as wide as the height.

1. I started with this:

flat.jpg

2. After resizing it if it needs it, then load it into the cubemap generator at https://matheowis.github.io/HDRI-to-CubeMap/ so you get this kind of thing:
cubemap.jpg


3. Run tile to qrc on it:

preview.jpg


preview2.jpg
 
You must log in or register to reply here.

Similar threads

Back
Top