PS3 un7zip library (PSL1GHT)

[Charles_n_town]

Glad it helps, also @Charles_n_town there is also a experimental version with even higher gcc support (v13) at
https://github.com/humbertodias/ps3toolchain
but not sure if anyone has tested it yet so not sure if it works or not, but if you find you need an even newer compiler its also a option.
There is also a gcc9 version for ArchLinux too
https://www.psx-place.com/threads/i-updated-the-ps3toolchain-to-gcc-9-5-0-binutils-2-40.39786/
anyway thats just a couple more options if you ever need newer gcc versions. But Bucanero's build has worked for anything i have compiled so far

the only homebrew I've seen weird issues with, when updating the toolchain, is sm64. I don't currently have a hacked ps3, so I have to test most stuff in rcps3, but that's not always accurate. as for sm64, something got broken with at least gcc 14 or post 13.2 iirc, so if you've updated, either downgrade or wait for fixes for the various versions out there. a lot of those ports have become abandoned, since the decomp came out so long ago, so you may be waiting a long time. I usually try not to update or downgrade unless there's a problem with something, since so much gets broken, but thanks anyway. that's actually why I use five distros. lol anyway, I'll look into those versions if I have future issues with something. I think I may have tried bucanero's before, but I'm not 100% sure. I remember reading one of his posts where he said he does most of his development on a toolchain from 2020, since black screens when booting the homebrew and what not seemed to be prevalent with some of the latter versions.

 

[bucanero]

on the master branch of PSL1GHT some changes were added to the RSX code, and that seems to bring a lot of issues with stuff like the tiny3d graphics library. Things will compile, but the binaries will just give you a black screen.

So I keep using the "2020" toolchain + psl1ght build I did, that works fine when building my homebrew apps.

I don't really need any special features from newer GCCs (that would probably make sense if you want to use C++ and some of the newer standards). My code is pure C and I try to keep it as simple as possible so I can move code around different platforms (psp, ps2, vita, or whatever else)

 

[Charles_n_town]

Yeah, that's what happened to me with sm64. Later versions of the toolchain caused the game to black screen unless you had like a save for the game or something already on the system. Really weird bug.

 

[Devildwarf]

Yeah i still use the 2020 toolchain from bucanero (i have a linux mint VM with it installed and also a windows cygwin build i compiled myself after a lot of trial and error) because i also believe if it isn't broke don't fix it, and i have had no issues using the 2020 build (compiled a few things like ScummVM, ResidualVM, SM64 and a few other things and all have worked fine).
And those bugs do sound odd, i guess a lot of changes broke things with newer builds

 

[Charles_n_town]

Yeah i still use the 2020 toolchain from bucanero (i have a linux mint VM with it installed and also a windows cygwin build i compiled myself after a lot of trial and error) because i also believe if it isn't broke don't fix it, and i have had no issues using the 2020 build (compiled a few things like ScummVM, ResidualVM, SM64 and a few other things and all have worked fine).
And those bugs do sound odd, i guess a lot of changes broke things with newer builds

Mania had a crashing issue that I thought was being caused by the toolchain. It would crash at boot, but it was actually SDL2 that was causing the problem. That has since been fixed.

 

[Siverslayer]

Here's my library to extract .7z files on the PS3

Since the linux p7zip was not an option due to the pthread requirement, I kept looking around for alternatives...
So, I found an Android implementation (AndroidUn7zip) that was clean from threads, and ported it to PSL1GHT.

Here's the Github repo:
https://github.com/bucanero/libun7zip

You can find a basic example here: https://github.com/bucanero/libun7zip/tree/master/example

@aldostools @Zar I guess that you can add .7z support to MGZ and Irisman too

Is there a limit on the file size that the library can handle for extraction? I have tried working with a 7GB file, and the program always crashes, while smaller files work well and are extracted quickly.

 

[bucanero]

Is there a limit on the file size that the library can handle for extraction? I have tried working with a 7GB file, and the program always crashes, while smaller files work well and are extracted quickly.

from what I remember, the library code is really simple and actually unpacks everything in memory before saving to disk.
So if the 7z archive has a huge file inside (like a 1 GB file) it will crash for sure, as the PS3 will run out of memory.

But in theory, if the 7z archive is huge (like 10 GB) yet full of small files (like 5 Mb files inside) then it should handle everything alright, like unpack each file from the big archive.

For unpacking huge files, I suggest to try the unRAR library that I ported from the original unRAR source by RAR Labs. That library is the full-fledged unrar tool from the original RAR creators, and can handle everything (for example you should be able to handle password-protected RARs too with the library)

 

[Siverslayer]

from what I remember, the library code is really simple and actually unpacks everything in memory before saving to disk.
So if the 7z archive has a huge file inside (like a 1 GB file) it will crash for sure, as the PS3 will run out of memory.

But in theory, if the 7z archive is huge (like 10 GB) yet full of small files (like 5 Mb files inside) then it should handle everything alright, like unpack each file from the big archive.

For unpacking huge files, I suggest to try the unRAR library that I ported from the original unRAR source by RAR Labs. That library is the full-fledged unrar tool from the original RAR creators, and can handle everything (for example you should be able to handle password-protected RARs too with the library)

The attempt didn't succeed, and I'm not sure if it's because of the compression method or not. However, I tried to decompress a 1GB file, but it didn't work.(Rar)

 

[Charles_n_town]

The attempt didn't succeed, and I'm not sure if it's because of the compression method or not. However, I tried to decompress a 1GB file, but it didn't work.

I believe he means it will decompress one file at a time, but if a file is too large in the .7z it will fail, because it's extracted to ram first. The PS3 only has like 256MBs of useable ram.

 

[Siverslayer]

I believe he means it will decompress one file at a time, but if a file is too large in the .7z it will fail, because it's extracted to ram first. The PS3 only has like 256MBs of useable ram.

I meant RAR but forgot to mention that. I will create a compression program that splits files differently, not using the `name.xx1` method, etc.

 

[Charles_n_town]

I meant RAR but forgot to mention that. I will create a compression program that splits files differently, not using the `name.xx1` method, etc.

maybe it is decompressing the entire .7z/.rar before it extracts the files? I dunno. I've never tested it, but I'd think, based on how @bucanero described it, this is more for something like roms, something small and quick.

 

[Siverslayer]

maybe it is decompressing the entire .7z/.rar before it extracts the files? I dunno. I've never tested it, but I'd think, based on how @bucanero described it, this is more for something like roms, something small and quick.

Alright, let me explain what's going on:
When you have a compressed file, for example, XX.7z, and its size is less than or equal to the available RAM, it will be successfully decompressed. Otherwise, it will fail.
Let's assume that:
We have a large file, and we divided it normally into several parts, but our library cannot handle them. So, we will create a program to compress them in a slightly different manner with these conditions:

1. Split files into parts smaller than or equal to 150 MB.
2. Each file will have the format filename_number.7z, and so on.

However, this will create a new problem: each decompressed file will have the same name, and the new file will replace the old one. Therefore, we need to assume a new approach.
For example, if we have a game named A with a size of 40 GB, what will we do?
We will invent our own compression method. For example, I created a compression program with the following conditions:

1. Any file larger than 150 MB should be split into parts smaller than 150 MB and should be suitable for MultiMan to merge them into one file.
2. You cannot repeat the name of any compressed file in any other archive.
3. Gather files that total up to 150 MB. If a file makes them exceed 150 MB, find other files that fit its size or make it into a single archive.

 

[Charles_n_town]

Alright, let me explain what's going on:
When you have a compressed file, for example, XX.7z, and its size is less than or equal to the available RAM, it will be successfully decompressed. Otherwise, it will fail.
Let's assume that:
We have a large file, and we divided it normally into several parts, but our library cannot handle them. So, we will create a program to compress them in a slightly different manner with these conditions:

1. Split files into parts smaller than or equal to 150 MB.
2. Each file will have the format filename_number.7z, and so on.

However, this will create a new problem: each decompressed file will have the same name, and the new file will replace the old one. Therefore, we need to assume a new approach.
For example, if we have a game named A with a size of 40 GB, what will we do?
We will invent our own compression method. For example, I created a compression program with the following conditions:

1. Any file larger than 150 MB should be split into parts smaller than 150 MB and should be suitable for MultiMan to merge them into one file.
2. You cannot repeat the name of any compressed file in any other archive.
3. Gather files that total up to 150 MB. If a file makes them exceed 150 MB, find other files that fit its size or make it into a single archive.

seems complicated to do what you're thinking if you're meaning compressing parts of a file, with each having a unique name, then combining them. I think the split files on the ps3 use something like the copy /b command to merge them on the system, so maybe splitting the file is a better idea. splitting and compressing together, I dunno how to do that.

 

[Siverslayer]

seems complicated to do what you're thinking if you're meaning compressing parts of a file, with each having a unique name, then combining them. I think the split files on the ps3 use something like the copy /b command to merge them on the system, so maybe splitting the file is a better idea. splitting and compressing together, I dunno how to do that.

I just did that, and it's a very successful idea. It's true that if you modify the library to handle memory, it would be better, but I'm not experienced enough to do that.
However, in general, this method is aimed at people with limited internet access. Many of them reach out and ask for a solution, so I believe this is the best solution