PS3 Latest PSL1GHT developments (PS3 OpenSDK) in 2020

There has been various people who have made improvements to psl1ght (the open source PS3SDK) in 2020 and developer @bucanero has certainly been leading that charge improving psl1ght with various implementations and additions. If you have been keeping a watch in the forum here at psx-place or some of our tweets you know dev has been busy. Added in recent we have seen unRAR library, followed by a un7zip library , then following by ar/tar.gz/tar.bz2 complete library's and libnfs/liborbisNfs library's and then to follow ibsmb2 SMBv2/v3 lib We have also seen developer @bucanero recently update Ps3LoadX (a tool originally created by Hermes (aka Estwald) in 3.55 era for developers to test code) to bring support for the latest PSL1GHT and Tiny3d code for latest firmware. We have also seen the developer create various homebrew PS3 Projects like the Apollo Save Tool which just seen a recent update to v1.4.0.

Other devs have made contributions that need a shoutout as well to developer @tps (also created PS3-Kara, a recent Karaoke app for the PS3) has also added to PSL1GHT: filesystem libraries for exFat support with a released lib and test file manager. (note Some apps like IRISMAN have adopted this support) Then we have seen developer @Crystal who has also made contributions to PSL1ght and you can see in the "guide" tab his recommendation and changes made to PSl1ght for compiling homebrew in 2020 and also bucanero has a great guide for setting up the PS3 Toolchain on a macOS so we have some recent resources from several devs that are advancing the OpenSDK

​

Recent Advancements to PSL1GHT PS3LoadX Network/File Logger Guides / Resources Powered by PSL!GHT

• 
  Recent PSL1GHT Contributions
  
  • via @bucanero
  • libsmb2 SMBv2/v3 lib (PSL1GHT)
    
  • libnfs/liborbisNfs (PSL1GHT)
  • ar/tar.gz/tar.bz2 complete lib (PSL1GHT)
  • un7zip lib (PSL1GHT)
    
  • unRAR lib (PSL1GHT)
    
  • PS3MAPI lib (PSL1GHT)
  • via @tps
    • exFAT Filesystem lib (w/ Simple Manager App & library) (PSL1GHT) (birth/development thread)
  • via @Crystal
    • Compiling open-source PS3 toolchain nowadays (in 2020) + GCC 7.2 support implementation
    • SQLite for PS3
  Update 2021:
  
  • TRE regex library for PSL1GHT
  • A BDF/FNT unicode fonts library
    
• 
  
  

  
  SEE LINKS & DISCUSSION:
  https://www.psx-place.com/threads/ps3loadx-v1-2.30665/
  
  via @bucanero This little app for developers allows to send .SELF binaries over the network to test code and speed up development of PS3 homebrew. Hermes also added other features like installing .zip files, but it wasn't updated since FW 3.55 and wasn't working with current ps3 firmwares (4.82+)
  
  So, I've ported the original PS3LoadX app by Hermes to the latest PSL1GHT & Tiny3d code, and made it compatible with the latest firmware.​
  
  
  
• 
  This is an older release from September 2019 but thought i still share it here as its releated

  ---

  .
  Simple network/file logger (+screenshot) library [PSL1GHT]​
  • Links and Discussions @: https://www.psx-place.com/threads/simple-network-file-logger-screenshot-library-psl1ght.26304/
  I've (@bucanero) been playing around trying to build some PS3 homebrew (using psl1ght and PS3HEN), and I found myself in the need of a simple debug logger library to get some information out while I was testing my binaries.
  
  With this dbglogger library you can setup network (UDP/TCP) and file logging, and it should be customizable enough to enable or disable it as needed, even at runtime.
  
  I also needed some additional helper functions, so the library includes a "screenshot" feature to save the current video buffer in PNG format on the fly, with alpha channel support (bye bye BMP files and big memory mallocs!).
  
  
• 
  Guides
  
  • [Guide] build a PS3 toolchain on macOS
  • [Guide] Compiling open-source PS3 toolchain nowadays (in 2020)
  
  Additional Resources for devs:
  
  • https://github.com/bucanero/PSL1GHT
  • https://github.com/bucanero/psl1ght-libs
  • https://github.com/bucanero/ps3toolchain
  • https://github.com/Estwald/PSDK3v2/tree/master/libraries-src/Tiny3D
  
  
• 
  This is not a complete list but highlights various projects powered by PSL1GHT OpenSDK
  
  • IRISMAN
  • ManaGunZ
    
  • RetroArch (PS3) - Latest versions have started to convert from the official SDK to the OpenSource PSL1GHT (still a WIP but is progressing)
  • FB NEO RL PLUS
    
  • and various other Homebrew

Shoutout to [COLOR=#006600]@sandungas[/COLOR] for the assist ​

 

I am sure i missed something or some other useful links could be added, See anything missed or have any suggestions, made a comment below with the suggestions

 

it'd be nice if someone fixed the speed issue psl1ght v2 still has. maybe @bucanero can address this. he has the necessary expertise for this i believe. by speed i mean that homebrew compiled with official sony sdk is much faster than homebrew compiled with psl1ght

The issue lies in the compiler somewhere, but i don't know exactly where. perhaps in crt0.S

 

it'd be nice if someone fixed the speed issue psl1ght v2 still has. maybe @bucanero can address this. he has the necessary expertise for this i believe. by speed i mean that homebrew compiled with official sony sdk is much faster than homebrew compiled with psl1ght

The issue lies in the compiler somewhere, but i don't know exactly where. perhaps in crt0.S

But how we calculate the speed ?
I was looking for some simple C code that could be compiled in both SDK's to do some benchmarks and found this, is a "fork" of the spectre exploit for powerpc. I know, the link is not related, im just dropping it because could be interesting
https://gist.github.com/kingsumos/1c718aa68ffd12ebb7899419cff8d34f

Spoiler

Forked from here ---> https://gist.github.com/ErikAugust/724d4a969fb2c6ae1bbd7b2a9e3d4bb6

 

But how we calculate the speed ?
I was looking for some simple C code that could be compiled in both SDK's to do some benchmarks and found this, is a "fork" of the spectre exploit for powerpc. I know, the link is not related, im just dropping it because could be interesting
https://gist.github.com/kingsumos/1c718aa68ffd12ebb7899419cff8d34f

Spoiler

Forked from here ---> https://gist.github.com/ErikAugust/724d4a969fb2c6ae1bbd7b2a9e3d4bb6

To verify what zeco said, I just made a hello world app with only:

int main() {
    printf("hello world");
    return 0;
}

I just compiled/linked this hello app with both SDK using default compiler options except optimisation set to -O2 then opened both elf in IDA.

Obviously, I realise that a simple printf call is not representative of all compiled code but we have to start somewhere & it gives a first idea of the potential scope of the problem.

With default compiler options & optimization set to 02, Psl1ght issued much more code to do the same thing than the official sdk.
The resulting executable code (only the executable segments) size is much larger
0x16500 bytes for psl1ght, nearly double the size of the official sdk made elf executable segments code which is only 0xCE70 bytes.
The number of total subs is of course proportionally higher too:
334 for psl1ght
171 for official sdk.

There are also many more exports in the Psl1ght made self stub table than in the official sdk made self. Am not even sure whether all those exports in the psl1ght made self are actually used by the self code.

I haven't done any benchmarking for execution speed but given the produced code, I have to assume that the entire life cycle of the hello world app compiled with psl1ght will be much longer than its official sdk counterpart.

Edit:
I can see in IDA that alongside libio functions linked for printf, psl1ght included in the self some network socket functions such as socket/accept/bind/send/recv etc.., those being seemingly used by similarly included librt functions.
There are also many sysPrxForUser exports including some for SPUPrintf threading but also functions like sysGetTemperature (?).

None of this extra stuff (network functions, SPU threading etc..) can be found in the official sdk made self, once added up, the extra code must just about make for the bulk of the total size difference.

Additionally, the official sdk implements a printf function whereas in psl1ght, the printf is automatically replaced by a puts call which makes sense here but the puts implementation appears somewhat more costly in total code size too. TBC.

As to the start/init subs that will eventually call the app's main() function, Psl1ght's init sub probably takes much longer to complete as it is the sub using all that extra stuff I mentioned, init initializes libc, memory usage, sets up synchronisation primitives etc..
It looks like the official sdk initializes a much smaller set of libc functions than psl1ght does with its librt library.

If anyone tries to benchmark a small app, it is important to keep in mind that the self launch function is the start() function which calls init() which in turn will call main() AFTER setting up the "environment". Using timers inside the main() function will only tell you about perf differences per sdk with your code, not with the self launch.
I would assume that with small amount of code, much of the perf difference would rest in the differences in the init sub.

 

what is app and what its function

 

what is app and what its function

SDK = Software Development Kit

 

what is app and what its function

This "app" is a set of tools and programming libraries used to compile homebrew applications for PS3.

It's intended for software developers and enthusiasts that want to create software or modify/compile the open source already available.

Unlike the official SDK that requires a legal license, this SDK don't use proprietary code and is free for use.

 

stdio function are also slow on psl1ght, especially with external devices. (fopen, fwrite, fread... etc.)

 

stdio function are also slow on psl1ght, especially with external devices. (fopen, fwrite, fread... etc.)

I have never really checked but of course it would be perfectly possible for the psl1ght stdio code to perform even more poorly than its official counterpart.

However those functions are already terribly slow when created on official sdk or even when using the vsh exports directly.
That's because around the corresponding syscall they can use up to 3 or 4 sets of locks, from the top of my head, Locksyslock, Mtxlock, Lockfilelock..
For every operation those get locked, unlocked, in some cases created and/or destroyed. You can get up to 10 syscalls in just one stdio call, each of them needing instructions to load all params & save outcomes.
If you disassemble fread/fwrite/fclose... you will find all this locking dance as well as a tiny bit of code dealing with the libc parameter transformation to use the syscall then the needed tweaking to return syscall results in libc format.
And in the middle of all that stuff, you will find a read/write/close... function which is the one calling the io syscall.

It's very costly & performance is terrible, especially on external devices, good enough just for dealing with the occasional small file.

 

The issue lies in the io yes. dumping nand from a psl1ght app takes ages ( 1h30 to 2h) while doing so in official sdk takes 5 minutes. this is unacceptable for a 239MB file

 

it'd be nice if someone fixed the speed issue psl1ght v2 still has. maybe @bucanero can address this. he has the necessary expertise for this i believe. by speed i mean that homebrew compiled with official sony sdk is much faster than homebrew compiled with psl1ght

The issue lies in the compiler somewhere, but i don't know exactly where. perhaps in crt0.S

oh I'm not sure that I'm such an expert on PS3 dev stuff to solve that kind of issue
one issue can be the compiler, and another one could be the PSL1GHT library itself

also, a good question would be which PSL1GHT sdk are you currently using?
the PSDKv2 build by Estwald is quite old and uses gcc 4
the latest psl1ght sdk (if you build it from scratch) is now using gcc 7

for any fix or improvement, I think we should only focus on the current/latest psl1ght sdk with gcc 7.2

btw, @bguerville your tests with "hello world" were made with the latest psl1ght, or the build by Estwald?

I also agree with Zar and the others, that psl1ght stdio is really slow, specially when using external media like USB drives. I know that Apollo is not super-optimized but for example you can try decrypting a ps2 .VME (8 Mb file) to USB, and you'll see how long it takes to write the decrypted file to the usb drive.

I wonder if the speed difference between SDKs would also be noticed with some classic CPU tests like Dhrystone or Whetstone.

 

...
one issue can be the compiler, and another one could be the PSL1GHT library itself

also, a good question would be which PSL1GHT sdk are you currently using?
the PSDKv2 build by Estwald is quite old and uses gcc 4
the latest psl1ght sdk (if you build it from scratch) is now using gcc 7

for any fix or improvement, I think we should only focus on the current/latest psl1ght sdk with gcc 7.2

btw, @bguerville your tests with "hello world" were made with the latest psl1ght, or the build by Estwald?

I also agree with Zar and the others, that psl1ght stdio is really slow, specially when using external media like USB drives.
...

The disassembled Hello World was compiled with Ps3sdkv2.

Someone could try comparing both gcc versions for perf, there might be a difference in some things, maybe in self loading times, but I would be surprised if the compiler was the cause of the perf issue with io, I could be wrong but I bet it's library based.

 

it can't be the compiler since sony's official sdk uses gcc 4.1.1 so psl1ght using the same gcc or newer should be at relatively the same speed or even faster but instead it's the opposite!

 

another example. i've finished compiling latest psl1ght and ps3toolchain from github (with gcc-7) and compiled this (https://github.com/sguerrini97/psl1ghtv2_ports/tree/master/dump_flash) and it takes more than 10 minutes to copy the flash.bin from /dev_hdd0/tmp to usb thumbdrive while with (https://github.com/sguerrini97/sdk_ports/tree/master/dump_flash) it's instantaneous and doesn't require go to to hdd

 

Starting from the axiom that an open and free sdk cannot compete with an official sdk made in close collaboration with the engineers who designed and produced a console, to compare them we should not use a printf or an access to an USB device, but a row use of 2D and 3D graphics. A comparision between a simple task tested using GCM on sdk sony and RSX on PSL1GHT. For HD filesystem access, LV2 funtions and syscall......

 

Thanks @Crystal .
Added https://www.psx-place.com/threads/sqlite-for-ps3.28412/ to main post

 

The convo was only focusing on zeco's question, whether or not psl1ght io perf could be improved. Any comparison drawn between the 2 sdk was related to that & nothing more, the official sdk perf serving only as reference.