PS3 [TUTORIAL] Unlock up to 8% extra total space on the PS3 internal hard drive

Discussion in 'Tutorials & Guides' started by einsteinx2, Sep 21, 2018.

By einsteinx2 on Sep 21, 2018 at 10:46 AM
  1. 26
    57
    12
    einsteinx2

    einsteinx2 Developer

    Joined:
    Sep 21, 2018
    Messages:
    26
    Likes Received:
    57
    Trophy Points:
    12
    Gender:
    Male
    Need some extra space on your internal PS3 HDD, if so then perhaps @einsteinx2 may have a genius method for unlocking upto 8% of space on your Internal HDD, while the space is more then likely allocated for performance reasons by Sony (use this guide at your own risk), some of this saved space could be considered overkill, as the creator of this method makes a valid point that some of this saved space on larger HDDs is more then some of the early model PS3 complete HDD size, so its likely some of that space allocation is a bit overkill on the part of Sony. einsteinx21 has provided a great tutorial found below explaining in detail about this recent discovery

    -STLcardsWS


    depositphotos_10075144-stock-photo-protected-hdd-chain-and-lock.jpg
    Unlock up to 8% extra total space on the PS3 internal hard drive


    • The What

      By default, even on custom firmware, the PS3 reserves 8% of your total internal hard drive space. From my searching online, I don't believe anyone has ever successfully unlocked that wasted space, so I decided to give it a shot. Turns out not only is it possible, but it's relatively easy thanks to some existing tools created by the community. This guide will explain how to reclaim that wasted space by manually modifying the metadata of the UFS2 formatted GameOS partition using Linux, as well as the potential cons (primarily performance, though I haven't experienced any performance issues yet myself).

      I believe the pros outweigh the cons though, and have been using this now without issue on my personal PS3 with a 1.5TB drive for a few days now, installing tons of games without a hitch.

      The Why


      While external drives used with the PS3 are all FAT32 formatted (or NTFS if you have CFW), the GameOS partition on the internal hard drive is formatted using the UFS2 filesystem with a layer of encryption on top.

      Like other *nix file systems such as Ext2/3/4, UFS2 can reserve part of the drive's space to only be used by the system or the root user. This is to reduce fragmentation and also prevent the drive from completely filling up, possibly freezing the computer.

      By default, UFS2 reserves 8% of any drive's free space, in this case meaning it can't be used by the PS3 for installing games. This is why when you first install that shiny new 1.5TB hard drive you will see that not only is it only actually 1.36 TB because hard drive manufacturers really love counting in base 10, but you also lose another 111.4 GB.

      Now generally having this reserved space is not a bad thing, however the problem is that as drives grow larger, the amount of wasted space becomes larger than some of the smaller PS3's entire hard drives! Clearly that much isn't needed to prevent fragmentation.

      The tunefs documentation mentions that "the file system's ability to avoid fragmentation will be reduced when the total free space, including the reserve, drops below 15%. As free space approaches zero, throughput can degrade by up to a factor of three over the performance obtained at a 10% threshold." Note that these numbers are already higher than the 8% default that all UFS tools as well as Sony use. Also note that it says as free space drops to 0%, performance may be up to 3 times slower than normal. However, it's unclear whether that only affects newly written--presumably more fragmented files--or all files. It seems like this is a worthy tradeoff, especially since if you do notice any performance issues you can simply delete some games or other data to free up the space. By changing the minimum free space, performance is not changed at all unless you choose to fill the drive completely...but at least you can now make that choice.

      UFS2 supports two write optimization modes: time and space. Time optimization is the default--and is used by the PS3--and allows for faster writes at the expense of potentially higher fragmentation (though generally only as the drive reaches capacity). Since we're allowing ourselves to fill almost the entire drive, these instructions also change the optimization mode to space. That means that the filesystem will spend more time during writes to ensure the files are less fragmented, insinuating that it will shuffle blocks around to make contiguous space.

      The tunefs documentation says that "the file system can either try to minimize the time spent allocating blocks, or it can attempt to minimize the space fragmentation on the disk. Optimization for space has much higher overhead for file writes. The kernel normally changes the preference automatically as the percent fragmentation changes on the filesystem." However, since the PS3 uses a custom operating system and this documentation is from FreeBSD, I think it's best to manually set the space option anyway. I have not noticed any performance degradation so far since making this change, though I still have plenty of free space on my drive at the moment.

      Photos
      Here are the before and after photos. They were taken immediately before removing the drives and immediately after reinserting them.
      BEFORE
      160GB_drive_before.jpg
      AFTER
      160GB_drive_after.jpg
      BEFORE
      1.5TB_drive_before.jpg
      AFTER
      1.5TB_drive_after.jpg


    • The How

      While it's possible to mount a PS3 hard drive in Linux and view its decrypted partitions, unfortunately the tunefs.ufs tool doesn't appear to work. It always complains that it can't find the superblock. However, the file command does work fine to show the UFS2 filesystem info when tested on a dump of the start of the partition.

      So instead of using the tunefs utility to change the minimum free space and optimization type, I wrote a script to manually scan and test single byte changes to a dump of the partition's superblock using the file command until I found the correct locations to change.

      I tested this on 2 hard drives of vastly different sizes: 160 GB and 1.5 TB. On both drives, I found that the superblock was located in the same location--the standard 128 block aka 65,536 byte offset. I also found that the locations to set the minimum free space percentage and optimization type were in the same place on both drives--byte 65,599 and 65,667 respectively. However, I highly recommend running the find_ps3_ufs2_byte_locations script anyway just to confirm before making any changes to your drive.

      Once you know the correct offsets, changing the values is simple. To adjust the minimum free space, simply write the percent in hex to byte 65,599--e.g. for 1% free space write 0x01 or for the default 8% write 0x08. To change the optimization type, write a hex 0x01 to byte 65,667--the default is 0x00 for time optimization.

      I think this should be possible to do as a PS3 homebrew app so that it can just be done directly on the device without even removing the drive. Unfortunately I have zero experience with writing PS3 homebrew, but maybe someone else with more experience can use this information to do it. I'd love to see this as an option in the REBUG custom firmware settings.

      Instructions

      1. Dump your eid root key using IrisMan/MultiMan/etc
      2. Setup a computer or virtual machine with Ubuntu 16.04. The rest of these steps are done on that machine. I tested using Parallels on a MacBook Pro, but it should work on just about anything as well as other distros.
      3. Clone my repository: git clone https://github.com/einsteinx2/PS3-Reclaim-HDD-Space.git
      4. Change to the new directory as we'll do all of the work there: cd PS3-Reclaim-HDD-Space
      5. Rename your eid root key file to eid_root_key.bin and place it in the PS3-Reclaim-HDD-Space directory
      6. Generate your hdd keys: ./ps3hdd_keygen.sh
      7. Become root since most of this requires it: sudo -s
      8. Find the device name: fdisk -l (In my case, using an external USB enclosure, it was /dev/sdb)
      9. Make virtual byte swapped encrypted device
        1. If you have a drive 1TB or less (not confirmed the exact limit): ./makedev bswap16.512 /dev/sdb
        2. If you have a drive larger than 1TB (or maybe it's 1TB and larger, I don't have a 1TB drive to test): ./makedev bswap16.1024 /dev/sdb
      10. Create decrypted device: cryptsetup create -c aes-xts-plain64 -d ./hdd_key.bin -s 256 ps3hdd_crypt /dev/nbd0
      11. Map decrypted partitions: ./kpartx -a /dev/mapper/ps3hdd_crypt
      12. View decrypted partitions (ps3hdd_crypt2 is the UFS2 GameOS partition): ls -la /dev/mapper/
      13. View current free space: [ -d /mnt/PS3GameOS ] || mkdir /mnt/PS3GameOS && mount -t ufs -o ufstype=ufs2,ro /dev/mapper/ps3hdd_crypt2 /mnt/PS3GameOS && df -h | grep "Avail\|ps3hdd_crypt2" && umount /mnt/PS3GameOS
      14. Dump the superblock of the GameOS partition: dd if=/dev/mapper/ps3hdd_crypt2 bs=512 count=256 of=GameOS_superblock.img
      15. Confirm the seek values for the next 2 commands: ./find_ps3_ufs2_byte_locations GameOS_superblock.img
      16. Set minimum free space to 1%: printf '\x01' | dd of=/dev/mapper/ps3hdd_crypt2 bs=1 seek=65599 count=1 conv=notrunc
      17. Set optimization type to "space": printf '\x01' | dd of=/dev/mapper/ps3hdd_crypt2 bs=1 seek=65667 count=1 conv=notrunc
      18. View the now larger free space: mount -t ufs -o ufstype=ufs2,ro /dev/mapper/ps3hdd_crypt2 /mnt/PS3GameOS && df -h | grep "Avail\|ps3hdd_crypt2 && umount /mnt/PS3GameOS
      19. Disconnect device: kpartx -d /dev/mapper/ps3hdd_crypt && cryptsetup remove ps3hdd_crypt && ./stop-nbd0
      20. Pop the drive back in your PS3 and enjoy the extra space! Note that I left 1% reserved space rather than going all the way to 0% to ensure that the drive never completely fills up, as I'm unsure what problems that would cause for the PS3's operating system.

    • Source code

      For ease of use, this repo contains precompiled binaries for Ubuntu 16.04 64bit. If you need or prefer to compile yourself, here are the tools used:

      bswap16: https://github.com/sguerrini97/nbdcpp (note that for >1TB drives you must change <unsigned BS=512> to <unsigned BS=1024>)

      kpartx: https://git.opensvc.com/multipath-tools/.git/

      ps3hdd_keygen.sh: http://www.psx-place.com/threads/hdd-keys-generating-scripts.10610/page-2#post-125197


    • Credits

      I would never have figured this out if it weren't for others' hard work.

      Huge thanks to Berion at PSX-Place for the hdd key generation script as well as pointing me to the information on mounting a PS3 HDD in Linux. His post here contains the script and the link: http://www.psx-place.com/threads/hdd-keys-generating-scripts.10610/page-2#post-125197

      Huge thanks to sguerrini97 at Playstation Hax for implemnenting PS3 hard drive mounting support for modern Linux kernels. Here's the post about it: https://playstationhax.xyz/forums/topic/4671-mounting-ps3-hdd-on-newer-linux-kernels and the GitHub repo: https://github.com/sguerrini97/nbdcpp.

      Thanks to dsroche for writing the original nbdcpp implementation that sguerrini97 forked, and thanks to Glevand for the original work on mounting the PS3 hard drive in OtherOS and for the great information here on the PS3 dev wiki: http://www.psdevwiki.com/ps3/Mounting_HDD_on_PC. Also thanks to anyone else that worked on PS3 hard drive mounting or anything else I'm not aware of.

     
    Last edited: Sep 26, 2018

Comments

Discussion in 'Tutorials & Guides' started by einsteinx2, Sep 21, 2018.

    1. einsteinx2
      einsteinx2
      @Zar I did some testing using the code you attached.

      I dumped the first 128KB of each HDD partition (there were 3). The second partition should be GameOS as I understand it. While the dumps worked, unfortunately it looks like I'm not getting the superblock, but rather the beginning of the partition's data, even though I'm starting from sector 0.

      I searched around for more info and found something interesting here: https://www.eurasia.nu/wiki/index.php/PS3_Hypervisor_Reverse_Engineering

      In the section called "Encryption and Decryption of Storage Devices", it shows a dump of the superblock and mentions "Superblock starts at sector 0x80".

      From reading that wiki, it looks like I need to use the the lvl1 syscalls to interact with the hypervisor rather than lvl2 which I guess is more abstracted (makes sense given the name haha).

      Presumably code similar to what you posted but using the lv1_storage_read (and eventually lv1_storage_write) functions will get me what I need.

      Definitely getting closer! All signs point to this definitely being possible.

      Off to look for a lv1_storage_read example now.

      -----------------

      Edit: I'm getting the impression that the only way to call lv1 syscalls is through the lv2_lv1_call syscall which takes in a lv1 syscall number and parameters. Some good info here: https://www.psdevwiki.com/ps3/LV2_Functions_and_Syscalls#LV2_Syscalls

      Then the list of lv1 syscalls is here: https://www.psdevwiki.com/ps3/HV_Syscalls

      And I found a possible example here: https://github.com/marcan/asbestos/blob/master/stage2/diskio.c

      ----------

      ehh nevermind looks like that last example uses lv1 syscalls directly which I can't do within GameOS. I have to figure out how to use the lv2 wrapper call instead. It seems simple enough to call the lv2syscall() functions, passing in 10 as the first parameter for lv2_lv1_call, then for example 170 as the second parameter for lv1_open_device, then the bus id and dev id.

      So far so good...

      Then it says here: https://www.psdevwiki.com/ps3/HV_Syscall_Reference#lv1_open_device that "Valid values for bus_id and dev_id can be obtained from the repository (see lv1_get_repository_node_value)" and that's where I get a bit stuck. In the reference for that syscall it seems to require a partition id and 4 keys, so I guess that means I need to get the decryption keys first and can't just access the partitions directly as with the lv2_storage calls? I guess that makes sense if I'm accessing it via the hypervisor before descryption...

      In any case, I'm gonna take a break and come back to this later with a fresh head.

      If anyone knows of any examples of GameOS code (not linux, etc) using these lv1 storage syscalls (presumably via the lv2_lv1_call syscall) that would be a huge help. Otherwise I'll keep just keep digging til I figure it out eventually (hopefully haha).
      Last edited: Feb 7, 2019
    2. x23are
      x23are
      any new progress ?
    3. Casavult
      Casavult
      Fantastic tutorial. I've been out of the scene for quite a while but I am going to get a crack on with this!
    4. PontiusPiratus
      PontiusPiratus
      Long time lurker, but the information in this thread made me register! I have had success with both my fat PS3s (NAND and NOR), and I wanted to share what I had to do to make it all work :)

      -First of all I had no success with Ubuntu 18.04. But by downloading 16.04.6 desktop and running it as a live system from USB, everything worked on the first attempt. The only small thing that needed attention was installing nbd-client: "Universe" software sources had to be added.

      -For step 9 I can confirm that bswap16.512 was correct for 1TB drives in both my systems.

      -The encryption is different for fat compared to slim. So for both my NAND and NOR, step 10 needs to be changed to:
      cryptsetup create -c aes-cbc-null -d ./hdd_key.bin -s 192 ps3hdd_crypt /dev/nbd0

      -On my NAND, then GameOS is on crypt1 rather than crypt2. Hence all commands in these steps needs to be changed accordingly:
      13, 14, 16, 17 and 18

      -For step 18 there is a missing quotation mark, see correction below
      "Avail\|ps3hdd_crypt2 --> "Avail\|ps3hdd_crypt2"

      @einsteinx2 :
      Once again thank you so much for sharing this method, wonderful stuff! I gained 63G on both my 1TB drives, very much worth the effort. Let me know if you need beta testing for your homebrew application, I will have access to these two fat NAND and NOR systems for about 4-5 months more before giving them away.
      @STLcardsWS :
      Can you update the how-to accordingly?
      Algol, Zar, Louis Garry and 2 others like this.
    5. Darknss
      Darknss
      Does this work on slims and super slims now that we have HEN? or we cant dump eid root key
      jcorrea likes this.
    6. atreyu187
      atreyu187

      Doesn't work for HEN and doubtful it will as it needs CFW lvl1 patches to obtain eID root key.
      jcorrea likes this.
    7. Berion
      Berion
      It works for all PS3 models for sure but... we don't know how to obtain ERK on consoles with fabric fw 3.60 and newer. ;)
      jcorrea likes this.
    8. Zar
      Zar
      @einsteinx2
      Sry, i took lot oftime to answer you...

      I never rly paid attention to lv1 function, so, i rly don't know how to use them...

      I saw lv1 functions in cobra/mamba payloads, maybe you can use them from the payload ?
      maybe @habib @aldostools or @Joonie can help us here ? (plz)

      Morevever the original function from these payloads come from abestos.
      For example :
      https://github.com/marcan/asbestos/blob/master/stage2/device.c
      https://github.com/Zarh/ManaGunZ/blob/master/payloads/MAMBA/lv1/src/device.c

      Also, get dev_id & bus_id, you can use find_device_by_type (it use the function lv1_read_node) but you can directly use diskio.c, it looks like it already configured for internal HDD :
      https://github.com/marcan/asbestos/blob/master/stage2/diskio.c#L41


      And fyi, here is the list of dev_id for the lv1 : https://psdevwiki.com/ps3/DeviceID ... Just in case you want to create ps2_netemu with FAT32 support ;)
    9. aldostools
      aldostools
      I haven't been following this thread, so I'm not sure what is the issue here...
      webMAN MOD allows to peek / poke lv1 using the web commands: /peek.lv1 & poke.lv1
      It also allows to dump lv1 & lv2... btw all lv2 peek/pokes in webMAN are done through lv1 syscalls (CFW only). So it needs syscalls 6/7 only for PS3HEN.

      Here is a simple example that uses syscalls 600 (sys_storage_open)
      https://github.com/aldostools/webMAN-MOD/blob/master/include/eject_insert.h
    10. Zar
      Zar
      We try to use lv1 syscalls
    11. aldostools
      aldostools
      Zar likes this.
    12. Zar
      Zar
      yeah, it's exactly what I think too...
      aldostools likes this.
    13. DotDotDot
      DotDotDot
      Hey, I'm trying to post how I got it working (compiled it myself, just wanted to post the instructions) but it seems Im unable to. Can anyone allow me to please?
      Zar likes this.
    14. x23are
      x23are
      can you post your instruction here ?
      i think you must have special privilage to post in main page of site.
    15. DotDotDot
      DotDotDot
      I was actually trying to post it here. I'll try again now and see.
    16. Zar
      Zar
      I think you must have at least 10 (?) messages to post any link. It's spam bot protection maybe @STLcardsWS can 'unlock' your account.
    17. STLcardsWS
      STLcardsWS
      No restrictions. He 9 @DotDotDot ) can create a new thread in the forum . Mainpage that has to get approved ..

      First post of a new user are moderated. ,
    18. DotDotDot
      DotDotDot
      Yeah I got the same message, which mentioned spam. The comment did have a link, the nbdcpp github.
      I guess I'll help some people around and get to the message minimumm
    19. DotDotDot
      DotDotDot
      I wasn't trying to create a new thread, just replying on this one. It seems that the problem is that I linked the nbdcpp's github and I am not allowed to post links.
    20. pinky
      pinky
      you might have to wait for the site to refresh itself. I was on a site a while back where it refreshed every 60-90 minutes.

Share This Page