PS3 [TUTORIAL] Unlock up to 8% extra total space on the PS3 internal hard drive

Discussion in 'Tutorials & Guides' started by einsteinx2, Sep 21, 2018.

By einsteinx2 on Sep 21, 2018 at 10:46 AM
  1. 18
    38
    12
    einsteinx2

    einsteinx2 Developer

    Joined:
    Sep 21, 2018
    Messages:
    18
    Likes Received:
    38
    Trophy Points:
    12
    Gender:
    Male
    Need some extra space on your internal PS3 HDD, if so then perhaps @einsteinx2 may have a genius method for unlocking upto 8% of space on your Internal HDD, while the space is more then likely allocated for performance reasons by Sony (use this guide at your own risk), some of this saved space could be considered overkill, as the creator of this method makes a valid point that some of this saved space on larger HDDs is more then some of the early model PS3 complete HDD size, so its likely some of that space allocation is a bit overkill on the part of Sony. einsteinx21 has provided a great tutorial found below explaining in detail about this recent discovery

    -STLcardsWS


    depositphotos_10075144-stock-photo-protected-hdd-chain-and-lock.jpg
    Unlock up to 8% extra total space on the PS3 internal hard drive


    • The What

      By default, even on custom firmware, the PS3 reserves 8% of your total internal hard drive space. From my searching online, I don't believe anyone has ever successfully unlocked that wasted space, so I decided to give it a shot. Turns out not only is it possible, but it's relatively easy thanks to some existing tools created by the community. This guide will explain how to reclaim that wasted space by manually modifying the metadata of the UFS2 formatted GameOS partition using Linux, as well as the potential cons (primarily performance, though I haven't experienced any performance issues yet myself).

      I believe the pros outweigh the cons though, and have been using this now without issue on my personal PS3 with a 1.5TB drive for a few days now, installing tons of games without a hitch.

      The Why


      While external drives used with the PS3 are all FAT32 formatted (or NTFS if you have CFW), the GameOS partition on the internal hard drive is formatted using the UFS2 filesystem with a layer of encryption on top.

      Like other *nix file systems such as Ext2/3/4, UFS2 can reserve part of the drive's space to only be used by the system or the root user. This is to reduce fragmentation and also prevent the drive from completely filling up, possibly freezing the computer.

      By default, UFS2 reserves 8% of any drive's free space, in this case meaning it can't be used by the PS3 for installing games. This is why when you first install that shiny new 1.5TB hard drive you will see that not only is it only actually 1.36 TB because hard drive manufacturers really love counting in base 10, but you also lose another 111.4 GB.

      Now generally having this reserved space is not a bad thing, however the problem is that as drives grow larger, the amount of wasted space becomes larger than some of the smaller PS3's entire hard drives! Clearly that much isn't needed to prevent fragmentation.

      The tunefs documentation mentions that "the file system's ability to avoid fragmentation will be reduced when the total free space, including the reserve, drops below 15%. As free space approaches zero, throughput can degrade by up to a factor of three over the performance obtained at a 10% threshold." Note that these numbers are already higher than the 8% default that all UFS tools as well as Sony use. Also note that it says as free space drops to 0%, performance may be up to 3 times slower than normal. However, it's unclear whether that only affects newly written--presumably more fragmented files--or all files. It seems like this is a worthy tradeoff, especially since if you do notice any performance issues you can simply delete some games or other data to free up the space. By changing the minimum free space, performance is not changed at all unless you choose to fill the drive completely...but at least you can now make that choice.

      UFS2 supports two write optimization modes: time and space. Time optimization is the default--and is used by the PS3--and allows for faster writes at the expense of potentially higher fragmentation (though generally only as the drive reaches capacity). Since we're allowing ourselves to fill almost the entire drive, these instructions also change the optimization mode to space. That means that the filesystem will spend more time during writes to ensure the files are less fragmented, insinuating that it will shuffle blocks around to make contiguous space.

      The tunefs documentation says that "the file system can either try to minimize the time spent allocating blocks, or it can attempt to minimize the space fragmentation on the disk. Optimization for space has much higher overhead for file writes. The kernel normally changes the preference automatically as the percent fragmentation changes on the filesystem." However, since the PS3 uses a custom operating system and this documentation is from FreeBSD, I think it's best to manually set the space option anyway. I have not noticed any performance degradation so far since making this change, though I still have plenty of free space on my drive at the moment.

      Photos
      Here are the before and after photos. They were taken immediately before removing the drives and immediately after reinserting them.
      BEFORE
      160GB_drive_before.jpg
      AFTER
      160GB_drive_after.jpg
      BEFORE
      1.5TB_drive_before.jpg
      AFTER
      1.5TB_drive_after.jpg


    • The How

      While it's possible to mount a PS3 hard drive in Linux and view its decrypted partitions, unfortunately the tunefs.ufs tool doesn't appear to work. It always complains that it can't find the superblock. However, the file command does work fine to show the UFS2 filesystem info when tested on a dump of the start of the partition.

      So instead of using the tunefs utility to change the minimum free space and optimization type, I wrote a script to manually scan and test single byte changes to a dump of the partition's superblock using the file command until I found the correct locations to change.

      I tested this on 2 hard drives of vastly different sizes: 160 GB and 1.5 TB. On both drives, I found that the superblock was located in the same location--the standard 128 block aka 65,536 byte offset. I also found that the locations to set the minimum free space percentage and optimization type were in the same place on both drives--byte 65,599 and 65,667 respectively. However, I highly recommend running the find_ps3_ufs2_byte_locations script anyway just to confirm before making any changes to your drive.

      Once you know the correct offsets, changing the values is simple. To adjust the minimum free space, simply write the percent in hex to byte 65,599--e.g. for 1% free space write 0x01 or for the default 8% write 0x08. To change the optimization type, write a hex 0x01 to byte 65,667--the default is 0x00 for time optimization.

      I think this should be possible to do as a PS3 homebrew app so that it can just be done directly on the device without even removing the drive. Unfortunately I have zero experience with writing PS3 homebrew, but maybe someone else with more experience can use this information to do it. I'd love to see this as an option in the REBUG custom firmware settings.

      Instructions

      1. Dump your eid root key using IrisMan/MultiMan/etc
      2. Setup a computer or virtual machine with Ubuntu 16.04. The rest of these steps are done on that machine. I tested using Parallels on a MacBook Pro, but it should work on just about anything as well as other distros.
      3. Clone my repository: git clone https://github.com/einsteinx2/PS3-Reclaim-HDD-Space.git
      4. Change to the new directory as we'll do all of the work there: cd PS3-Reclaim-HDD-Space
      5. Rename your eid root key file to eid_root_key.bin and place it in the PS3-Reclaim-HDD-Space directory
      6. Generate your hdd keys: ./ps3hdd_keygen.sh
      7. Become root since most of this requires it: sudo -s
      8. Find the device name: fdisk -l (In my case, using an external USB enclosure, it was /dev/sdb)
      9. Make virtual byte swapped encrypted device
        1. If you have a drive 1TB or less (not confirmed the exact limit): ./makedev bswap16.512 /dev/sdb
        2. If you have a drive larger than 1TB (or maybe it's 1TB and larger, I don't have a 1TB drive to test): ./makedev bswap16.1024 /dev/sdb
      10. Create decrypted device: cryptsetup create -c aes-xts-plain64 -d ./hdd_key.bin -s 256 ps3hdd_crypt /dev/nbd0
      11. Map decrypted partitions: ./kpartx -a /dev/mapper/ps3hdd_crypt
      12. View decrypted partitions (ps3hdd_crypt2 is the UFS2 GameOS partition): ls -la /dev/mapper/
      13. View current free space: [ -d /mnt/PS3GameOS ] || mkdir /mnt/PS3GameOS && mount -t ufs -o ufstype=ufs2,ro /dev/mapper/ps3hdd_crypt2 /mnt/PS3GameOS && df -h | grep "Avail\|ps3hdd_crypt2" && umount /mnt/PS3GameOS
      14. Dump the superblock of the GameOS partition: dd if=/dev/mapper/ps3hdd_crypt2 bs=512 count=256 of=GameOS_superblock.img
      15. Confirm the seek values for the next 2 commands: ./find_ps3_ufs2_byte_locations GameOS_superblock.img
      16. Set minimum free space to 1%: printf '\x01' | dd of=/dev/mapper/ps3hdd_crypt2 bs=1 seek=65599 count=1 conv=notrunc
      17. Set optimization type to "space": printf '\x01' | dd of=/dev/mapper/ps3hdd_crypt2 bs=1 seek=65667 count=1 conv=notrunc
      18. View the now larger free space: mount -t ufs -o ufstype=ufs2,ro /dev/mapper/ps3hdd_crypt2 /mnt/PS3GameOS && df -h | grep "Avail\|ps3hdd_crypt2 && umount /mnt/PS3GameOS
      19. Disconnect device: kpartx -d /dev/mapper/ps3hdd_crypt && cryptsetup remove ps3hdd_crypt && ./stop-nbd0
      20. Pop the drive back in your PS3 and enjoy the extra space! Note that I left 1% reserved space rather than going all the way to 0% to ensure that the drive never completely fills up, as I'm unsure what problems that would cause for the PS3's operating system.

    • Source code

      For ease of use, this repo contains precompiled binaries for Ubuntu 16.04 64bit. If you need or prefer to compile yourself, here are the tools used:

      bswap16: https://github.com/sguerrini97/nbdcpp (note that for >1TB drives you must change <unsigned BS=512> to <unsigned BS=1024>)

      kpartx: https://git.opensvc.com/multipath-tools/.git/

      ps3hdd_keygen.sh: http://www.psx-place.com/threads/hdd-keys-generating-scripts.10610/page-2#post-125197


    • Credits

      I would never have figured this out if it weren't for others' hard work.

      Huge thanks to Berion at PSX-Place for the hdd key generation script as well as pointing me to the information on mounting a PS3 HDD in Linux. His post here contains the script and the link: http://www.psx-place.com/threads/hdd-keys-generating-scripts.10610/page-2#post-125197

      Huge thanks to sguerrini97 at Playstation Hax for implemnenting PS3 hard drive mounting support for modern Linux kernels. Here's the post about it: https://playstationhax.xyz/forums/topic/4671-mounting-ps3-hdd-on-newer-linux-kernels and the GitHub repo: https://github.com/sguerrini97/nbdcpp.

      Thanks to dsroche for writing the original nbdcpp implementation that sguerrini97 forked, and thanks to Glevand for the original work on mounting the PS3 hard drive in OtherOS and for the great information here on the PS3 dev wiki: http://www.psdevwiki.com/ps3/Mounting_HDD_on_PC. Also thanks to anyone else that worked on PS3 hard drive mounting or anything else I'm not aware of.

     
    Last edited: Sep 26, 2018
    T.A.U, Rommy667, Zazenora and 15 others like this.

Comments

Discussion in 'Tutorials & Guides' started by einsteinx2, Sep 21, 2018.

    1. sandungas
      sandungas
      Very interesting tutorial, welcome to the forum :)
    2. Louis Garry
      Louis Garry
      I hope too
      einsteinx2 likes this.
    3. einsteinx2
      einsteinx2
      Thanks :)

      I know this can definitely be done from OtherOS Linux, as there are tools for mounting the drive in a similar way, so I don't see why it couldn't be done from within CFW. I like trying out new development environments, so unless someone ends up doing it first, when I have a spare minute soon I'll get a PS3 homebrew build environment set up and try and see if I can get something together. Since the drive is already mounted, the only part I'll need to focus on will be writing the correct bytes, but without having the standard linux tools I'll have to see how simple it is.
      esc0rtd3w and Louis Garry like this.
    4. Berion
      Berion
      Script is not proper for Fat models. I have misinterpreted wiki informations. For all hacked PS3, user should always choose Slim option. I'll fix this in this weekend and refactorize it little.

      And thanks for interesting material! I have also idea to simplify mounting by cryptmount and his preconfiguration (but I'm not sure if byteswap16 byte swapping bytes on the fly, ;) if yes, then it is possible to reduce steps).
      Algol, esc0rtd3w and einsteinx2 like this.
    5. einsteinx2
      einsteinx2
      Sounds great! And thanks for your script, it really helped me out :) Once you have it modified I'll update the repository with the new script. Also I'm almost positive that byteswap16 is doing it on the fly, just like the decryption in cryptsetup is on the fly.

      Also I have a suspicion that a similar, but more complicated, technique could also be used to essentially manually format a 2TB PS3 drive to use the full 2TB. That's my next project. I'm going to play around with manually modifying partition tables and the GameOS partition superblock to change the partitions sizes as well as moving partition 3 (the small FAT32 partition) to the end of the drive. My first test is to format a 40GB drive on my PS3, image it to my 160GB drive, make sure it still works, then see if I can manually modify the drive to expand it to 160GB. If that works, the same technique should work for 2TB drives as well, which will be my next test. The hard limit is likely 2TB as the PS3 uses MBR with 512 byte sectors, so hard limit on the drive size as it's unlikely we can get it to work with larger sector sizes (it's worth at least trying, though there are currently no drives available greater than 2TB that physically fit in the console).

      I'll post more once I've made some progress on that front. Or maybe I'll make a thread for ongoing progress so others can chime in.
      Last edited: Sep 21, 2018
      Algol, STLcardsWS, esc0rtd3w and 2 others like this.
    6. Berion
      Berion
      No problem, Your welcome. :)

      - - -

      I used "lsblk" instead of "fdisk -l" because it doesn't need root privileges and it's in all modern distros (and less writing ;p).

      I have tried deployed makedev but it doesn't work so I used one from Your repo ("don't know how to make device"). Whoever, nor "bswap16.512" nor "bswap16.1024" works with my Mint 18.3, so I complied it from the source. After that, I got "device is ready at /dev/nbd0".

      Then I tried create mapper at nbd0 but I don't know if it works or not. No confirmation, anything. BTW: If there is any occupied loop slot, createstup failed talking about size beyond bound or something like that (so this kill the idea to use LiveCDVD/USB).

      kpartx also doesn't complain but nothing happened, and I see nothing listing /dev/mapper besides ".", ".." and "ps3hdd_crypt -> ../dm-0" (whatever it means...). Trying listing /dev/mapper/ps3hdd_crypt gave me literally nothing.

      Code:
      lsblk
      sudo '/sbin/MAKEDEV' '/home/przemek/ps3/elfs/bswap16' /dev/sda
      sudo makedev bswap16 /dev/sda
      sudo '/home/przemek/ps3/elfs/makedev' '/home/przemek/ps3/elfs/bswap16' /dev/sda
      sudo cryptsetup create -c aes-xts-plain64 -d /home/przemek/ps3/hdd_key.bin -s 256 ps3hdd_crypt /dev/nbd0
      sudo kpartx -a /dev/mapper/ps3hdd_crypt
      sudo ls -la /dev/mapper/
      
      So, I'm stuck... :|

      /dev/sda is of course my PS3 HDD from CECHL04, key is also proper.

      I have tried making control dump from nbd0 but I got junk (maybe because of bswap?). I remember that first sectors of this HDD is unencrypted.

      - - -

      I have updated the script to v1.0.
      Last edited: Sep 21, 2018
      Algol, mr_ota and esc0rtd3w like this.
    7. einsteinx2
      einsteinx2
      I wasn't too worried about the sudo since all the commands afterward need it, however, if it's more available across distros then that's definitely a big advantage. However, I was under the impression that fdisk was pretty much standard for Linux. Is it not included in Mint Linux?

      Interesting, there must be some dependency or something in how it was compiled that only works in Ubuntu. I thought Mint was based on Ubuntu/Debian though, so I'm surprised there are so many differences. Oh well.

      How large is your drive? If I recall correctly, I had exactly this experience with my 1.5TB drive. It was because bswap16 hard codes a 512 byte block size. On a hunch, I tried changing the source code to 1024 bytes instead and it worked first try after that. That's why I created the two versions "bswap16.512" and "bswap16.1024". So if you have a large drive (not sure the actual limit, but worth a shot if it's larger than 500GB), change line 9 of bswap16.cpp to say "template <unsigned BS=1024>" and recompile. That fixed it for me. If it still doesn't work, then try setting up an Ubuntu 16.04 VM and using my instructions exactly and see if you still have issues, just to rule out an issue with the distro.

      Sweet! Where can I download the updated version?
      Algol and mr_ota like this.
    8. Berion
      Berion
      It is included, it is just my habit. I'm newcomer in Linux world so I'm not stick too much to old ways. ^^

      Yeah, this is what I hate in Linux's, and this is the reason why software is source distributed rather than binaries. Everything is ok if the application is something which millions of peoples uses, but the problems start appearing when we have something born in dark space of most dark minds and something stop works in newer distributions/kernels (I'm not a programmer but graphic artist, so for me it is a real pain ;p). Mint is based on Ubuntu LTSes. Yeah, should works, in theory. :D

      80GiB (yes, 80). To be honest, I'm not interesting much in size mods (I'm no longer playing games on PS3), more to just mount it and read something from it. But I found Your tutorial fit to my needs, at least to some point, and my problems cover up possible problems for someone who also would going though those steps. ^^ That's the first validation procedure part, at the end I want achieve -rw compiled kernel and looking for something much easier than writing in terminal another Homer's Illiad (oh, and also creating install package). But first things first. :}

      You can download it from the same place as old one, I just added the new post with new attachment.
      Algol likes this.
    9. einsteinx2
      einsteinx2
      Yeah that's definitely an annoying thing about Linux. Sounds like it's possible that the kernel in your version of Mint is too new or old or something and no longer compatible, or maybe it's something totally different. I'm honestly not even sure if this will work in newer/older versions of Ubuntu, that's why I specified 16.04 in the tutorial. Unfortunately my time is pretty limited, so getting this working in other distros will have to be left as an exercise to the reader.
    10. tinostar91
      tinostar91
      Even if you don't succeed with the 2 TB disk, just being able to easily swap drives for bigger ones would've been amazing. Good luck with that.
      einsteinx2 likes this.
    11. einsteinx2
      einsteinx2
      First simple test of imaging a smaller drive to a larger drive didn’t work, I wonder if the PS3 is somehow including the drive serial number or other info in the metadata that needs to be modified. I’m going to format the 160GB and dump it to compare the partition headers and other metadata with the 40GB drive.

      But I still feel confident this is possible, even if it takes a while to reverse engineer. I really want it personally, it’s ridiculous that my PS2 can have a larger drive than my PS3! So I’ll just keep picking at it til I get somewhere.

      Not to mention that if I can figure out how to edit whatever info it needs to image between drives it would allow for keeping a raw copied backup drive on hand that could be swapped in if your drive ever dies or gets corrupted. I have a second 1.5TB drive that I essentially got for free due to a shipping screw up that I was planning to use that way, but now it seems that isn’t possible without more tweaking.
    12. Berion
      Berion
      HDD fw version + serial number is written in "/dev_flash2/etc/xRegistry.sys". Also You can find it in internal flash (I don't remember the details now, maybe I'm even wrong).

      About: my/our problem. I know, I'll figure this out. I was just lazy, so I have used my environment instead of Your tested-proof. I think there is something wrong with decryption. I have compared few MB dumps of sda, byte swapped sda (works) and nbd0 (different data but still unreadable chaos which should be decrypted, right?). Could You attach 2MiB from nbd0 (of course after attach to it PS3 HDD)?

      Oh, and is worth to mention that this cannot be done on HAN, because there is no known way to retrieve EID Root Key. This feature stick to CFW only for at least time of key retrieving.
      Last edited: Sep 22, 2018
    13. einsteinx2
      einsteinx2
      Good to know about the xRegisry file thanks! That will give me a good head start on where to look.

      I’m not near my computer this weekend, but Monday I can dump the data for you from one of my drives.

      Also thanks for the clarification on HAN compatibility.
      Last edited: Sep 22, 2018
    14. littlebalup
      littlebalup
    15. einsteinx2
      einsteinx2
      Interesting. Then in that case, it must only be checking it against the info stored in that xRegistry.sys file then which I should be able to modify which is great.
    16. Zar
      Zar
      great post, thanks.

      I'm suggesting you to build just one script to do every steps.
      Algol and STLcardsWS like this.
    17. Berion
      Berion
      @Zar Can be done but /dev is something which user must choose, as everyone have different environment and mistake would be fatal. :) The same is with user dir, everyone have it's own and $home is not always in system variables.
      einsteinx2 likes this.
    18. einsteinx2
      einsteinx2
      Exactly. Right now I haven’t tested this enough and the dd command can be very dangerous if it’s accidentally used on the wrong disk.

      In any case, this is more of a proof of concept and intentionally in a sense restricted to people with enough technical knowledge to ensure they don’t do anything too wrong and can fix any issues they run across if it doesn’t work perfectly so I didn’t want to provide a premade VM image and automatic scripts to make it too easy.

      I’m currently working on writing a utility that will run from the XMB that will be faster, safer, and easier to use for those that are put off by a guide this technical. I have a pretty good idea of how to write it, but I’ve never written any PS3 homebrew (though I’ve done some other homebrew dev) so I’m not sure how long it’ll take. I’ll post more info on that when I get a proof of concept app working.
      Last edited: Sep 22, 2018
      Louis Garry likes this.
    19. uyjulian
      uyjulian
      The dd command can be dangerous, but if you follow safety tips, like copy-pasting straight from lsblk, not running dd as root, and chmodding a+rw to the specific disk, you should be fine.
      einsteinx2 and Algol like this.

Share This Page