PS3 [TUTORIAL] Unlock up to 8% extra total space on the PS3 internal hard drive

Update: picard (a.k.a. 3141card) has released a homebrew app (available here) that changes the reserved space from 8% to 1% automatically. Running the app again will revert the changes.

https://mega.nz/file/zdk1lKqZ#aj4WFdOeosH_ncIB5flNtUxPs341PSYaXdLF2DdA45A

PS3 Unlock HDD Space by picard (aka 3141card)
All credits to einsteinx2 on psx-place.com for his great tutorial:
https://www.psx-place.com/threads/t...l-space-on-the-ps3-internal-hard-drive.20773/​

HOWTO:

• Install the package on your custom firmware PS3 and run it. It finds the active superblock of the gameOS partition and do the necessary patches. Usually 8 percent of the memory is reserved, the app sets this value to 1 percent.
  
• The optimization value is set to 1, for minimize disk fragmentation.
  
• The changes are permanent. The changed active superblock is written to the HDD with the system fs umount of the gameOS partition.
• Running the app again restores the original values, if desired.

You use this app at your own risk! I am not responsible for any damage caused to your PS3 system!​

If my PS3 app was useful to you, I would be very happy about a donation
https://www.paypal.com/paypalme/3141card

picard (aka 3141card)

-- (Original Post Continued Below) --

Need some extra space on your internal PS3 HDD, if so then perhaps @einsteinx2 may have a genius method for unlocking upto 8% of space on your Internal HDD, while the space is more then likely allocated for performance reasons by Sony (use this guide at your own risk), some of this saved space could be considered overkill, as the creator of this method makes a valid point that some of this saved space on larger HDDs is more then some of the early model PS3 complete HDD size, so its likely some of that space allocation is a bit overkill on the part of Sony. einsteinx21 has provided a great tutorial found below explaining in detail about this recent discovery​

-STLcardsWS ​

Unlock up to 8% extra total space on the PS3 internal hard drive
​

About The How + Instructions Source Code Credits

• 
  The What
  
  By default, even on custom firmware, the PS3 reserves 8% of your total internal hard drive space. From my searching online, I don't believe anyone has ever successfully unlocked that wasted space, so I decided to give it a shot. Turns out not only is it possible, but it's relatively easy thanks to some existing tools created by the community. This guide will explain how to reclaim that wasted space by manually modifying the metadata of the UFS2 formatted GameOS partition using Linux, as well as the potential cons (primarily performance, though I haven't experienced any performance issues yet myself).
  
  I believe the pros outweigh the cons though, and have been using this now without issue on my personal PS3 with a 1.5TB drive for a few days now, installing tons of games without a hitch.
  
  The Why
  
  While external drives used with the PS3 are all FAT32 formatted (or NTFS if you have CFW), the GameOS partition on the internal hard drive is formatted using the UFS2 filesystem with a layer of encryption on top.
  
  Like other *nix file systems such as Ext2/3/4, UFS2 can reserve part of the drive's space to only be used by the system or the root user. This is to reduce fragmentation and also prevent the drive from completely filling up, possibly freezing the computer.
  
  By default, UFS2 reserves 8% of any drive's free space, in this case meaning it can't be used by the PS3 for installing games. This is why when you first install that shiny new 1.5TB hard drive you will see that not only is it only actually 1.36 TB because hard drive manufacturers really love counting in base 10, but you also lose another 111.4 GB.
  
  Now generally having this reserved space is not a bad thing, however the problem is that as drives grow larger, the amount of wasted space becomes larger than some of the smaller PS3's entire hard drives! Clearly that much isn't needed to prevent fragmentation.
  
  The tunefs documentation mentions that "the file system's ability to avoid fragmentation will be reduced when the total free space, including the reserve, drops below 15%. As free space approaches zero, throughput can degrade by up to a factor of three over the performance obtained at a 10% threshold." Note that these numbers are already higher than the 8% default that all UFS tools as well as Sony use. Also note that it says as free space drops to 0%, performance may be up to 3 times slower than normal. However, it's unclear whether that only affects newly written--presumably more fragmented files--or all files. It seems like this is a worthy tradeoff, especially since if you do notice any performance issues you can simply delete some games or other data to free up the space. By changing the minimum free space, performance is not changed at all unless you choose to fill the drive completely...but at least you can now make that choice.
  
  UFS2 supports two write optimization modes: time and space. Time optimization is the default--and is used by the PS3--and allows for faster writes at the expense of potentially higher fragmentation (though generally only as the drive reaches capacity). Since we're allowing ourselves to fill almost the entire drive, these instructions also change the optimization mode to space. That means that the filesystem will spend more time during writes to ensure the files are less fragmented, insinuating that it will shuffle blocks around to make contiguous space.
  
  The tunefs documentation says that "the file system can either try to minimize the time spent allocating blocks, or it can attempt to minimize the space fragmentation on the disk. Optimization for space has much higher overhead for file writes. The kernel normally changes the preference automatically as the percent fragmentation changes on the filesystem." However, since the PS3 uses a custom operating system and this documentation is from FreeBSD, I think it's best to manually set the space option anyway. I have not noticed any performance degradation so far since making this change, though I still have plenty of free space on my drive at the moment.
  
  Photos
  Here are the before and after photos. They were taken immediately before removing the drives and immediately after reinserting them.
  

  BEFORE	AFTER
  BEFORE	AFTER

• 
  
  The How
  
  While it's possible to mount a PS3 hard drive in Linux and view its decrypted partitions, unfortunately the tunefs.ufs tool doesn't appear to work. It always complains that it can't find the superblock. However, the file command does work fine to show the UFS2 filesystem info when tested on a dump of the start of the partition.
  
  So instead of using the tunefs utility to change the minimum free space and optimization type, I wrote a script to manually scan and test single byte changes to a dump of the partition's superblock using the file command until I found the correct locations to change.
  
  I tested this on 2 hard drives of vastly different sizes: 160 GB and 1.5 TB. On both drives, I found that the superblock was located in the same location--the standard 128 block aka 65,536 byte offset. I also found that the locations to set the minimum free space percentage and optimization type were in the same place on both drives--byte 65,599 and 65,667 respectively. However, I highly recommend running the find_ps3_ufs2_byte_locations script anyway just to confirm before making any changes to your drive.
  
  Once you know the correct offsets, changing the values is simple. To adjust the minimum free space, simply write the percent in hex to byte 65,599--e.g. for 1% free space write 0x01 or for the default 8% write 0x08. To change the optimization type, write a hex 0x01 to byte 65,667--the default is 0x00 for time optimization.
  
  I think this should be possible to do as a PS3 homebrew app so that it can just be done directly on the device without even removing the drive. Unfortunately I have zero experience with writing PS3 homebrew, but maybe someone else with more experience can use this information to do it. I'd love to see this as an option in the REBUG custom firmware settings.
  
  Instructions
  
  
  1. Dump your eid root key using IrisMan/MultiMan/etc
  2. Setup a computer or virtual machine with Ubuntu 16.04. The rest of these steps are done on that machine. I tested using Parallels on a MacBook Pro, but it should work on just about anything as well as other distros.
  3. Clone my repository: git clone https://github.com/einsteinx2/PS3-Reclaim-HDD-Space.git
  4. Change to the new directory as we'll do all of the work there: cd PS3-Reclaim-HDD-Space
  5. Rename your eid root key file to eid_root_key.bin and place it in the PS3-Reclaim-HDD-Space directory
  6. Generate your hdd keys: ./ps3hdd_keygen.sh
  7. Become root since most of this requires it: sudo -s
  8. Find the device name: fdisk -l (In my case, using an external USB enclosure, it was /dev/sdb)
  9. Make virtual byte swapped encrypted device
     1. If you have a drive 1TB or less (not confirmed the exact limit): ./makedev bswap16.512 /dev/sdb
     2. If you have a drive larger than 1TB (or maybe it's 1TB and larger, I don't have a 1TB drive to test): ./makedev bswap16.1024 /dev/sdb
  10. Create decrypted device: cryptsetup create -c aes-xts-plain64 -d ./hdd_key.bin -s 256 ps3hdd_crypt /dev/nbd0
  11. Map decrypted partitions: ./kpartx -a /dev/mapper/ps3hdd_crypt
  12. View decrypted partitions (ps3hdd_crypt2 is the UFS2 GameOS partition): ls -la /dev/mapper/
  13. View current free space: [ -d /mnt/PS3GameOS ] || mkdir /mnt/PS3GameOS && mount -t ufs -o ufstype=ufs2,ro /dev/mapper/ps3hdd_crypt2 /mnt/PS3GameOS && df -h | grep "Avail\|ps3hdd_crypt2" && umount /mnt/PS3GameOS
  14. Dump the superblock of the GameOS partition: dd if=/dev/mapper/ps3hdd_crypt2 bs=512 count=256 of=GameOS_superblock.img
  15. Confirm the seek values for the next 2 commands: ./find_ps3_ufs2_byte_locations GameOS_superblock.img
  16. Set minimum free space to 1%: printf '\x01' | dd of=/dev/mapper/ps3hdd_crypt2 bs=1 seek=65599 count=1 conv=notrunc
  17. Set optimization type to "space": printf '\x01' | dd of=/dev/mapper/ps3hdd_crypt2 bs=1 seek=65667 count=1 conv=notrunc
  18. View the now larger free space: mount -t ufs -o ufstype=ufs2,ro /dev/mapper/ps3hdd_crypt2 /mnt/PS3GameOS && df -h | grep "Avail\|ps3hdd_crypt2 && umount /mnt/PS3GameOS
  19. Disconnect device: kpartx -d /dev/mapper/ps3hdd_crypt && cryptsetup remove ps3hdd_crypt && ./stop-nbd0
  20. Pop the drive back in your PS3 and enjoy the extra space! Note that I left 1% reserved space rather than going all the way to 0% to ensure that the drive never completely fills up, as I'm unsure what problems that would cause for the PS3's operating system.
• 
  Source code
  
  For ease of use, this repo contains precompiled binaries for Ubuntu 16.04 64bit. If you need or prefer to compile yourself, here are the tools used:
  
  bswap16: https://github.com/sguerrini97/nbdcpp (note that for >1TB drives you must change <unsigned BS=512> to <unsigned BS=1024>)
  
  kpartx: https://git.opensvc.com/multipath-tools/.git/
  
  ps3hdd_keygen.sh: http://www.psx-place.com/threads/hdd-keys-generating-scripts.10610/page-2#post-125197
  
  
• 
  Credits
  
  I would never have figured this out if it weren't for others' hard work.
  
  Huge thanks to Berion at PSX-Place for the hdd key generation script as well as pointing me to the information on mounting a PS3 HDD in Linux. His post here contains the script and the link: http://www.psx-place.com/threads/hdd-keys-generating-scripts.10610/page-2#post-125197
  
  Huge thanks to sguerrini97 at Playstation Hax for implemnenting PS3 hard drive mounting support for modern Linux kernels. Here's the post about it: https://playstationhax.xyz/forums/topic/4671-mounting-ps3-hdd-on-newer-linux-kernels and the GitHub repo: https://github.com/sguerrini97/nbdcpp.
  
  Thanks to dsroche for writing the original nbdcpp implementation that sguerrini97 forked, and thanks to Glevand for the original work on mounting the PS3 hard drive in OtherOS and for the great information here on the PS3 dev wiki: http://www.psdevwiki.com/ps3/Mounting_HDD_on_PC. Also thanks to anyone else that worked on PS3 hard drive mounting or anything else I'm not aware of.
  

​

 

Thanks for the share @einsteinx2 , nice find and very interesting.

 

The dd command can be dangerous, but if you follow safety tips, like copy-pasting straight from lsblk, not running dd as root, and chmodding a+rw to the specific disk, you should be fine.

You cannot do anything with dd without root permission and this doesn't change anything anyway, as mistake of identify PS3 HDD will end tragic for the user. Also, as I remember chmod works on mounted filesystems, not devices. Or maybe I don't understand the concept, so I'm curious how You imagine this.

 

HDD fw version + serial number is written in "/dev_flash2/etc/xRegistry.sys". Also You can find it in internal flash (I don't remember the details now, maybe I'm even wrong).

About: my/our problem. I know, I'll figure this out. I was just lazy, so I have used my environment instead of Your tested-proof. I think there is something wrong with decryption. I have compared few MB dumps of sda, byte swapped sda (works) and nbd0 (different data but still unreadable chaos which should be decrypted, right?). Could You attach 2MiB from nbd0 (of course after attach to it PS3 HDD)?

Oh, and is worth to mention that this cannot be done on HAN, because there is no known way to retrieve EID Root Key. This feature stick to CFW only for at least time of key retrieving.

You can write blocks on han using traditional syscalls with rop

 

@habib PS3Xploit proofed this, yes, but tutorial is about editing fs table on PC and this is what I have in mind. And this cannot be done without ERK. If we are be able to get this key on HAN it would be more than fantastic (besides the main topic, it would be awesome to keep untouched environment and have access to all data via PC).

 

To adjust the minimum free space, simply write the percent in hex to byte 65,599--e.g. for 1% free space write 0x01 or for the default 8% write 0x08.
To change the optimization type, write a hex 0x01 to byte 65,667--the default is 0x00 for time optimization.

Im wondering what happens in the next reboot of the PS3 after changing that 2 bytes... it boots normally, right ? (without any advise of warning, ike if nothing happened)

I have no idea where was located that "reserved space", but after recovering it i guess it would be nice to make some kind of "defragmentation"

Did you try to install a firmware after the change ?
And did you try to enter "recovery menu" and use the option "restore filesystem" ?

 

You cannot do anything with dd without root permission

Actually, you can, if your user and input/output device has the correct permissions.

chmod works on mounted filesystems, not devices

Chmod works on mostly everything on the filesystem.

 

You can add lot of checks at beginning of the script with optional error messages for each wrong scenario, like...

"you are not root, please run this script again from an user with root permissions"
And exit

Or...
"the hdd you are going to modify is a seagate whatever with 500GB capacity, is this correct ?"
Y/N

And so on

 

Thanks for the share @einsteinx2 , nice find and very interesting.

Thanks for turning it into an article

You can write blocks on han using traditional syscalls with rop

Very interesting. My first priority is writing a simple homebrew app for CFW that will write the necessary blocks, as that seems the most straightforward and I have a CFW console. I don't have a HAN console and that method sounds a bit more difficult that the homebrew app, so I'm probably not the one to write that version. But the info here should be enough for anyone with the knowledge to do it.

Im wondering what happens in the next reboot of the PS3 after changing that 2 bytes... it boots normally, right ? (without any advise of warning, ike if nothing happened)

I can confirm with both drives it booted right up, no message or anything and immediately displayed the free space. I have a feeling the PS3 itself doesn't really pay any attention to that setting, rather it's the USF2 filesystem driver that is handling it and just reporting the new free space.

I have no idea where was located that "reserved space", but after recovering it i guess it would be nice to make some kind of "defragmentation"

Sorry I wasn't very clear where the free space was coming from in my writeup. Basically it's not coming from anywhere. The space was in that partition all along. The filesystem driver was just choosing to report less free space, preventing from filling that partition more than 92%. Now you can fill it up to 99% percent (or 100% if you write a 0x00 instead of 0x01). It's not actually new space, it's the same space you've already been using, you just have access to more of it.

Did you try to install a firmware after the change ?
And did you try to enter "recovery menu" and use the option "restore filesystem" ?

Didn't try any of that stuff unfortunately, but I do have a few smaller drives I can run some tests with.

You can add lot of checks at beginning of the script with optional error messages for each wrong scenario, like...

"you are not root, please run this script again from an user with root permissions"
And exit

Or...
"the hdd you are going to modify is a seagate whatever with 500GB capacity, is this correct ?"
Y/N

And so on

Yep definitely. Ideally the homebrew app will solve the usability issue. Getting a script to be "full proof" like that will probably take almost as much time as just writing the homebrew haha. Once I have it written, assuming I do so successfully, I'll edit the guide to recommend that instead and leave the rest for informational purposes only.

 

Sorry I wasn't very clear where the free space was coming from in my writeup. Basically it's not coming from anywhere. The space was in that partition all along. The filesystem driver was just choosing to report less free space, preventing from filling that partition more than 92%. Now you can fill it up to 99% percent (or 100% if you write a 0x00 instead of 0x01). It's not actually new space, it's the same space you've already been using, you just have access to more of it.

Ok, my worry was if that reserved space was located at beginning of the partition and could create a "gap" after recovered, but now i get it, the reserved sectors are always located at the end
So the files that are already written in the hdd doesnt needs to be displaced/defragmented
Cool

I can confirm with both drives it booted right up, no message or anything and immediately displayed the free space. I have a feeling the PS3 itself doesn't really pay any attention to that setting

Thx, i wanted a confirmation of this, i think you are right, the PS3 doesnt complains about the 2 changed bytes at boot time

Is needed to know if there are other "boot enviroments" where the PS3 could complain and is going to try to restore them (or will display a warning about formatting the hdd entirelly)

I guess installing a firmware, and using the option from recovery for "restore fylesystem" could be two of this scenarios (and im not sure if could be more)

If the hack survives that scenarios is going to be very cool
Anyway, even if it doesnt survives, is a step in the right direction to try to find who is the responsible of setting that 2 bytes... maybe can be patched statically in a .self

Is interesting also to see if this change affects the max capacity limit of the hdd allowed by the firmware, as far i know nobody tryed to find the exact max number of sectors allowed (without interfering other firmware functions, like the "restore filesystem" maintenance procedure that imo is something very important and should not be "nuked") but maybe this hack reduced it a bit :/

 

Having looked at the open source PS3 homebrew SDK I don't see why I can't easily set the bytes a using the standard write function and the FreeBSD partition device as the file (I'm fairly certain the PS3 OS is just FreeBSD modified for their uses). It would basically be skipping right to the final "dd" steps in the tutorial since the PS3 does all the decryption and device enumeration for us. In that case if anything ever resets those bytes, it would be trivial to use the app to rewrite them.

I'll have to read up more on what install firmware and restore file system actually do, but if they involve reformatting the GameOS partition then they will likely reset this.

My main goal is to eventually get 2 TB drive support which would almost certainly not work with any of the PS3 recovery options assuming it works at all. But I was planning on keeping a backup anyway and just restoring from that (or swapping drives) if anything happened to the drive and it needed fixing, but I agree keeping compatibility with those tools would be nice.

 

We know how to patch emer_init.self to liberate free space for otherOS installation. But it imply to reformat the HDD to take effect.
Can we use that kind of method to liberate free space on disk instead of extending free space of existing partition, without need to reformat?

 

@littlebalup As I understand, this method editing filesystem table, not partition table, so those things are two different actions.

 

Berion is correct. This method only allows some you to utilize more space on the existing GameOS partition. It doesn't open up new space for use with a new partition.

However with that said...the manual partition manipulation I'm investigating to allow for the use of 2TB drives without formatting in the PS3 (no promise it will ever work or any ETA) would hypothetically allow for full control of partitioning via a Linux computer and could allow this. But expanding a partition is easier than contracting one... Also that method will be much more complicated than the method I explain here.

 

BTW:

makedev: This was included in binary form in the nbdcpp repo, so I never had the source code.

It is script, not a binary file but just ordinary text file.

 

BTW:

It is script, not a binary file but just ordinary text file.

hahaha you know I never bothered to actually check, I just made a quick assumption. I'll edit the info.

 

Dumb question but will it work with Han?

 

Dumb question but will it work with Han?

The first method discussed here is to take the hdd to pc and write in it... but the PS3 hdd is encrypted with a unique key (only valid for your PS3) and is not posible to get that key in HAN :/

The second method is by using a homebrew program (running in the PS3 so there is no need to worry about the hdd encryption) that makes the change by using system functions, but HAN cant run homebrew programs :/

But hmm... HAN can hijack and ab-use system functions, and is running in the PS3 (no need to worry about hdd encryption)

 

Btw guys good news on the PS3 app progress. I have a working development environment now and I found a fantastic starter project called PS3 GamePad Test that shows everything from how to create an XMB icon, a pkg installer, read controller inputs, and write text and graphics to the screen. I'm really busy at the moment, but later next week I'm going to try and get a proof of concept working that displays the current minimum free space and optimization settings on the drive so I can confirm I can access the data I need. Once I have that working, it'll be simple to add the ability to set it. So I'm hoping to have something to release in a couple weeks.

Also my SATA data extension cable arrived, so I can start testing with a 3.5" 2TB drive I have available to see if I can get it to use all of the space, that is once I get the proof of concept of expanding a drive image from my 40GB drive to my 160GB drive. If I can do that, I should be able to expand to any size up to 2TB. Note this would almost definitely be something that would only work on a computer with dumped HDD keys, but if I get it working, I'll make it into a bit of a more user friendly tool than just a set of instructions.

I'll update when I make more progress, or if it turns out to be a dead end and just won't work.

 

For the tests about expanding capacity you can start by reducing it with HPA in PC (part of the hdd smart functions)
The 160gb, reduce it to a "fake" 80gb... this works on PS3
Allow the PS3 to format it with that "fake" capacity, then unlock the HPA limitation and try to expand partitions

I think this is faster than doing hdd images, actually, incase you want to make images play with the HPA limits using very small sizes (for the PS3 an hdd with around 10gb capacity will be taken as valid and formatted normally, and even less)

 

That's a great idea! Part of the issue I've run into already is that an image from one drive doesn't work on another due to the drive serial number being saved in vflash (presumably).

Now I'd like to get that working as well just so I can keep a working clone of my drive (and probably build a drive cloning tool, especially if I can eventually allow users to clone their drive directly to a larger drive a fill up the extra space), however skipping that for now by formatting using a reduced size should get me to a proof of concept much faster as I can just focus on the drive resizing part which is difficult enough.

I think it's just a matter of understanding the partition table layout (there's some example code in kpartx for that) to adjust the partition sizes there, then understand better the UFS2 superblock metadata to add the additional sectors there, and then finally use dd to physically move the data of the third 2GB FAT32 partition to the end of the drive. All of that seems technically doable, especially since once I have the drive mounted and decrypted in Linux I do seem to have full access to the entire physical drive as well as each partition, including all headers and metadata. Though I'm sure it'll take me a while to get working. Should be fun though!

Do you know if that HPA tool works for all sata drives? The one I'm using for testing is a 2TB Seagate drive, but I'd also like to make sure this will work on a wide range of drives.