PS3 [TUTORIAL] Unlock up to 8% extra total space on the PS3 internal hard drive

Update: picard (a.k.a. 3141card) has released a homebrew app (available here) that changes the reserved space from 8% to 1% automatically. Running the app again will revert the changes.

https://mega.nz/file/zdk1lKqZ#aj4WFdOeosH_ncIB5flNtUxPs341PSYaXdLF2DdA45A

PS3 Unlock HDD Space by picard (aka 3141card)
All credits to einsteinx2 on psx-place.com for his great tutorial:
https://www.psx-place.com/threads/t...l-space-on-the-ps3-internal-hard-drive.20773/​

HOWTO:

• Install the package on your custom firmware PS3 and run it. It finds the active superblock of the gameOS partition and do the necessary patches. Usually 8 percent of the memory is reserved, the app sets this value to 1 percent.
  
• The optimization value is set to 1, for minimize disk fragmentation.
  
• The changes are permanent. The changed active superblock is written to the HDD with the system fs umount of the gameOS partition.
• Running the app again restores the original values, if desired.

You use this app at your own risk! I am not responsible for any damage caused to your PS3 system!​

If my PS3 app was useful to you, I would be very happy about a donation
https://www.paypal.com/paypalme/3141card

picard (aka 3141card)

-- (Original Post Continued Below) --

Need some extra space on your internal PS3 HDD, if so then perhaps @einsteinx2 may have a genius method for unlocking upto 8% of space on your Internal HDD, while the space is more then likely allocated for performance reasons by Sony (use this guide at your own risk), some of this saved space could be considered overkill, as the creator of this method makes a valid point that some of this saved space on larger HDDs is more then some of the early model PS3 complete HDD size, so its likely some of that space allocation is a bit overkill on the part of Sony. einsteinx21 has provided a great tutorial found below explaining in detail about this recent discovery​

-STLcardsWS ​

Unlock up to 8% extra total space on the PS3 internal hard drive
​

About The How + Instructions Source Code Credits

• 
  The What
  
  By default, even on custom firmware, the PS3 reserves 8% of your total internal hard drive space. From my searching online, I don't believe anyone has ever successfully unlocked that wasted space, so I decided to give it a shot. Turns out not only is it possible, but it's relatively easy thanks to some existing tools created by the community. This guide will explain how to reclaim that wasted space by manually modifying the metadata of the UFS2 formatted GameOS partition using Linux, as well as the potential cons (primarily performance, though I haven't experienced any performance issues yet myself).
  
  I believe the pros outweigh the cons though, and have been using this now without issue on my personal PS3 with a 1.5TB drive for a few days now, installing tons of games without a hitch.
  
  The Why
  
  While external drives used with the PS3 are all FAT32 formatted (or NTFS if you have CFW), the GameOS partition on the internal hard drive is formatted using the UFS2 filesystem with a layer of encryption on top.
  
  Like other *nix file systems such as Ext2/3/4, UFS2 can reserve part of the drive's space to only be used by the system or the root user. This is to reduce fragmentation and also prevent the drive from completely filling up, possibly freezing the computer.
  
  By default, UFS2 reserves 8% of any drive's free space, in this case meaning it can't be used by the PS3 for installing games. This is why when you first install that shiny new 1.5TB hard drive you will see that not only is it only actually 1.36 TB because hard drive manufacturers really love counting in base 10, but you also lose another 111.4 GB.
  
  Now generally having this reserved space is not a bad thing, however the problem is that as drives grow larger, the amount of wasted space becomes larger than some of the smaller PS3's entire hard drives! Clearly that much isn't needed to prevent fragmentation.
  
  The tunefs documentation mentions that "the file system's ability to avoid fragmentation will be reduced when the total free space, including the reserve, drops below 15%. As free space approaches zero, throughput can degrade by up to a factor of three over the performance obtained at a 10% threshold." Note that these numbers are already higher than the 8% default that all UFS tools as well as Sony use. Also note that it says as free space drops to 0%, performance may be up to 3 times slower than normal. However, it's unclear whether that only affects newly written--presumably more fragmented files--or all files. It seems like this is a worthy tradeoff, especially since if you do notice any performance issues you can simply delete some games or other data to free up the space. By changing the minimum free space, performance is not changed at all unless you choose to fill the drive completely...but at least you can now make that choice.
  
  UFS2 supports two write optimization modes: time and space. Time optimization is the default--and is used by the PS3--and allows for faster writes at the expense of potentially higher fragmentation (though generally only as the drive reaches capacity). Since we're allowing ourselves to fill almost the entire drive, these instructions also change the optimization mode to space. That means that the filesystem will spend more time during writes to ensure the files are less fragmented, insinuating that it will shuffle blocks around to make contiguous space.
  
  The tunefs documentation says that "the file system can either try to minimize the time spent allocating blocks, or it can attempt to minimize the space fragmentation on the disk. Optimization for space has much higher overhead for file writes. The kernel normally changes the preference automatically as the percent fragmentation changes on the filesystem." However, since the PS3 uses a custom operating system and this documentation is from FreeBSD, I think it's best to manually set the space option anyway. I have not noticed any performance degradation so far since making this change, though I still have plenty of free space on my drive at the moment.
  
  Photos
  Here are the before and after photos. They were taken immediately before removing the drives and immediately after reinserting them.
  

  BEFORE	AFTER
  BEFORE	AFTER

• 
  
  The How
  
  While it's possible to mount a PS3 hard drive in Linux and view its decrypted partitions, unfortunately the tunefs.ufs tool doesn't appear to work. It always complains that it can't find the superblock. However, the file command does work fine to show the UFS2 filesystem info when tested on a dump of the start of the partition.
  
  So instead of using the tunefs utility to change the minimum free space and optimization type, I wrote a script to manually scan and test single byte changes to a dump of the partition's superblock using the file command until I found the correct locations to change.
  
  I tested this on 2 hard drives of vastly different sizes: 160 GB and 1.5 TB. On both drives, I found that the superblock was located in the same location--the standard 128 block aka 65,536 byte offset. I also found that the locations to set the minimum free space percentage and optimization type were in the same place on both drives--byte 65,599 and 65,667 respectively. However, I highly recommend running the find_ps3_ufs2_byte_locations script anyway just to confirm before making any changes to your drive.
  
  Once you know the correct offsets, changing the values is simple. To adjust the minimum free space, simply write the percent in hex to byte 65,599--e.g. for 1% free space write 0x01 or for the default 8% write 0x08. To change the optimization type, write a hex 0x01 to byte 65,667--the default is 0x00 for time optimization.
  
  I think this should be possible to do as a PS3 homebrew app so that it can just be done directly on the device without even removing the drive. Unfortunately I have zero experience with writing PS3 homebrew, but maybe someone else with more experience can use this information to do it. I'd love to see this as an option in the REBUG custom firmware settings.
  
  Instructions
  
  
  1. Dump your eid root key using IrisMan/MultiMan/etc
  2. Setup a computer or virtual machine with Ubuntu 16.04. The rest of these steps are done on that machine. I tested using Parallels on a MacBook Pro, but it should work on just about anything as well as other distros.
  3. Clone my repository: git clone https://github.com/einsteinx2/PS3-Reclaim-HDD-Space.git
  4. Change to the new directory as we'll do all of the work there: cd PS3-Reclaim-HDD-Space
  5. Rename your eid root key file to eid_root_key.bin and place it in the PS3-Reclaim-HDD-Space directory
  6. Generate your hdd keys: ./ps3hdd_keygen.sh
  7. Become root since most of this requires it: sudo -s
  8. Find the device name: fdisk -l (In my case, using an external USB enclosure, it was /dev/sdb)
  9. Make virtual byte swapped encrypted device
     1. If you have a drive 1TB or less (not confirmed the exact limit): ./makedev bswap16.512 /dev/sdb
     2. If you have a drive larger than 1TB (or maybe it's 1TB and larger, I don't have a 1TB drive to test): ./makedev bswap16.1024 /dev/sdb
  10. Create decrypted device: cryptsetup create -c aes-xts-plain64 -d ./hdd_key.bin -s 256 ps3hdd_crypt /dev/nbd0
  11. Map decrypted partitions: ./kpartx -a /dev/mapper/ps3hdd_crypt
  12. View decrypted partitions (ps3hdd_crypt2 is the UFS2 GameOS partition): ls -la /dev/mapper/
  13. View current free space: [ -d /mnt/PS3GameOS ] || mkdir /mnt/PS3GameOS && mount -t ufs -o ufstype=ufs2,ro /dev/mapper/ps3hdd_crypt2 /mnt/PS3GameOS && df -h | grep "Avail\|ps3hdd_crypt2" && umount /mnt/PS3GameOS
  14. Dump the superblock of the GameOS partition: dd if=/dev/mapper/ps3hdd_crypt2 bs=512 count=256 of=GameOS_superblock.img
  15. Confirm the seek values for the next 2 commands: ./find_ps3_ufs2_byte_locations GameOS_superblock.img
  16. Set minimum free space to 1%: printf '\x01' | dd of=/dev/mapper/ps3hdd_crypt2 bs=1 seek=65599 count=1 conv=notrunc
  17. Set optimization type to "space": printf '\x01' | dd of=/dev/mapper/ps3hdd_crypt2 bs=1 seek=65667 count=1 conv=notrunc
  18. View the now larger free space: mount -t ufs -o ufstype=ufs2,ro /dev/mapper/ps3hdd_crypt2 /mnt/PS3GameOS && df -h | grep "Avail\|ps3hdd_crypt2 && umount /mnt/PS3GameOS
  19. Disconnect device: kpartx -d /dev/mapper/ps3hdd_crypt && cryptsetup remove ps3hdd_crypt && ./stop-nbd0
  20. Pop the drive back in your PS3 and enjoy the extra space! Note that I left 1% reserved space rather than going all the way to 0% to ensure that the drive never completely fills up, as I'm unsure what problems that would cause for the PS3's operating system.
• 
  Source code
  
  For ease of use, this repo contains precompiled binaries for Ubuntu 16.04 64bit. If you need or prefer to compile yourself, here are the tools used:
  
  bswap16: https://github.com/sguerrini97/nbdcpp (note that for >1TB drives you must change <unsigned BS=512> to <unsigned BS=1024>)
  
  kpartx: https://git.opensvc.com/multipath-tools/.git/
  
  ps3hdd_keygen.sh: http://www.psx-place.com/threads/hdd-keys-generating-scripts.10610/page-2#post-125197
  
  
• 
  Credits
  
  I would never have figured this out if it weren't for others' hard work.
  
  Huge thanks to Berion at PSX-Place for the hdd key generation script as well as pointing me to the information on mounting a PS3 HDD in Linux. His post here contains the script and the link: http://www.psx-place.com/threads/hdd-keys-generating-scripts.10610/page-2#post-125197
  
  Huge thanks to sguerrini97 at Playstation Hax for implemnenting PS3 hard drive mounting support for modern Linux kernels. Here's the post about it: https://playstationhax.xyz/forums/topic/4671-mounting-ps3-hdd-on-newer-linux-kernels and the GitHub repo: https://github.com/sguerrini97/nbdcpp.
  
  Thanks to dsroche for writing the original nbdcpp implementation that sguerrini97 forked, and thanks to Glevand for the original work on mounting the PS3 hard drive in OtherOS and for the great information here on the PS3 dev wiki: http://www.psdevwiki.com/ps3/Mounting_HDD_on_PC. Also thanks to anyone else that worked on PS3 hard drive mounting or anything else I'm not aware of.
  

​

 

@Sonic3320 It doesn't matter because different speed zones are for early and later parts of i.e 40GiB HDD and 1TiB. 8% for both are in different places with different thresholds.

Developers designing games in a way to dealt with minimum 40MiB/s read speed from HDD (but hard to say the same about IO/s).

 

@Sonic3320 It doesn't matter because different speed zones are for early and later parts of i.e 40GiB HDD and 1TiB. 8% for both are in different places with different thresholds.
.

Unless Sony had a way to determine the speed based on cluster access, which would take A LONG TIME to do this properly (nobody would want to go back to Windows 98 days of formatting hard drives), 8% is quite "safe", many other consoles do create empty partitons at the end of the drive for the same reason.
The reserved space is bigger when you use a bigger drive because it compresses a lot more data physically in a smaller space, so that is why the reserved space needs to be proportional of the size of the drive. One example on Xbox One:

500GB Standard (365GB) (779MB Unallocated)
1TB Standard (781GB) (50.51GB Unallocated)
2TB Standard (1662GB) (101.02GB Unallocated)


This doesnt apply on Series consoles, it does have unallocated space but it tends to be much smaller and it is not because of physical structure limitation, but for that function (which i forgot the name) to use as a backup in case of a sector failure so that the nand chip lasts a bit longer, instead of using the full capacity which would make the storage size shrink when a failed sector appears.

Bigger hard drives tend to have more sectors and more data on the same physical space as the small ones, especially 2,5' (3,5' sometimes have multiple platters, never seen multiple platters on 2,5 drives because it can make the drive too fragile. It is not much of a concern on 3,5 since they are only used on stationary machines).

PS4 does have unallocated space, but unlike Xbox One it is harder to notice unless someone makes a similar tool for it. Connecting a Xbox One hard drive on a computer you can see right from the start the "blank partitions".

 

Many other consoles? Which one?

 

Many other consoles? Which one?

finished to edit my comment, I had some issues pasting the Xbox One partition sizes, so I posted then edited..

 

many other consoles do create empty partitons at the end of the drive for the same reason.

Actually, I asked about it because it is not really truth - I mean, not intentionally for a reason You mentioned:

• PS2 making unallocated space from end, due to APA partitioning scheme which partitions must follow 128MiB rounding. Space at the end is not in use for anything.
• PS3 have unallocated space at the end but is not in use, mainly because is not encrypted and blank. That's high probably because of clusters rounding. In addition, dev_hdd1 (cache partition) is after dev_hdd0 also physically and in OtherOS firmwares with installed Linux, dev_hdd2 is last partition (on all models). That's another proof I think that, 8% of reserves doesn't make sense for reason You mentioned.
• PSV does not using unallocated space on internal flash and for memory cards in the same way as i.e Windows. Small part to cluster round.
• PS4, I didn't check how it looks like, but I do not have hacked fw so I cannot decrypt it and investigate partitions. However, special tool about You said is not needed as she using standard GPT, not encrypted. Drive should be fully mapped by standard tools.
• Xbox Classic doesn't use unallocated space (only in models with 10GiB, cutting last 2GiB, but he never using it for anything).
• Xbox 360 - I don't know.
• Xbox One - I don't know too. BTW: XBO using (at least for external drive) GPT but altered in some way. IsoBuster should tells You if there is anything from the end or not. I heard that they use the same strange GPT for internal and is also not encrypted. Is this true? If yes, then also You could check it easily without any dedicated tools.
• Wii for mass storage use full space (I have in mind SDHC with WBFS). @pinky is that true? ;]

The reserved space is bigger when you use a bigger drive because it compresses a lot more data physically in a smaller space

None of Sony or Nintendo or MS (at least in XC case) consoles compressing data.

but for that function (which i forgot the name) to use as a backup in case of a sector failure so that the nand chip lasts a bit longer, instead of using the full capacity which would make the storage size shrink when a failed sector appears.

NAND or NOR bad blocks, aren't remapped from HDD. On HDD there is only VFLASH area on "NOR models" which have nothing to do with other parts of HDD (it is not remapped neither).

Remapping is made transparently by mass storage firmware (i.e HDD firmware) and bad sectors/blocks are taken from reserved one, but from Service Area (not accessible by ata commands), not unallocated space.

Bigger hard drives tend to have more sectors and more data on the same physical space as the small ones

How so?

 

I heard that they use the same strange GPT for internal and is also not encrypted. Is this true? If yes, then also You could check it easily without any dedicated tools.

The files are encrypted, but the partitions are standard NTFS, you can view the file structure easily, but to make it work on the console, it has to follow a specific pattern of partitions and a specific guid table. It is even possible to extend the User Content partition and make use of the unallocated space, the issue is that, if you have any issue with the console, the User Partition is formatted and it will shrink if the Troubleshooting tools are used and trying to extend beyond 2TB will corrupt the entire data of the hard drive. The console do recognize bigger hard drives but it is only safe to use 2TB on internal max. External is up to 16TB as far as I remember.

None of Sony or Nintendo or MS (at least in XC case) consoles compressing data.

Yes, but I kinda used the wrong words. What I meant is that a bigger hard drive have more sectors using the same physical space as a smaller one. If these manufacturers made some kind of "agreement" to only unallocate data based on raw size (for example, 2GB regardless of hard disk size), it would be bad because it would waste too much sectors on smaller hard drives for no tangible reason and would be ineffective on big hard drives since it would allow a big chunk of data to be stored on the slower portion.

NAND or NOR bad blocks, aren't remapped from HDD. On HDD there is only VFLASH area on "NOR models" which have nothing to do with other parts of HDD (it is not remapped neither).
Remapping is made transparently by mass storage firmware (i.e HDD firmware)

It was just a quick mention, if someone was curious if the Series X or S have these giant unallocated partitions and no because they dont need for this specific reason. What they have and I forgot the name is over provisioning, to keep the NAND from shrinking over use. It is not a thing from any console manufacturer, but from SSD manufacturers. On PC some manufacturers allow the user to change the over provisioning size but they highly advise to not change the default values so the SSD lifespan will decrease (and it is not worth it to gain a few megabytes). When a sector fails on an SSD, it is remapped to one of these free sectors.

One thing that I forgot and you are right is rounding, but rounding doesnt tend to have such big unallocated data.

PSV does not using unallocated space on internal flash and for memory cards in the same way as i.e Windows. Small part to cluster round.

Because it using mainly a flash chip, it doesnt need, the closest thing it will have is over provisioning which tends to be small and not visible in any way.

 

I have a 1tb ssd in my ps3. Should I try this mod do we think?

By that I mean how many of you have done it, and are there any known issues? I understand that it's my risk to take, just looking for opinions.

 

I have a 1tb ssd in my ps3. Should I try this mod do we think?

By that I mean how many of you have done it, and are there any known issues? I understand that it's my risk to take, just looking for opinions.

I'm using this for about 1 year on CFW without any problems. I have a 1TB HDD.

 

I'm using this for about 1 year on CFW without any problems. I have a 1TB HDD.

Do you have it close to full? Thanks for the reply.

 

Do you have it close to full? Thanks for the reply.

Yes, about 910GB.

 

Hey, how soon after running the app did the freezing start? Could it just be an old HDD dying after 10 years?

I haven't actually installed this app yet, that's why I asked if anyone who had installed the app had any BSODs.

 

After installing this on a FERROX 4.87 CECH-2504B (last one that can be run CFW), my ps3 keeps crashing when launching games. I had to remove it

 

Working fine here on my superslim's ssd, got 15gb extra.

 

it was because of PSNPatch plugin.

I 1) deleted the plugin and 2) disabled block PSN at startup and 3) deleted PSNPatch, and then just reinstalled it and reinstalled the plugin and it all worked.

i think it was because it renames the partitions? it seems that renaming then messes with any homebrew apps that were already installed, like PSNPatch, which point at those old partition names. could a dev please confirm if this plausible?

 

Hey guys does anyone know how to get to 1.5 or 2tb on the PS3?
I put in a2tb hdd in but the PS3 could not pick it up..
I could t find any info any have any tips?

 

If PS3 don't want format it, then You are out of luck and must use another HDD. There is theory that supported so large drives are only those with physical sector equal to 512 bytes, instead to 4KiB, but no one ever check this.

If we be able to prepare HDD on PC, then probably it would works. But we cannot, at least currently.

 

This, but it doesnt hurt to try to rebuild mbr using gparted on a linux partition, highly recommend puppy linux because if you dont want to install on the machine, it is perfect to use from a USB drive. Some well known distros like Ubuntu or Linux Mint work too, but they are not as optimized to run from a flash USB drive, these distros run horrible regardless of your machine specs, even the most simple of tasks take long to do. One time a 500gb hard drive was refusing to install the firmware on the PS3 because the partition table was screwed up (even right now I had to do this because my external hard drive was not being recognized) and Windows is horrible to deal with this stuff, using the included tool or many formatting apps on it dont work (because many use the same features on Windows, just with a different interface). Granted, it is not recommended to use 2tb hard drives at all, but 1,5tb should work.

 

@Sonic3320 PS3 doesn't use MBR on internal HDD and doesn't care what is on HDD because all will be overwrite anyway by custom and encrypted shit. For NOR consoles there are even two partition tables (one the same format as on NAND models, and another inside first partition):

 

@Sonic3320 PS3 doesn't use MBR on internal HDD and doesn't care what is on HDD because all will be overwrite anyway by custom and encrypted shit. For NOR consoles there are even two partition tables (one the same format as on NAND models, and another inside first partition):

I didnt know specifically about this, but I had multiple hard drives working inside Windows but not visible in other machines unless I used gparted and used the "Rebuild partition table" option on it, both from internal and external drives. The Windows (since XP) utilities never worked for me when I had this issues of drives failing to recognize on other devices (on PS3 was worse because it failed to install the firmware all the time and the drive was in perfect condition, I test using hd tune pro before using any hard drive, so that I can assure that I can transfer my files without the risk of losing them).

 

That's interesting. From logical point of view: if they replacing data, they shouldn't check anything else than theirs partition table existence.

If I ever have such drive which is not recognized, I'll try this hypothesis.

I remember that I once have some MBR leftovers in not encrypted form (after format on PS3 side). In those times I seeking a way to decrypt drive and scene didn't know yet that NOR and NAND models using different AES cipher and key generation. Later, I again needs to format and everything was gone, nice zeroes from start. So who knows what they really doing with met MBR and why. :|