Recover from a linux boot issue - no free disk space

The other day, I ended up in an unenviable position. I was running low on disk space on my Linux laptop, and was cleaning up to free some space. I ran BleachBit to do a system-wide clean. I was doing quite a bit in parallel, processing some raw images and playing a video in the background while writing code, too much! Long story short, I suspected BleachBit did not finish the cleaning task, but I did not notice anything unusual. So, I shutdown my laptop.

Next time, I turned on the laptop, it would not boot properly. The splash screen for Ubuntu came up, and after that, just a black screen.

My guess was that disk space was the issue. I just needed to troubleshoot to see how I could free up enough space. Luckily, I had a bootable pen drive with Manjaro on it, so I could boot into another OS quickly and look at the disk to see what state it was in.

Spoiler alert: I fixed the issue.

Maybe, this might save your bacon some day, but I hope you never get into this situation!

Troubleshooting

Once Manjaro was booted, I opened the file explorer to see if the hard disk was mounted.

It was.

Next, I opened a terminal to explore. The df command can report on system disk usage. Using it with the -h flag gives you a human-readable summary of your disk usage. Here is an abbreviated view of my system (recreated):

df -hFilesystem      Size  Used Avail Use% Mounted onudev            7.5G     0  7.5G   0% /devtmpfs           1.6G  2.1M  1.6G   1% /run/dev/sda5       480G  456G    0M  95% /...

And there you have it, my harddrive - /dev/sda5 - has no available space!

How the hell do I fix that?

I deleted some files and ran the command again. There still was no available space.

I looked up how to empty the trash bin of a disk on the command-line. Since I am doing this external to the installed operating system, it is not straightforward. I couldn't get it to work.

The issue is likely to be that the space allocated to the files that have been deleted has not been released. How do I trigger that release process?

The most common root cause suggested is that the space is being held by open files, files with a process still attached. In my case, a BleachBit process was killed improperly. Maybe that is the issue.

I looked at this troubleshooting guide - Space is not released after deleting files in Linux? - to see if I could replicate what they did. The first suggestion they made is use the lsof command. This requires that you are logged into the operating system, so that was out.

The second suggestion requires flushing the proc filesystem. This did not work for me.

I was stuck.

I called it a day. After a while away from the screen, I remembered that the Use% field showed 95% of the disk was being used. What about that 5%?

The backdoor

By default, ext3 and ext4 filesystems on Linux reserve 5% of their capacity for special processes that can continue to run when diskspace is running low. We can change this allocation and reclaim that space!

We can use the tune2fs command to adjust filesystem parameters on ext3 and ext4 filesystems. We can use this command to free the reserved space. The -m flag allows you to set the reservered blocks percentage. To set it to zero, run the following command:

tune2fs -m 0 /dev/sda5

And bingo, we have a few gigabytes of disk space available!

I restart the laptop and Ubuntu boots. Great!

However, we still have gigabytes of dead space.

I tried the lsof command to see about the open files from deleted processes. We can run the following command to get that list:

lsof | grep -i deleted

I did not see anything that looked like it belonged to BleachBit. Not a clear way to free that space.

Maybe, I can just run BleachBit?

I did a data backup and gave it a go.

And it finished. I checked the disk usage, andddddd I got the reported 30.55G of free space back! Halleujah!

We are done yet though! Don't forget to restore that reserved system space. I decided to reduce the reserved space to 3% as below:

tune2fs -m 3 /dev/sda5

Now, we are fully recovered. Business as usual!