Reclaiming Space That 'df' Says Should Be Available

Question

I have a linux server hosted on one of Rackspace's CloudServers and it is showing that I have both no room available, yet also shows I have 12GB not used on the root partition. This server's only job is to rsync over files nightly from another server via rsync and keep an emergency backup of those files.

I found a directory that had some backup files that weren't needed so deleted those and that is what gave me the 12GB unused. Deleting files shrinks the "Used" space, but doesn't give anything to the "Avail" space.

lsof doesn't show that it is holding on to anything and df -i says I have plenty of free inodes.

I've also rebooted this server multiple times and the problem persists. I've even rebooted in to rescue mode and ran an fsck on the device and didn't see any glaring errors:

root@RESCUE-nightly-snapshot:~# fsck /dev/xvdb1
fsck from util-linux 2.20.1
e2fsck 1.42 (29-Nov-2011)
/dev/xvdb1: recovering journal
Setting free inodes count to 20630768 (was 20630766)
Setting free blocks count to 2845985 (was 2755815)
/dev/xvdb1: clean, 340752/20971520 files, 81040087/83886072 blocks
root@RESCUE-nightly-snapshot:~#

So I'm kind of stuck because normal users can't create any new files and the syslog user can't append to any log files.

Lastly, the filesystem is ext3 and something in my brain is saying maybe there is something weird with the journal. I don't know if this space could be taken up by the journal or how to even tell.

I'd appreciate any tips on how to regain this space that should've been freed.

root@nightly-snapshot:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      315G  303G     0 100% /
udev            3.9G  4.0K  3.9G   1% /dev
tmpfs           1.6G  244K  1.6G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.9G     0  3.9G   0% /run/shm
overflow        1.0M     0  1.0M   0% /tmp
/dev/md0        5.0T  4.2T  593G  88% /mnt/raid
root@nightly-snapshot:~# df -i
Filesystem        Inodes    IUsed     IFree IUse% Mounted on
/dev/xvda1      20971520   340750  20630770    2% /
udev             1016188      422   1015766    1% /dev
tmpfs            1018170      319   1017851    1% /run
none             1018170        3   1018167    1% /run/lock
none             1018170        1   1018169    1% /run/shm
overflow         1018170        1   1018169    1% /tmp
/dev/md0       167772160 12607068 155165092    8% /mnt/raid
root@nightly-snapshot:~#
root@nightly-snapshot:~# lsof | grep -i deleted
root@nightly-snapshot:~#

Answer 1

There is a chance that this space is reserved for root user. Please check out how much space is reserved for root,with:

tune2fs -l /dev/xvda1

This will give you the block count reserved for root user, multiply it by the block size and you will get the size of reserved space (in bytes).

You can modify the reserved space with: tune2fs -m 1 /dev/xvda1 The number after -m is the percent of disk space which should be reserved. In this example it is 1%.