I have a linux server hosted on one of Rackspace's CloudServers and it is showing that I have both no room available, yet also shows I have 12GB not used on the root partition. This server's only job is to rsync over files nightly from another server via rsync and keep an emergency backup of those files.
I found a directory that had some backup files that weren't needed so deleted those and that is what gave me the 12GB unused. Deleting files shrinks the "Used" space, but doesn't give anything to the "Avail" space.
lsof doesn't show that it is holding on to anything and
df -i says I have plenty of free inodes.
I've also rebooted this server multiple times and the problem persists. I've even rebooted in to rescue mode and ran an
fsck on the device and didn't see any glaring errors:
root@RESCUE-nightly-snapshot:~# fsck /dev/xvdb1 fsck from util-linux 2.20.1 e2fsck 1.42 (29-Nov-2011) /dev/xvdb1: recovering journal Setting free inodes count to 20630768 (was 20630766) Setting free blocks count to 2845985 (was 2755815) /dev/xvdb1: clean, 340752/20971520 files, 81040087/83886072 blocks root@RESCUE-nightly-snapshot:~#
So I'm kind of stuck because normal users can't create any new files and the syslog user can't append to any log files.
Lastly, the filesystem is ext3 and something in my brain is saying maybe there is something weird with the journal. I don't know if this space could be taken up by the journal or how to even tell.
I'd appreciate any tips on how to regain this space that should've been freed.
root@nightly-snapshot:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 315G 303G 0 100% / udev 3.9G 4.0K 3.9G 1% /dev tmpfs 1.6G 244K 1.6G 1% /run none 5.0M 0 5.0M 0% /run/lock none 3.9G 0 3.9G 0% /run/shm overflow 1.0M 0 1.0M 0% /tmp /dev/md0 5.0T 4.2T 593G 88% /mnt/raid root@nightly-snapshot:~# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvda1 20971520 340750 20630770 2% / udev 1016188 422 1015766 1% /dev tmpfs 1018170 319 1017851 1% /run none 1018170 3 1018167 1% /run/lock none 1018170 1 1018169 1% /run/shm overflow 1018170 1 1018169 1% /tmp /dev/md0 167772160 12607068 155165092 8% /mnt/raid root@nightly-snapshot:~# root@nightly-snapshot:~# lsof | grep -i deleted root@nightly-snapshot:~#