Ubuntu server: hard drive always full

Question

I have a 50GB vps

when I run df I get

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             49982172  46580100    863064  99% /
none                    507212       172    507040   1% /dev
none                    511676         4    511672   1% /dev/shm
none                    511676        76    511600   1% /var/run
none                    511676         0    511676   0% /var/lock
none                    511676         0    511676   0% /lib/init/rw

When I run du -h from / I get

...
20K     ./tmp/vmware-root
4.0K    ./tmp/.webmin
4.0K    ./tmp/.X11-unix
4.0K    ./tmp/.ICE-unix
4.0K    ./tmp/hsperfdata_root
48K     ./tmp
7.7M    ./bin
3.8G    .

I've also tried to find out about files bigger then 100MB to try find what is taking up my space with find / -size +100M -ls

4026531985    0 -r--------   1 root     root     140737486266368 Mar 23 08:55 /proc/kcore
find: `/proc/3969/task/3969/fd/5': No such file or directory
find: `/proc/3969/task/3969/fdinfo/5': No such file or directory
find: `/proc/3969/fd/5': No such file or directory
find: `/proc/3969/fdinfo/5': No such file or directory

I have no idea what else I can do to find my problem.

Edit:

More info

df --inodes /

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            3178496  178211 3000285    6% /

find / -xdev -print | wc -l

178773

lsof +L1

COMMAND     PID  USER   FD   TYPE DEVICE SIZE/OFF NLINK    NODE NAME
miniserv.  1496  root  txt    REG    8,1    10416     0 2364381 /usr/bin/perl (deleted)
mysqld    12869 mysql    4u   REG    8,1        0     0  537495 /tmp/ibkCwnHJ (deleted)
mysqld    12869 mysql    5u   REG    8,1        0     0  537497 /tmp/ibFyPrrv (deleted)
mysqld    12869 mysql    6u   REG    8,1        0     0  537499 /tmp/ibdUnwbh (deleted)
mysqld    12869 mysql    7u   REG    8,1        0     0  537500 /tmp/iblHYOV2 (deleted)
mysqld    12869 mysql   11u   REG    8,1        0     0  537501 /tmp/ibUW9YGO (deleted)

Distributor ID: Ubuntu Description: Ubuntu 10.04.4 LTS Release: 10.04 Codename: lucid

Answer 1

There is usually some discrepancy between du and df but it's not normally as large as you're seeing.

• du reports the disk space that is used by reading the information from the directory tree, it is accurate but slow.
• df reports the disk space that is used by reading the filesystem meta data, it's quick but less accurate as it works with blocks.

The usual cause for what you are seeing is a file that has been deleted but hasn't yet been closed by the process that is writing to it.

As the file has been deleted, du won't see a directory entry for it and therefore cannot include it in it's report.

As the file is still open, the blocks it uses aren't free so df will report that they are in use.

You should be able to track down the file that is causing the problem with

lsof +L1

Check the SIZE/OFF column.

A specification of the form +L1 will select open files that have been unlinked. A specification of the form +L1 will select unlinked open files on the specified file system.