Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

This question already has an answer here:

I have a fileserver where df reports 94% of / full. But according to du, much less is used:

# df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             270G  240G   17G  94% /
# du -hxs /
124G    /

I read that open but deleted files could be responsible for it but a reboot did not fix this.

This is Linux, ext3.


share|improve this question

marked as duplicate by Chris S Nov 19 '14 at 15:07

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

Combine @TCampbell and @Kyle Brandt's answers - reboot and if that doesn't fix it, boot from a rescue CD and run fsck on the unmounted partition. – Paul Tomblin Aug 21 '09 at 11:49
I already rebooted before. Extensive fsck running right now. – Andreas Kuntzagk Aug 21 '09 at 12:08
Ok, after an hour of counting, multiplying allocated blocks with block sizes, counting inodes, the pure truth came out - in my case the huge difference was caused not by a mount, but a hidden .trash folder! It was right under my nose. – user176156 Jun 1 '13 at 14:21
up vote 27 down vote accepted

Ok, found it.

I had a old backup on /mnt/Backup in the same filesystem and then an external drive mounted in that place. So du didn't see the files. So cleaning up this gave me back my disk space.

It probably happened this way: the external drive once was unmounted while the daily backup script run.

share|improve this answer
Interesting, didn't think of that. Mounting a fs on a non-empty directory can do funny things... – sleske Aug 21 '09 at 14:20
Ya, that was one of the ones in the link I gave you. That is called an 'overlay mount'. – Kyle Brandt Aug 21 '09 at 15:13
Youre right Kyle. I totally missed that in this long page. – Andreas Kuntzagk Aug 21 '09 at 15:35
Andreas, it also doesn't make it that clear, I didn't think of it either. – Kyle Brandt Aug 21 '09 at 17:08
chmod mountpoints to 000, so you get errors from scripts instead of them silently filling your root partition – hop Aug 23 '09 at 0:07

I don't think you will find a more thorough explanation that then this link for all the reasons it could be off. Some highlights that might help:

  • What is your inode usage, if it is almost at 100% that can mess things up:

    df -i

  • What is your block size? Lots of small files and a large block size could skew it quite a bit.

    sudo tune2fs -l /dev/sda1 | grep 'Block size'

  • Deleted files, you said you investigated this, but to get the total space you could use the following pipeline (I like find instead of lsof just because lsof is a pain to parse):

    sudo find /proc/*/fd -printf "%l\t%s\n" | grep deleted | cut -f2 | (tr '\n' +; echo 0) | bc

However, that is almost 2x off. Run fsck on the partition while it is unmounted to be safe.

share|improve this answer
df -i does not report anything unusual, will go the fsck way now. – Andreas Kuntzagk Aug 21 '09 at 11:58
quite many ~blocksize/2 size files can make this problem by filling up entire blocks, creating enormously big unavailable space (in the unavailable part of remaining space in the blocks) so do you store lots of small files there? – asdmin Aug 21 '09 at 12:36
How would I find out the number of files with such a size? – Andreas Kuntzagk Aug 21 '09 at 13:51
First you need to find you block size, so if it 4096, You want a file less then 4KB, so find / -size -4k | wc -l – Kyle Brandt Aug 21 '09 at 14:26
Used to fill up volumes with several 100 thousand 3k files... changing filesystem from xfs (on sgi) to reiserfs helped make diskspace more efficient. Not an option for many, but worked for us. – ericslaw Aug 21 '09 at 21:55

It looks like a case of files being removed while processes still have them open. This disconnect happens because the du command totals up space of files that exist in the file system, while df shows blocks available in the file system. The blocks of an open and deleted file are not freed until that file is closed.

You can find what processes have open but deleted files by examining /proc

find /proc/*/fd -ls | grep deleted
share|improve this answer
I already investigated this problem (see original post) btw. you can also get that by lsof| grep deleted – Andreas Kuntzagk Aug 21 '09 at 11:56
Good point about lsof, thanks. – TCampbell Aug 21 '09 at 12:13

The most likely reason in your case is that you have lots of files that are very small (smaller than your block size on the drive). In that case df will report the sum of all used blocks, whereas du will report the actual sum of file sizes.

share|improve this answer
df can show you the status of inodes for the disk or partition too which may shed some more light as to whether or not this is your situation - df -i / – DisabledLeopard Aug 21 '09 at 14:41
@wolfgangsz Not exactly, unless you use du --apparent. – sourcejedi Jun 1 '13 at 15:19

I agree that

lsof +L 1 /home | grep -i deleted

is a good place to start, in my case I notice that I had lots of perl scripts that was running, and keeping a lot of files alive, even though they was supposed to be deleted.

I killed the perl functions, and this made du and df almost identical, case closed.

share|improve this answer

Is it possible that perhaps du doesn't add in the size of the directories as well? Still, seems like a HUGE difference though, that can't be responsible for all of it.

share|improve this answer

By default, when you format a filesystem with EXT3, 5% of the drive is reserved for root. df accounts for this reserve when it reports what is available, while du shows what is actually in use.

You can view the reserved blocks by running:

tune2fs -l /dev/sda|grep -i reserve

and you will get something like:

Reserved block count:     412825
Reserved GDT blocks:      1022
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)

If you would like to adjust that to a lower percent, you can do so with something like

tune2fs -m 1 /dev/sda

You can reduce it to 0, however since this is your root filesystem I would be wary of doing that. If the filesystem actually filled it may make maintenance tasks required to clean it up difficult.

share|improve this answer
That is true, but seems beside the point. The "in use" size reported by du and df differs, and that is independent of reserved blocks. – sleske Aug 21 '09 at 12:52
The size of disappeared space is not comparable to 5% – drAlberT Aug 21 '09 at 12:54
Yeah, it still doesn't add up, but I figured that was adding to the discrepancies. – Alex Aug 21 '09 at 13:46

Not the answer you're looking for? Browse other questions tagged or ask your own question.