Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

On a virtualized server running Ubuntu 10.04, df reports the following:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             7.4G  7.0G     0 100% /
none                  498M  160K  498M   1% /dev
none                  500M     0  500M   0% /dev/shm
none                  500M   92K  500M   1% /var/run
none                  500M     0  500M   0% /var/lock
none                  500M     0  500M   0% /lib/init/rw
/dev/sda3             917G  305G  566G  36% /home

This is puzzling me for two reasons: 1.) df says that /dev/sda1, mounted at /, has a 7.4 gigabyte capacity, of which only 7.0 gigabytes are in use, yet it reports / being 100 percent full; and 2.) I can create files on / so it clearly does have space left.

Possibly relevant is that the directory /www is a symbolic link to /home/www, which is on a different partition (/dev/sda3, mounted at /home).

Can anyone offer suggestions on what might be going on here? The server appears to be working without issue, but I want to make sure there's not a problem with the partition table, file systems or something else which might result in implosion (or explosion) later.

share|improve this question
    
Thanks to all for the helpful answers. I can't create files as a normal user so it does appear that it's the 5 percent buffer that's preventing catastrophe. Now I just need to figure out why the disk is full (I'm a bit worried something malicious could be going on because none of the log files is taking up that much space and there's not much software installed, just a simple LAMP server)... –  Chris Sep 25 '11 at 9:16
2  
First place I'd look is /tmp. Another possibility is that you have a deleted file that a running program is holding on to. I think you can run 'lsof | grep deleted' as root to find those. –  Scott Sep 29 '11 at 17:53

7 Answers 7

5% (by default) of the filesystem is reserved for cases where the filesystem fills up to prevent serious problems. Your filesystem is full. Nothing catastrophic is happening because of the 5% buffer -- root is permitted to use that safety buffer and, in your setup, non-root users have no reason to write into that filesystem.

If you have daemons that run as a non-root user but that need to manage files in that filesystem, things will break. One common such daemon is named. Another is ntpd.

share|improve this answer
1  
To the question of WHY your disk is full, 7G really isn't that much space. You also appear to have everything dumped under one partition/filesystem (/). This is generally considered to be a Bad Thing (because if something goes haywire / fills up, and the world ends) but Linux distributions still persist in doing it because it's "simpler". I'd start by looking in /var (esp. /var/log) for huge logfiles. du -hs / (as root) will help you find the biggest directories and possibly point you at what needs cleaning up. –  voretaq7 Sep 27 '11 at 20:14

You may be out of inodes. Check inode usage with this command:

df -i
share|improve this answer

Most Linux filesystems reserve 5% space for use only the root user.

You can see this with e.g

dumpe2fs /dev/sda1 | grep -i reserved

You can change the reserved amount using :

tune2fs -m 0 /dev/sda1

In most cases the server will appear to continue working fine - assuming all processes are being run as 'root'.

share|improve this answer

It's possible that a process has opened a large file which has since been deleted. You'll have to kill that process to free up the space. You may be able to identify the process by using lsof. On Linux deleted yet open files are known to lsof and marked as (deleted) in lsof's output.

You can check this with sudo lsof +L1

share|improve this answer
    
It's solved the the mystery for me. I removed a large log file from uwsgi without restart the service. When queried df -ah, I got disk full, but du -sh / tells that I should have free space. After retart uwsgi I got a lot of free space! –  Fabio Montefuscolo Oct 15 at 16:03

df -h is rounding the values. Even the percentages are rounded. Omit the -h and you see finer grained differences.

Oh. And ext3 and derivates reserve a percentage (default 5%) for the file-system for exactly this problematic constellation. If your root filesystem would be really full (0 byte remaining) you can't boot the system. So the reserved portion prevents this.

share|improve this answer
    
Could also be that he's ran out of free inodes. Run 'df -i' to get inodes usage. –  Andrew Case Sep 24 '11 at 21:52
    
He didn't provide information that the disk is full. He only thinks that the disk is full. 100% used space without error is only "virtually full". –  mailq Sep 24 '11 at 22:32

I had this problem and was baffled by the fact deleting various large files did not improve the situation (didn't know about the 5% buffer) anyway following some clues here

From root walked down the largest directories revealed by repetitively doing:-

du -sh */ 

until I came a directory for webserver log files which had some absolutely massive logs

which I truncated with

:>lighttpd.error.log

suddenly df -h was down to 48% used!

share|improve this answer
3  
That should really end with "... then I set up log rotation." –  hayalci Dec 3 '12 at 17:02
    
hayalci : found that the logrotation was pointing to the wrong directory. –  zzapper Dec 18 '12 at 10:08

In addition to already suggested causes, in some cases it could be also following:

  • a different disk is mounted "over" the existing folder which is full of data
  • du will calculate the size spent of mounted disk and df will show really spent
  • solution: (when possible) unmount all non-root disks and check the size with du -md 1 again. Fix situation by moving hidden folder to some other place or mount on different place.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.