Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have weird situation because Linux df command says that there is no free disk space

[root@backup cache]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              72G   70G     0 100% /
/dev/sda1             190M   11M  170M   7% /boot
tmpfs                 248M     0  248M   0% /dev/shm

but du -sh /* says

[root@backup cache]# du -sh /*
4.0K    /bacula-restores
7.4M    /bin
5.4M    /boot
3.6T    /data
116K    /dev
55M     /etc
204K    /home
76M     /lib
16K     /lost+found
12K     /media
0       /misc
16K     /mnt
8.0K    /mount
0       /net
8.0K    /opt
0       /proc
2.3G    /root
32M     /sbin
8.0K    /selinux
168K    /share
8.0K    /srv
0       /sys
361M    /test
20K     /tmp
3.2G    /usr
1.5G    /var

Could you tell me where is a problem? Where is my space? I can't figure it out :(

share|improve this question
    
/data 3.6T is another disk <- don't look there :) –  skomak Apr 16 '10 at 8:13
    
File system is ext2/3, right? –  pehrs Apr 16 '10 at 8:32
    
Interesting that df shows a 72G disk with 70G in use as being 100% used –  Cez Apr 16 '10 at 8:37
5  
@Cez: Reserved space for root user. Free space is free space for the unpriviledged user, not total free space. Linux doesn't display negative free space, unlike some other UNIX systems, when you start eating up root's reserved space. –  pehrs Apr 16 '10 at 8:42
    
@pehrs +1 Thanks for clarifying –  Cez Apr 16 '10 at 9:01

7 Answers 7

Try:

$ lsof +L1 

This will find files that have a link count less than 1 (files removed but still beeing written to).

For those times when du and df don't match up.

share|improve this answer

Short version: Use lsof to find the file that is unlinked but still open. Restart the application holding the file (or the whole system, if you are lazy) and you will get your free space back.

Long version: Most likely you have a file that is open by an application and being written to, but has been unlinked (deleted) from the filesystem.

Unlike NTFS you have a sort of reference counting in ext. This means that when you delete a file you just remove the reference to it. Opening the file in a program adds a reference. So you will have to figure out which program holds the file. This is why you often see references to link/unlink when it comes to file operations in UNIX. Typically finding the file is done with the tool lsof.

The good side of this behaviour is that you never get the "File is open so can't delete it" error that you often get in Windows. You can also replace system files, such as shared libraries and your software will use the old libraries until it restarts (and loads the new ones from disk), instead of having to reboot for software upgrades (Windows again). The bad side you see right now. The size of the visible files in the filesystem will typically not match the amount of data on disk.

share|improve this answer

Or the poor-man lsof in this case would be (for Linux)

ls -l /proc/*/fd/ | grep deleted
share|improve this answer

I always do a "chattr +i /mount-point" on the empty directory while it is unmounted.

If it happens that something tries to write there while it is not mounted, a "Permission denied" error message is returned. And you notice the error right away.

If there is a properly mounted resource on "/mount-point", then the "chattr +i" has no effect and everything functions as expected.

share|improve this answer
    
very nice, good idea! –  rkthkr Apr 17 '10 at 18:14

you may also want to check if your inodes has been exhausted. df -i.

share|improve this answer
    
Not exactly what the problem of the original post, but it's still a valuable hint. –  Denilson Sá May 6 '10 at 20:32

I know what went wrong. I have NFS mounted as /data, NFS has got network error connection and /data was on disk mounted as / and not as NFS share.

I thought that it may be related to mechanism like this

example dir listing: / /a [dir] file1 file2 /b [dir]

another device: / fileX fileY

If we mount 'another device' to /a we got access to this device via /a directory but original /a still exist there but is overhandled. We can get access to the original directory after unmount last mounted device. It is interesting property but in fact can bring some troubles (Not mine at this moment).

share|improve this answer
    
I was about to add an answer suggesting that as a possibility. Have seen this happen many times with some cron jobs not checking to see if an external drive was mounted before moving or syncing large amounts of data. –  3dinfluence Apr 16 '10 at 13:59
  • You can try to reduce the root's reserved space thought tune2fs
  • Monitor your network from another machine to check if some hidden evil process is using your machine as a server for warez traffic
  • if you have a monolithic kernel try rebooting with it, this should disable LKM rootkit if present
  • run rkhunter, chkrootkit
  • reboot with a live distro and analize the disk from RAM
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.