Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Possible Duplicate:
Linux different size for df and du for root partition

On an Ubuntu 12.04 server, normal users can no longer create or add to files in /home, encountering a "No space left on device" error.

The /home directory has a capacity of 1.7 terabytes and as far as I can tell is nowhere near full in terms of actual data stored or inodes used.

df -h shows:

Filesystem      Size  Used Avail Use% Mounted on
/dev/md2        1.0T   18G  955G   2% /
udev            7.7G  4.0K  7.7G   1% /dev
tmpfs           3.1G  320K  3.1G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            7.7G     0  7.7G   0% /run/shm
cgroup          7.7G     0  7.7G   0% /sys/fs/cgroup
/dev/md3        1.7T  1.7T     0 100% /home
/dev/md1        496M   45M  426M  10% /boot

/home indeed looks rather full.

du -hs /home suggests otherwise:

1.4G    /home

There appears no inode issue - df -i:

Filesystem        Inodes  IUsed     IFree IUse% Mounted on
/dev/md2        67108864  75334  67033530    1% /
udev             2013497    527   2012970    1% /dev
tmpfs            2015816    440   2015376    1% /run
none             2015816      2   2015814    1% /run/lock
none             2015816      1   2015815    1% /run/shm
cgroup           2015816      9   2015807    1% /sys/fs/cgroup
/dev/md3       113909760 105981 113803779    1% /home
/dev/md1          131072    239    130833    1% /boot

I recently deleted a many gigabytes of application cache and log data from /home, however this was in the tens of gigabytes at best and nowhere near the capcity of /home.

Update 1:

du -hs --apparent-size /home
1.2G    /home
du -hs /home
1.4G    /home

What might be going on here?

share|improve this question

marked as duplicate by Jeff Ferland, mdpc, rnxrx, Scott Pack, HopelessN00b Sep 27 '12 at 22:43

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1  
Can you try du with --apparent-size, please? The man page says: "print apparent sizes, rather than disk usage; although the apparent size is usually smaller, it may be larger due to holes in ('sparse') files, internal fragmentation, indirect blocks, and the like". Maybe that helps? –  Axel Knauf Sep 20 '12 at 15:10
    
@AxelKnauf: thanks for the suggestion, I've updated the question with these details –  Jon Cram Sep 20 '12 at 15:15
    
Look at the answers to this question: serverfault.com/q/132998/37499 –  mjcopple Sep 20 '12 at 15:18
    
Are you running du as root? –  Shane Madden Sep 20 '12 at 15:28
    
Have you tried fsck-ing the volume? –  James Yale Sep 20 '12 at 15:46

1 Answer 1

Check the files are still opened in write mode:

lsof | awk '/deleted/ && $4 ~ /[0-9]+w/ { print $0 }'

you will see something like this:

ossec-mai  1111    ossecm    4w      REG                3,3     ... (deleted)

then find the process that still uses the file:

cd /proc/1111/fd
ls -l 4

You can free up the space without restarting the service by getting the size down to zero:

> /proc/1111/fd/4

Give it a try.

share|improve this answer
    
Good suggestion, I was about to ask the same. Also mentioned in this SO question. –  Axel Knauf Sep 20 '12 at 15:19
1  
This may well have been it. There were some very large log files related to this. I've just rebooted the machine and df -h now shows more what I'd expect. Since things are now fine I can't be sure of the cause. I expect this issue may arise again in a couple of weeks. I'll mark this as the accepted answer then if that's the cause. –  Jon Cram Sep 20 '12 at 15:22
    
In addition to already suggested causes, it could be also following: * a different disk is mounted "over" the existing folder which is full of data * du will calculate the size spent of mounted disk and df will show really spent * solution: (when possible) unmount all non-root disks and check the size with du -md 1 again. Fix situation by moving hidden folder to some other place or mount on different place. –  Robert Lujo Oct 23 at 18:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.