Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Output of df is:

[root@backup log]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGro  1889811408 1861658948         0 100% /
/dev/sda1               101086     16235     79632  17% /boot
tmpfs                  1815760         0   1815760   0% /dev/shm

So available block should be 28.152.460, but it's 0. I've been deleting a crap load of files, and used blocks is going down, but available remains at 0.

Output of df -i is:

[root@backup log]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/VolGro 487751680 238360803 249390877   49% /
/dev/sda1              26104      37   26067    1% /boot
tmpfs                 219784       1  219783    1% /dev/shm

So it's not a lack of inodes.

Output of lsof +L1 is:

[root@backup log]# /usr/sbin/lsof +L1
COMMAND     PID  USER   FD   TYPE DEVICE SIZE NLINK      NODE NAME
mysqld     2444 mysql    4u   REG  253,0    0     0 268795908 /tmp/ibSlaKC7 (deleted)
mysqld     2444 mysql    5u   REG  253,0    0     0 268795909 /tmp/ibhFuyGr (deleted)
mysqld     2444 mysql    6u   REG  253,0    0     0 268795910 /tmp/ibbNinKL (deleted)
mysqld     2444 mysql    7u   REG  253,0    0     0 268795911 /tmp/ibz1ia55 (deleted)
mysqld     2444 mysql   11u   REG  253,0    0     0 268795912 /tmp/ibM3IHvr (deleted)
crond      2549  root    3u   REG  253,0    5     0 248579098 /var/run/crond.pid (deleted)
yum-updat  2620  root   14w   REG  253,0    0     0 248611115 /var/run/yum.pid (deleted)
ssh       16256  root    0u   CHR  136,0          0         2 /dev/pts/0 (deleted)
ssh       16256  root    1u   CHR  136,0          0         2 /dev/pts/0 (deleted)
ssh       16256  root    2u   CHR  136,0          0         2 /dev/pts/0 (deleted)

I can't run 'du' because 99% of the disk usage is under /var/backups which contains probably ~100 million files (some idiot decided to rsync code from live server with subversion directories, so it's a lot of tiny files), so running 'du' would take days or weeks.

Does anyone have any suggestions on how to proceed?

share|improve this question
    
Which filesystem is this, and is all that rsync activity under /var/backups being done as root? –  DerfK Sep 8 '11 at 15:33
    
It's an ext3 filesystem. All the backup processes are run under root –  jw. Sep 9 '11 at 8:06
add comment

4 Answers

If this is an ext filesystem, the default root reserved space would be 5% of 1889811408 blocks, or 94490570 blocks. In other words, you have about 66GB more to delete before df will report free space available.

Use tune2fs -m 1 /dev/mapper/VolGro to decrease the reserved amount to 1%, or -r NNNN to set it to a specific number of blocks. There needs to be enough reserved space that logging can continue even after users have "filled" the disk (though if you're filling the disk as root, this won't save you from problems when the drive is absolutely full)

Other filesystems probably have reserved blocks as well, but the commands to adjust these will differ.

share|improve this answer
    
Damn, I was just writing up that answer. +1 –  bahamat Sep 8 '11 at 15:48
add comment

Run an fsck to see if the filesystem is damaged.

share|improve this answer
    
+1 - On linux this may also mean needing to unmount the affected filesystem (or touch the .forcefsck file in the FS root (IIRC) and then reboot). –  voretaq7 Sep 8 '11 at 15:39
add comment

If you have deleted things like apache log files- it requires a apache restart before the space is "released"

share|improve this answer
    
If this were the case his blocks would not be showing up as free (df uses the kernel's idea of allocated space, which includes open file handles. du would incorrectly report that space as free since it just walks the directory tree (and only counts space from the inodes it finds)) –  voretaq7 Sep 8 '11 at 15:36
    
lsof +L1 shows that there's only 5 bytes in deleted files –  jw. Sep 9 '11 at 9:35
add comment

So available block should be 28.152.460, but it's 0. I've been deleting a crap load of files, and used blocks is going down, but available remains at 0.

I suspect that these files were still in use from your service when you delete it. So, simply restart the service, df command will update the available space.

share|improve this answer
1  
lsof +L1 shows that there's only 5 bytes in deleted files –  jw. Sep 9 '11 at 9:35
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.