Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a CentOS 5.5 box with a seperate partition for /home. df -h shows it at 100% usage. df -i shows tons of free inodes. du -sh /home /home* shows that there are about 100MB worth of files in /home. lsof -n|grep /home shows no open files in /home. lsof -n|grep deleted shows no files touching /home.

The partition is shared out via NFS and is mounted on two other CentOS 5.5 clients. lsof -n|grep /home and lsof -n|grep deleted show no files for for /home or /.

When I unmount and remount the partition, disk usage drops from 100% to less than 3%. However the disk usage is back at 100% within a few days.

The only things using /home are NFSd, a custom script, and SSHd. This script moves files from /home/somedir to /home/tmp, then moves the files to /tmp, then parses the files. This script runs on all three boxes.

New files are placed in /home by a script that SCPs the files into a chroot envoriment setup in /home. Only a couple binaries plus a /dev/null node are in /home/somepath/....

I have used service nfs stop and service portmap stop to stop nfs, stopped sshd, validated that the script mentioned above isn't running, and then checked disk usage. It remains at 100%. Only unmounting and remounting it seems to clear it up.

An fsck -f of /home (while unmounted of course) shows that the file system is fine.

All boxes are fully updated and running CentOS 5.5.

Where is my disk space going?

share|improve this question
    
Is this NFSv3 or NFSv4? –  DerfK Feb 28 '11 at 21:11
    
NFSv4 I think. Whatever the default is for CentOS 5. –  GpMidi Feb 28 '11 at 21:20
    
Turns out it was using v3. Trying v4 now... –  GpMidi Feb 28 '11 at 21:44
    
No change when switching to v4 only - Still getting ever increasing disk usage. –  GpMidi Feb 28 '11 at 22:00
add comment

1 Answer

Speculation: your custom script, running on all three boxes, is tripping over the other instances of itself while trying to manipulate the same files from three places; NFS client running on the remote system tells its kernel that it has deleted a file, but it's actually been deleted by a process on another system in the interim, so the remote client gets stuck.

Solution: Use NFS to allow the remote systems to access the filesystem, but only run the cleanup script on the exporting system.

share|improve this answer
    
Possible. I'll try changing the mounts from async to sync. Since the whole move from /home/somedir to /home/tmp is to have an "atomic operation" that (at least probably) won't happen at the exact moment on all three boxes...well...i was hoping that would do. But perhaps not. –  GpMidi Mar 1 '11 at 4:42
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.