0

I have a CentOS 5.5 box with a seperate partition for /home. df -h shows it at 100% usage. df -i shows tons of free inodes. du -sh /home /home* shows that there are about 100MB worth of files in /home. lsof -n|grep /home shows no open files in /home. lsof -n|grep deleted shows no files touching /home.

The partition is shared out via NFS and is mounted on two other CentOS 5.5 clients. lsof -n|grep /home and lsof -n|grep deleted show no files for for /home or /.

When I unmount and remount the partition, disk usage drops from 100% to less than 3%. However the disk usage is back at 100% within a few days.

The only things using /home are NFSd, a custom script, and SSHd. This script moves files from /home/somedir to /home/tmp, then moves the files to /tmp, then parses the files. This script runs on all three boxes.

New files are placed in /home by a script that SCPs the files into a chroot envoriment setup in /home. Only a couple binaries plus a /dev/null node are in /home/somepath/....

I have used service nfs stop and service portmap stop to stop nfs, stopped sshd, validated that the script mentioned above isn't running, and then checked disk usage. It remains at 100%. Only unmounting and remounting it seems to clear it up.

An fsck -f of /home (while unmounted of course) shows that the file system is fine.

All boxes are fully updated and running CentOS 5.5.

Where is my disk space going?

Improve this question
4
  • Is this NFSv3 or NFSv4?
    – DerfK
    Feb 28, 2011 at 21:11
  • NFSv4 I think. Whatever the default is for CentOS 5.
    – GpMidi
    Feb 28, 2011 at 21:20
  • Turns out it was using v3. Trying v4 now...
    – GpMidi
    Feb 28, 2011 at 21:44
  • No change when switching to v4 only - Still getting ever increasing disk usage.
    – GpMidi
    Feb 28, 2011 at 22:00

1 Answer 1

Reset to default
1

Speculation: your custom script, running on all three boxes, is tripping over the other instances of itself while trying to manipulate the same files from three places; NFS client running on the remote system tells its kernel that it has deleted a file, but it's actually been deleted by a process on another system in the interim, so the remote client gets stuck.

Solution: Use NFS to allow the remote systems to access the filesystem, but only run the cleanup script on the exporting system.

Improve this answer
1
  • Possible. I'll try changing the mounts from async to sync. Since the whole move from /home/somedir to /home/tmp is to have an "atomic operation" that (at least probably) won't happen at the exact moment on all three boxes...well...i was hoping that would do. But perhaps not.
    – GpMidi
    Mar 1, 2011 at 4:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .