up vote 6 down vote favorite
share [g+] share [fb]

I have file servers which are used to store files. Files might reside there for a week, or for a year. Unfortunately, when I remove files from the server, df command doesnt reflect the freed up space, so eventually, the server gets filled up (df shows 99%), and my script doesn't send any more files there, except there might be a few dozen GB of free space on there.

I got noatime flag on the mounted partitions if that makes any difference.

link|improve this question
91% accept rate
Is this happening on a single partition or on all partitions? – Khaled Feb 8 '11 at 7:40
Well, its happening on my main data partition, which is the only one I care about, since I only write/remove files onto it. – k1lljoy Feb 8 '11 at 8:09
50 % of serverfaults's questions had been already asked and answered at least twice. – poige Feb 8 '11 at 8:27
Please enlighten me with the solution, or a link to one. – k1lljoy Feb 8 '11 at 8:51
What filesystem(s)? DF does a stat of the superblock, it may be that your filesystem is not updating the sb inode. Have you tried flushing cache? – beans Feb 8 '11 at 17:32
show 2 more comments
feedback

3 Answers

up vote 8 down vote accepted

Deleting the filename doesn't actually delete the file. Some other process is holding the file open, causing it to not be deleted; restart or kill that process to release the file.

link|improve this answer
Files that are removed were not accessed in over a month, and the only process that accesses them is nginx, so its doubtful. – k1lljoy Feb 8 '11 at 8:08
7  
+1. Also, "lsof +L1" will tell you which program is holding the files open. – pehrs Feb 8 '11 at 8:10
3  
as root run "lsof -n | grep file," you'd be surprised at how long files can stick around due to processes keeping them open for whatever reason. If all else fails, reboot, I feel bad suggesting it but it will definitely make sure nothing is holding on to the file. Per pehrs, lsof +L1 is probably the better way to go. – ScottZ Feb 8 '11 at 8:11
That returns a single file, which is being encoded with ffmpeg. – k1lljoy Feb 8 '11 at 8:16
feedback
up vote 1 down vote

One possibility is that the file(s) you deleted have more references in the filesystem. If you've created hardlinks, several filenames will point to the same data, and the data (the actual contents) won't be marked as free/usable until all references to it has been removed. Before you delete files, either stat them (Entry named Links) or do ls -l on them (should be the second column).

If it does turn out that the files are referenced elsewhere, I guess you'll have to ls -i the file(s) to find the inode-number, and then do a find with -inum <inode-number> to find the other references to that file (you probably also want to use -mount to stay within the same filesystem as well).

link|improve this answer
feedback
up vote 0 down vote

The other answers are correct: If you delete a file, and space does not get freed, it's usually either because the file is still kept open, or there are other hardlinks to it.

To help in troubleshooting, use a tool that tells you where the drive space is being spent: You can use du to get an overview of where space is going. Even better, use a graphical tool like xdiskusage (there are many like this) to hunt down the culprit. xdiskusage and friends let you drill down into the biggest space hogs to find where space is going.

That way, you'll quickly find files that still occupy space because of a second hardlink. It will also show space occupied by deleted, but open files (as (permission denied), I believe, since it cannot read the file name).

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.