Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've got a Mac OS X 10.6.4 Snow Leopard Server file server (AFP) which has been running out of storage space on the boot volume for the past few weeks. It takes about two days for the remaining 42GB on the 80GB boot volume to be eaten up, even though a sudo du -chsx -I dev / still shows only 29GB used.

I've run into this in the past w/a Linux server who's Apache logs were deleted after N days, but the log was still kept open by Apache, causing the storage to not be freed. I had been able to track it down relatively easily in that case w/a sudo lsof, but I'm not easily finding the culprit in this case (being a file server, there are a ton of open files and sockets). How can I sort lsof output by file size (and show sized in a human friendly format) so I can find the culprit?

The server has 3GB of RAM. After being up 4hrs, Activity Monitor shows 700MB free, 1.5GB inactive, and 200GB of VM. mds has the largest VM usage at 1.8GB, AppleFileServer in 2nd place w/500MB, and everything else is using 10MB-75MB of VM. That said, /private/var/vm is only 128MB.

Rebooting the system clears the issue, hence my belief that it's free storage space that's still being held open by some process or processes.

Any other hypothesis, suggestions, etc., greatly appreciated.

share|improve this question

2 Answers 2

up vote 7 down vote accepted

You can find the biggest open files with:

sudo lsof -s | awk '$5 == "REG"' | sort -n -r -k 7,7 | head -n 50

This will list the regular files (not pipes, sockets, etc) sort by size in descending order, and take the top 50.

You might also look at what processes have the most files open, with something like

sudo lsof | awk '$5 == "REG" {freq[$2]++ ; names[$2] = $1 ;} END {for (pid in freq) print freq[pid], names[pid], pid ; }' | sort -n -r -k 1,1
share|improve this answer
    
Thanks, those two work perfectly. Will still need to dig and explore to determine what the culprit is, but those are much needed tools. –  morgant Nov 29 '10 at 22:11
    
FYI, the second awk command is taken almost verbatim from the "Word Sorting" page of the GNU awk manual. –  Ben Jencks Nov 29 '10 at 23:18
    
Those two examples were enough to get my head around the syntax w/the awk man page to tweak them as necessary. I've found the culprit: lots (30, at this time) of 1.2GB files like /private/var/folders/zz/zzzivhrRnAmviuee+++++++++++/-Tmp-/termIdFile.XXXXXX which are open by mds (Spotlight), but result in a file-not-found error when trying to list/stat them. So, I just need to determine why mds is holding onto them. –  morgant Nov 30 '10 at 16:11

I'd look at sudo lsof | grep deleted

share|improve this answer
    
That doesn't return any results. The man page for lsof 4.82 notes that "(deleted)" will be appended under Solaris 10 and later if using the "-X" option (which isn't even available on OS X). The man page also implies that under Linux there's a "DEL" type for deleted maps, but sudo lsof -s | awk '$5 == "REG"' doesn't pull up anything. –  morgant Nov 29 '10 at 22:20
    
Ah sorry, it works on Linux –  James Lawrie Nov 29 '10 at 22:34
    
I'll certainly try it again when the volume is nearly full to see if maybe there just weren't any culprits at the time I ran sudo lsof | grep deleted. –  morgant Nov 29 '10 at 23:00
    
I've found the culprits: lots of files like /private/var/folders/zz/zzzivhrRnAmviuee+++++++++++/-Tmp-/termIdFile.XXXXXX being held open by mds (Spotlight), but which are no longer files on disk. Unfortunately, they neither have "deleted" appended to their description by lsof, nor have a type of "DEL". I still appreciate the suggestion. –  morgant Nov 30 '10 at 16:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.