4

I've got a Mac OS X 10.6.4 Snow Leopard Server file server (AFP) which has been running out of storage space on the boot volume for the past few weeks. It takes about two days for the remaining 42GB on the 80GB boot volume to be eaten up, even though a sudo du -chsx -I dev / still shows only 29GB used.

I've run into this in the past w/a Linux server who's Apache logs were deleted after N days, but the log was still kept open by Apache, causing the storage to not be freed. I had been able to track it down relatively easily in that case w/a sudo lsof, but I'm not easily finding the culprit in this case (being a file server, there are a ton of open files and sockets). How can I sort lsof output by file size (and show sized in a human friendly format) so I can find the culprit?

The server has 3GB of RAM. After being up 4hrs, Activity Monitor shows 700MB free, 1.5GB inactive, and 200GB of VM. mds has the largest VM usage at 1.8GB, AppleFileServer in 2nd place w/500MB, and everything else is using 10MB-75MB of VM. That said, /private/var/vm is only 128MB.

Rebooting the system clears the issue, hence my belief that it's free storage space that's still being held open by some process or processes.

Any other hypothesis, suggestions, etc., greatly appreciated.

Improve this question

3 Answers 3

Reset to default
18

You can find the biggest open files with:

sudo lsof -s | awk '$5 == "REG"' | sort -n -r -k 7,7 | head -n 50

This will list the regular files (not pipes, sockets, etc) sort by size in descending order, and take the top 50.

You might also look at what processes have the most files open, with something like

sudo lsof | awk '$5 == "REG" {freq[$2]++ ; names[$2] = $1 ;} END {for (pid in freq) print freq[pid], names[pid], pid ; }' | sort -n -r -k 1,1
Improve this answer
3
  • Thanks, those two work perfectly. Will still need to dig and explore to determine what the culprit is, but those are much needed tools.
    – morgant
    Nov 29, 2010 at 22:11
  • FYI, the second awk command is taken almost verbatim from the "Word Sorting" page of the GNU awk manual.
    – Ben Jencks
    Nov 29, 2010 at 23:18
  • Those two examples were enough to get my head around the syntax w/the awk man page to tweak them as necessary. I've found the culprit: lots (30, at this time) of 1.2GB files like /private/var/folders/zz/zzzivhrRnAmviuee+++++++++++/-Tmp-/termIdFile.XXXXXX which are open by mds (Spotlight), but result in a file-not-found error when trying to list/stat them. So, I just need to determine why mds is holding onto them.
    – morgant
    Nov 30, 2010 at 16:11
2

I'd look at sudo lsof | grep deleted

Improve this answer
4
  • That doesn't return any results. The man page for lsof 4.82 notes that "(deleted)" will be appended under Solaris 10 and later if using the "-X" option (which isn't even available on OS X). The man page also implies that under Linux there's a "DEL" type for deleted maps, but sudo lsof -s | awk '$5 == "REG"' doesn't pull up anything.
    – morgant
    Nov 29, 2010 at 22:20
  • 1
    Ah sorry, it works on Linux
    – James L
    Nov 29, 2010 at 22:34
  • I'll certainly try it again when the volume is nearly full to see if maybe there just weren't any culprits at the time I ran sudo lsof | grep deleted.
    – morgant
    Nov 29, 2010 at 23:00
  • I've found the culprits: lots of files like /private/var/folders/zz/zzzivhrRnAmviuee+++++++++++/-Tmp-/termIdFile.XXXXXX being held open by mds (Spotlight), but which are no longer files on disk. Unfortunately, they neither have "deleted" appended to their description by lsof, nor have a type of "DEL". I still appreciate the suggestion.
    – morgant
    Nov 30, 2010 at 16:08
1

sudo lsof +L1 will show deleted files. For speed, combine this with -Pn.

e.g. lsof -Pn +L1

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .