Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have been asked this question in two consecutive interviews, but after some research and checking with various systems administrators I haven't received a good answer. I am wondering if somebody can help me out here.

A server is out of disk space. You notice a very large log file and determine it is safe to remove. You delete the file but the disk still shows that it is full. What would cause this and how would you remedy it? And how would you find which process is writing this huge log file?

share|improve this question
3  
You need to talk to better systems administrators. This is trivial stuff. –  womble Mar 5 '12 at 1:45
1  
Trivial, but the situation and the question comes up often enough... –  ewwhite Mar 5 '12 at 2:02
    
Will the OP be able to accept this? –  ewwhite Mar 6 '12 at 1:38
5  
Trivial or not, for someone who doesn't speak *nix fluently (e.g. a primarily Windows admin) this is good stuff to learn. –  John Gardeniers Mar 6 '12 at 3:12
add comment

migrated from stackoverflow.com Mar 5 '12 at 1:32

This question came from our site for professional and enthusiast programmers.

5 Answers

This is a common interview question and a situation that comes up in a variety of production environments.

The file's directory entries have been deleted, but the logging process is still running. The space won't be reclaimed by the operating system until all file handles have been closed (e.g., the process has been killed) and all directory entries removed. To find the process writing to the file, you'll need to use the lsof command.

The other part of the question can sometimes be "how do you clear a file that's being written to without killing the process?" Ideally, you'd "zero" or "truncate" the log file with something like : > /var/log/logfile instead of deleting the file.

share|improve this answer
1  
... or fuser. –  Steven Monday Mar 5 '12 at 1:40
1  
Expanding a bit: until all references to a file on disk vanish, that space can't be used by something else. That includes file handles. That also allows this trick to work: serverfault.com/questions/45237/link-to-a-specific-inode –  Jeff Ferland Mar 5 '12 at 2:48
1  
If you have no-clobber set, try: >| /var/log/logfile –  Beaming Mel-Bin Mar 5 '12 at 11:35
2  
I ask a variant of this question on every interview: "You're getting disk full messages. df says your out of space, du says you're barely using any. What's causing it, and why don't the two tools agree?" –  voretaq7 Mar 5 '12 at 17:16
    
What to do if after > /var/log/file the space on disk still at 100%? The log file seems to be empty... but only after restarting the program that writes on this log file the space is recovered. Is there a way to recover the disk space without restarting the program? –  alemani Mar 9 '12 at 17:42
add comment

There's still another link to the file (either hard link or open file handle). Deleting a file only deletes the directory entry; the file data and inode hang around until the last reference to it has been removed.

It's somewhat common practice for a service to create a temporary file and immediately delete it while keeping the file open. This creates a file on disk, but guarantees that the file will be deleted if the process terminates abnormally, and also keeps other processes from accidentally stomping on the file. MySQL does this, for example, for all its on-disk temporary tables. Malware often uses similar tactics to hide its files.

Under Linux, you can conveniently access these deleted files as /proc/<pid>/fd/<filenumber>.

share|improve this answer
add comment

I'm not a sysadmin, but from what I've gathered on Unix.SE, a Linux system won't actually delete a file (mark the space as free/reusable) after it is unlinked until all file descriptors pointing to them have been closed. So to answer the first part, the space isn't yet free because a process is still reading it. To answer the second, you can see which process is using the file with lsof.

share|improve this answer
add comment

If the process writing the file is root, it'll write into the superuser reserved file space. The file system has this space to keep a system operational in case a user task fills up the disk. This space (imho per default 5%) is invisible to many tools.

lsof can show you, which process has locked the file, ergo is writing to it.

share|improve this answer
    
You can also adjust this reserve percentage using tune2fs. This can be a quick way to allow the server to continue running while you free up disk space. –  sjbotha Mar 5 '12 at 15:18
add comment

One alternative answer besides the obvious hard link/open file answer: that file is a (very) sparse file such as /var/log/lastlog on RHEL that wasn't actually taking up all that much space. Deleting it had very little impact, so you need to look at the next biggest file.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.