Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

A process identified by process id 2147 is running, but its logfile is nowhere to be found. The filesystem is ext3 and 100MB of space remains. nothing went wrong in process 2147. An lsof of process 2147 reveals that the log file is still open.

a. What could have happened to make the log file disappear?

b. Can you access the contents of the file? If so, how?

share|improve this question
Could really use more info, such as what daemon the process is, but, it might be possible that the file was opened, and subsequently a volume was mounted over the folder that containted the file, thus hiding it. – JeffG Mar 2 '11 at 19:54

Ad a). The file was deleted, possibly by process 2147 itself. It's a common practice if you need a scratch file (not a log file) to create a file, get a handle to it and then delete it. A delete will remove entry in the directory in which the file was created, so it is not possible for other processes to access it. A handy scratchpad that no-one can look into. When the process closes the file and there are no more handles to it, the file system will mark blocks occupied by the scratchpad file data as unused. If it really was a log file, it could have been deleted (e.g. by accident), so the process still writes to its log, but no one else can read it (think: write only memory ;) ).

Ad b) You cannot access the contents of the file in the normal way, i.e. accessing file system in a normal way. You would have to start accessing the data below the filesystem, looking for blocks which hold scratchpad's data. If you want to try that, then mount the filesystem holding the deleted file read only as soon as you kill 2147, or you risk the data being overwriten once 2147 quits and the blocks holding its data will be marked as unused. How exactly find the data blocks you need? Sorry, cant help you there :(.

share|improve this answer

Is it possible the file has been deleted? If you delete a file on Linux to which a process still has a handle opened, then it'll not be visible on the filesystem, but will still stay open until the process exits.

share|improve this answer

If the directoty that the log file is in has been overmounted (that directory or one of its parents has been used as a mount point), then you may not see the log file. This is only likely if the log file descriptor was opened before the mount occured, AND the file descriptor has NOT been closed (i.e. the program isn't closing the log file after each entry and reopening it to write subsequent entries).

If this is happening there will be some evidence in differences between the output of the mount command and /proc/2147/mounts. You might (i'm guessing) be able to access the over-mounted file (a big might) via the file descriptor links in /proc/2147/fd.

share|improve this answer

As mentioned in Relinking a deleted file, you can use the /proc/<pid>/fd/N from lsof to get a copy of the open file descriptor and copy the data out. See also Bring back deleted files with lsof. Note that you should make the original process stop writing to the file descriptor (e.g. stop the process). Also, it seems to me that you'd want to lseek to position 0 on the file descriptor (would require a little code), but I haven't any article mention it, so maybe I'm mistaken.

The first article also mentions the somewhat riskier technique of using debugfs also.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.