Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have noticed recently that sometimes tail -f <logfile> will stop updating to the screen.

Doing a Ctrl>-C and restarting the tail works fine, though. And I checked to make sure the logfile isn't being rotated midstream (which can make tail lose its mind).

What would cause this? I'm running RHEL 5.2 x64.

share|improve this question
    
Does this happen on all logfiles, or just certain ones? What filesystem is the logfile on? What application is writing the logfile? Have you timed the failure to see if it happens either (1) a certain period of time after you started the tail command; or (2) at certain fixed intervals (eg always at :00 :10 :20 :30 :40 and :50 after the hour)? –  daveadams Feb 8 '11 at 18:42
    
@daveadams - so far as I've noticed, it's only been on two (one on each of two servers): in this case, the dataminer logfile for BSAE (reporting tool) on an HPSA Core server (data center automation) and the server.log for BSAE on the reporting host –  warren Feb 8 '11 at 19:05
    
as an aside, tail -F will continue to work even if the file is removes and later replaced by another file entirely. –  Sirex Feb 16 '11 at 16:30

4 Answers 4

up vote 4 down vote accepted

Try wrapping your tail command with strace if you have it:

strace -Tt -o /tmp/tail.trace tail -f /var/log/messages

Then just for crazy recursive kicks you can tail the strace output (doesnt' matter if that breaks because its going out to a file):

 tail -f /tmp/tail.trace

Mine looks like:

8:39:00 write(1, "ng SMAC\n", 8)       = 8 <0.000026>
18:39:00 read(3, "", 0)                 = 0 <0.000019>
18:39:00 fstat64(3, {st_mode=S_IFREG|0640, st_size=92990, ...}) = 0 <0.000019>
18:39:00 fstatfs64(3, 84, {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=4807069, f_bfree=1924458, f_bavail=1680271, f_files=1221600, f_ffree=820806, f_fsid={-1331083162, -1313908385}, f_namelen=255, f_frsize=4096}) = 0 <0.000021>
18:39:00 inotify_init()                 = 4 <0.000033>
18:39:00 inotify_add_watch(4, "/var/log/messages", IN_MODIFY|IN_ATTRIB|IN_DELETE_SELF|IN_MOVE_SELF) = 1 <0.000041>
18:39:00 fstat64(3, {st_mode=S_IFREG|0640, st_size=92990, ...}) = 0 <0.000019>
18:39:00 read(4,

The -t switches on the time and -T switches on time spent in calls.

Hit return 4 or 5 times to make a bit of vertical space, then wait for it to stop tailing. Hopefully there will be some clues in the output.

share|improve this answer

Try using:

tail --follow=name <logfile>

And see if that works better. You don't have to worry about it being rotated out from under you.


Any pattern to it stopping? Certain length of time? Certain time of day?

share|improve this answer
    
the file is not being rotated-out from underneath tail - it's just periodically (sometime between 2 and 20 hours) stopping to follow anymore.. wish there was more of a pattern :-\ –  warren Feb 8 '11 at 18:29
    
If you hit other keys on the keyboard (other than ctrl-c) does it get better? When it stops, you could try attaching to it using strace/ltrace/etc. to see if it's doing anything. –  MikeyB Feb 8 '11 at 18:31
    
good thought on the strace - neither enter nor other keys get anything to happen either; I sometimes like to leave a tail-f running in a screen session for extended debugging, and this is worrisome –  warren Feb 8 '11 at 18:35
    
Ah… possibly not your issue, but ensure you don't accidentally leave a screen window open in copy mode with tail -f running. Copy mode will make command output block eventually (when the buffer fills up). –  MikeyB Feb 9 '11 at 1:37

Given that both problematic logfiles are written by different components of the same application, I wonder if it's not some part of the logging code for that application that's causing the problem. I propose two tests to get a better idea of what's going on:

  • Note the inode of the logfile (ls -i logfile) prior to starting the tail, and once the tail fails, check it again. If the inode has changed, then the logger is rewriting the entire logfile, which would break tail's connection.

  • Note the last line before tail stops working, and then visit the file and find the first log entry after that line. Do this 3-5 times if possible. If it's a problem with the program doing the logging, the part of the program that wrote the log entry immediately after you see tail break is most likely responsible. If that log entry is always the same, or if it comes from the same component of the program, you may have enough data to submit a problem report to the application vendor.

Good luck.

share|improve this answer

Having the same problem here.

Problem was, that the file i'm watching was mounted from a different machine. The change notify was not propagated through the mount.

Solution was to use tail on the original machine.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.