Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I wrote a script on CentOS and now I'm trying to run it on Ubuntu and it is behaving unexpectedly. Here's something like the script, which I run under bash:

sudo tail -F /var/opt/my-application/log/my-application.log |
awk '
BEGIN {
    ORS=" "
}
{ if ($8 ~ /MATCH-TEXT/) {
            # do a whole bunch of stuff here
            # like look in /proc and calculate CPU and interface stats
    };
 };
} '

Previously when a line with MATCH-TEXT appeared in the $8 field of my log all of the "do a whole bunch of stuff" (the meat of the script) would happen and I would see the output. My application generates MATCH-TEXT lines about once per second (sometimes more, sometimes less) and so does the output of this script.

Now, however, when I start this script I get no output for a long time and then I get maybe 30 lines of output all at once. It seems like the script is queuing the log lines it receives, and then executing several times quickly because the calculations on the CPU and interface are right for "since a tiny fraction of a second ago" rather than "since a second ago".

Stranger still is that commenting out all of the meat of the awk command to replace it with a "print $0" (turning the whole script into an awkward grep command) produces the same result. So I'm confident it is the way the system is executing this script and not something in the script itself.

Meanwhile, in another window a plain "sudo tail -F my-application.log | grep MATCH-TEXT" gets output every second or so (just as expected).

Any ideas what's going on here? Any hints where I should start hunting for what's causing this queueing behavior?

share|improve this question
1  
Sounds like a buffering problem. Have a look at stdbuf and/or unbuffer (expect). There was a thread about this a few days ago (either here or on unix). –  Hauke Laging Apr 24 '13 at 20:08

3 Answers 3

If this is not due to a difference in tail, then I agree with Hauke Laging that it is probably a buffering problem. To flush the output buffer of awk, try adding fflush() after your print statements like I did in the following test code:

tail -F /var/log/apache2/access_log | awk '{ if ($8 ~ /MATCH-TEXT/) { print $0; fflush(); }}'
share|improve this answer

mawk -W interactive fixed my problem on Debian Wheezy.

    tail -f data.log | grep --line-buffered '$DataString' | mawk -W interactive -F "," '{print "var2", $2, "var4", $4}'
share|improve this answer
up vote 0 down vote accepted

After researching all the answers and trying fflush and stdbuf (which I did not know about previously, thanks), I found the answer is in two parts:

1) CentOS uses gawk, Ubuntu uses mawk (by default).

I have no comment on which is better or why the decision to use either has been made, differences are what make the two distributions different. (:

2) mawk has a command line option for "interactive", from the manual page:

 -W interactive

 sets unbuffered writes to stdout and line buffered reads from stdin.
 Records from stdin are lines regardless of the value of RS.

I can only assume it is because the buffering was within awk and not in the script itself that makes this work where stdbuf did not.

share|improve this answer
    
Ubuntu may use mawk - it's the default. It depends on whether other versions of AWK are installed and what is pointed to by the alternatives system (see man update-alternatives). –  Dennis Williamson May 10 '13 at 0:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.