Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have written a python script that takes log entries from syslog-ng and writes them to MongoDB (I couldn't do it with afmongodb driver because I need to do some special processing).

This is how it looks like in syslog-ng.conf:

destination d_mongodb_events {
  program("/home/test/syslog_piper.py"
    template("$UNIXTIME|$PRIORITY|$FACILITY|$SOURCEIP|$SEQNUM|$PID|$PROGRAM|$MSGONLY\n")
    flags(no_multi_line)
    flush_lines(1)
    flush_timeout(1000)
  );
};

And this is the script: (with logic removed)

import sys
try:
    lines = sys.stdin.readlines()
    for line in lines:
        # process `line` and save to DB

except Exception, e:
    f = open('/tmp/error.txt','ab')
    f.write(e)
    f.close()
    exit(0)

Script works - that is, if I run it from the command line it waits for input and inserts the lines in DB as soon as I press Ctrl+D - and exits.

With syslog-ng it is different. The entries are passed to the script, but only after I stop syslog-ng daemon. Also, while syslog-ng is running, I see my script running too (in the process list). I have tried setting flush_lines() and flush_timeout() (see configuration above) but I can't make syslog-ng flush its output.

My guess is that there is something wrong with the way I handle pipelines, but I can't figure it out. Can anyone spot the problem?

UPDATE: if I send 1000 messages some of them get pushed through, so I guess there is some buffering going on. Anyone knows which setting to tune?

SOLUTION: It looks like Python buffers a lot of its input/output. This is from the man pages:

 -u       Force  stdin,  stdout  and  stderr to be totally unbuffered.  On
          systems where it matters, also put stdin, stdout and  stderr  in
          binary  mode.   Note  that there is internal buffering in xread-
          lines(), readlines() and file-object  iterators  ("for  line  in
          sys.stdin")  which  is  not  influenced by this option.  To work
          around this, you will want to use "sys.stdin.readline()"  inside
          a "while 1:" loop.

So basically I had to change the program to use sys.stdin.readline(). Huge thanks to Janne.

share|improve this question
add comment

1 Answer

up vote 2 down vote accepted

I have not hooked any Python scripts to syslog-ng, but with Perl scripts I have to turn off the output buffering before they work in real-time. In Perl-speak, that is $|=1.

I am not much a Python guy, but I guess running your Python script with -u or setting up variable PYTHONUNBUFFERED might help.

share|improve this answer
    
Janne, big THANK YOU for this one! :) Actually -u didn't help, but reading manual pages (man python) revealed this gem: -u: ... Note that there is internal buffering in xreadlines(), readlines() and file-object iterators ("for line in sys.stdin") which is not influenced by this option. To work around this, you will want to use "sys.stdin.readline()" inside a "while 1:" loop. Solved. :) –  johndodo Apr 25 '12 at 6:42
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.