Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We have an authentication daemon on a Debian 6.0 box generating logs at a rate of about 4-6 lines\entries every second. For legal reasons this log file has to be written to disk locally first, before we can back it up or transfer it anywhere. These logs are rotated every 2 hours by the daemon itself.

I'd like to implement a system where the logs are entered into a database and looked into syslog-ng (the service logging to syslog-ng and then syslog-ng writing the entry to a db and disk) but this was rejected as if the syslog-ng process ever failed for whatever reason, we would loose the log entries.

I have been toying with using rsync and cron set to synchronise every 5 minutes or so, but the receiving end obviously requires a full copy of the log for rsync to 'compare' to, and really all I want is the difference between the previous check.

Is there a decent way I can pick up the changes in the log file (from a remote machine preferably) since the last check, and then feed just those lines to a script or directly to a database?

share|improve this question
up vote 1 down vote accepted

Any problem with using tail -F <logfile> | db-import-script? Quite honestly, the "legal reasons" you cite seem nonsensical, even for lawyers, and so I'd be inclined to seek clarification on the intent of the regulations, and then work to that rather than try to vet implementation details with people who presumably have no ability to assess the suitability of an implementation. If they do have the technical ability, then tell 'em to give you the design and you'll implement it to spec.

share|improve this answer
I did not know of the -F switch before and it's certainly opened up quite a few options. I could nicely have the script open up an SSH session and continually poll those results and push them into the db, with little extra load. – Tim Jones Jul 30 '12 at 12:28

I have to be up-front and admit that I'm one of their biggest customers but for this kind of thing I'd suggest you at least consider Splunk - if for no other reason it can be free and it's what we use for doing almost exactly what you're doing.

share|improve this answer
How's splunk going to solve the problem of the entry being written to disk locally before being sent anywhere else? – womble Jul 30 '12 at 10:00

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.