1

We have an authentication daemon on a Debian 6.0 box generating logs at a rate of about 4-6 lines\entries every second. For legal reasons this log file has to be written to disk locally first, before we can back it up or transfer it anywhere. These logs are rotated every 2 hours by the daemon itself.

I'd like to implement a system where the logs are entered into a database and looked into syslog-ng (the service logging to syslog-ng and then syslog-ng writing the entry to a db and disk) but this was rejected as if the syslog-ng process ever failed for whatever reason, we would loose the log entries.

I have been toying with using rsync and cron set to synchronise every 5 minutes or so, but the receiving end obviously requires a full copy of the log for rsync to 'compare' to, and really all I want is the difference between the previous check.

Is there a decent way I can pick up the changes in the log file (from a remote machine preferably) since the last check, and then feed just those lines to a script or directly to a database?

2 Answers 2

1

Any problem with using tail -F <logfile> | db-import-script? Quite honestly, the "legal reasons" you cite seem nonsensical, even for lawyers, and so I'd be inclined to seek clarification on the intent of the regulations, and then work to that rather than try to vet implementation details with people who presumably have no ability to assess the suitability of an implementation. If they do have the technical ability, then tell 'em to give you the design and you'll implement it to spec.

1
  • I did not know of the -F switch before and it's certainly opened up quite a few options. I could nicely have the script open up an SSH session and continually poll those results and push them into the db, with little extra load.
    – Tim Jones
    Jul 30, 2012 at 12:28
1

I have to be up-front and admit that I'm one of their biggest customers but for this kind of thing I'd suggest you at least consider Splunk - if for no other reason it can be free and it's what we use for doing almost exactly what you're doing.

1
  • How's splunk going to solve the problem of the entry being written to disk locally before being sent anywhere else?
    – womble
    Jul 30, 2012 at 10:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .