We have an authentication daemon on a Debian 6.0 box generating logs at a rate of about 4-6 lines\entries every second. For legal reasons this log file has to be written to disk locally first, before we can back it up or transfer it anywhere. These logs are rotated every 2 hours by the daemon itself.
I'd like to implement a system where the logs are entered into a database and looked into syslog-ng (the service logging to syslog-ng and then syslog-ng writing the entry to a db and disk) but this was rejected as if the syslog-ng process ever failed for whatever reason, we would loose the log entries.
I have been toying with using rsync and cron set to synchronise every 5 minutes or so, but the receiving end obviously requires a full copy of the log for rsync to 'compare' to, and really all I want is the difference between the previous check.
Is there a decent way I can pick up the changes in the log file (from a remote machine preferably) since the last check, and then feed just those lines to a script or directly to a database?