I've built a system that needs to collect new apache logs from four machines every hour, merge them, process them and store them. For the task of collecting the new logs I've written a bunch of cron script that reads in the line of already sent lines and start sending only the new lines, then store the new line number. This is synchronized with a merging script using some .lock files.

Although this is working quite well I would like to substitute this system with a more reliable system with good error handling and perhaps autodetect of new log files (they are rotated every day). Is there something I could use to do this kind of task? Thank you!


There are many solutions for collecting/monitoring logs.

Some of them: syslog, syslog-ng, logsurfer, splunk... and I think much more. But I will recommend syslog-ng.

It is very configurable (for example he can store logs in database wich sometimes is very useful), elastic and do all you want.

With syslog-ng and MongoDB you can make quite fun stuff using mojology(A simple application to browse syslog messages stored in MongoDB, as logged by syslog-ng).

You can use any of this software.

Zabbix can collect logs but it's much more harder to configure what you want.

Syslog is not bad idea but sysylog-ng has much more features. Here is comparison of syslog and syslog-ng http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/features/comparison

So I would choose syslog-ng :)

  • 1
    +1 for syslog-ng, it's definitely the best syslog daemon available. Though for many purposes the now-widespread and popular rsyslogd is not bad either, and for simple "Send your logs to central log server X" and syslogd will do. But at that centralized log server I would install syslog-ng. – Janne Pikkarainen May 23 '12 at 8:23

I think it will be a good idea to use syslog server. You can send all your logs to this server and process them after that. Syslog enables you also to define filters and add multiple destinations for logged information.


I'm not sure if syslog is really the right tool to use for apache logs.

On the four webservers, I would simply rotate the logs once every hour.

Then, from a central log server, I would fetch all logs using rsync and do all the 'smart' stuff (merging logs etc.) at one central location.

  • Logs have a filename that contains the current date like access_log20120523. I can not change their name every hour, I can only do it once a day. Can rsync do incremental sync? How does it keep "state"? – angaran May 23 '12 at 8:30
  • As far as I know, unless you specify --whole-file rsync will automatically use its delta-transfer algorithm, which will only transfer the changes between the source and destination file. The "state" is the file. – Oliver May 23 '12 at 8:41
  • I see, my problem is that I should do all the calculation only on the new portion of the logs because it would take too much time instead. I'll think about keeping a second copy of the logs divided by hour so I can integrate with the existing environment – angaran May 23 '12 at 10:22

There is are nice tool to aggregate webserver logs, writen by Dmitry Koterov. It is specially designed to overcome some flaws of syslog solution. See http://en.dklab.ru/lib/dklab_logreplica/

Another nice way is to use log management software, like splunk, or logstash.

  • logstash is a pretty interesting project, thanks! And open source too. – angaran May 23 '12 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.