Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've built a system that needs to collect new apache logs from four machines every hour, merge them, process them and store them. For the task of collecting the new logs I've written a bunch of cron script that reads in the line of already sent lines and start sending only the new lines, then store the new line number. This is synchronized with a merging script using some .lock files.

Although this is working quite well I would like to substitute this system with a more reliable system with good error handling and perhaps autodetect of new log files (they are rotated every day). Is there something I could use to do this kind of task? Thank you!

share|improve this question
add comment

4 Answers

There are many solutions for collecting/monitoring logs.

Some of them: syslog, syslog-ng, logsurfer, splunk... and I think much more. But I will recommend syslog-ng.

It is very configurable (for example he can store logs in database wich sometimes is very useful), elastic and do all you want.

With syslog-ng and MongoDB you can make quite fun stuff using mojology(A simple application to browse syslog messages stored in MongoDB, as logged by syslog-ng).

You can use any of this software.

Zabbix can collect logs but it's much more harder to configure what you want.

Syslog is not bad idea but sysylog-ng has much more features. Here is comparison of syslog and syslog-ng http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/features/comparison

So I would choose syslog-ng :)

share|improve this answer
1  
+1 for syslog-ng, it's definitely the best syslog daemon available. Though for many purposes the now-widespread and popular rsyslogd is not bad either, and for simple "Send your logs to central log server X" and syslogd will do. But at that centralized log server I would install syslog-ng. –  Janne Pikkarainen May 23 '12 at 8:23
add comment

I think it will be a good idea to use syslog server. You can send all your logs to this server and process them after that. Syslog enables you also to define filters and add multiple destinations for logged information.

share|improve this answer
    
Thank you, I will investigate that. –  angaran May 23 '12 at 7:50
add comment

I'm not sure if syslog is really the right tool to use for apache logs.

On the four webservers, I would simply rotate the logs once every hour.

Then, from a central log server, I would fetch all logs using rsync and do all the 'smart' stuff (merging logs etc.) at one central location.

share|improve this answer
    
Logs have a filename that contains the current date like access_log20120523. I can not change their name every hour, I can only do it once a day. Can rsync do incremental sync? How does it keep "state"? –  angaran May 23 '12 at 8:30
    
As far as I know, unless you specify --whole-file rsync will automatically use its delta-transfer algorithm, which will only transfer the changes between the source and destination file. The "state" is the file. –  Oliver May 23 '12 at 8:41
    
I see, my problem is that I should do all the calculation only on the new portion of the logs because it would take too much time instead. I'll think about keeping a second copy of the logs divided by hour so I can integrate with the existing environment –  angaran May 23 '12 at 10:22
add comment

There is are nice tool to aggregate webserver logs, writen by Dmitry Koterov. It is specially designed to overcome some flaws of syslog solution. See http://en.dklab.ru/lib/dklab_logreplica/

Another nice way is to use log management software, like splunk, or logstash.

share|improve this answer
    
logstash is a pretty interesting project, thanks! And open source too. –  angaran May 23 '12 at 16:48
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.