Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have this problem or rather an itch that I could really use help with. I use autoscale and several instances on Amazon. Problem is I have all these logs PHP, Apache, system but also my web app generating traffic logs etc etc. I want to consolidate all these logs. I know syslogd etc exist, but I was wondering whats the most effective , low administrative solution devops are using currently since my experience on Amazon AWS isn't that extensive. Plus I need the solution to be low resource usage if possible. Hitting database to save all that data every second is not a good solution for me.

Please note I am NOT talking about system monitoring which is served by CloudWatch etc. more for the applications and servers running as well as writing custom application logs. Also all systems are assumed to be Redhat Linux.

Any help or direction is appreciated.

share|improve this question

1 Answer 1

up vote 3 down vote accepted

The simplest, easy solution is probably syslogging to a central logserver, e.g. with syslogd. It's simple, free, and widely used in production environments.

You could then use Graylog2 as a free log analyzer. It's a bit like Splunk, but much more affordable.

Also interesting for your toolbox is logstash, a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). You can use the built-in web interface or search the logs via Graylog2, if you prefer its interface. Logstash helps you take logs and other event data from your systems, normalize them and move it into a central place.

share|improve this answer
    
i am still considering syslog, however I have some doubts about log rotation. what do you think about munin. btw thanks for the fast answer. Edit: great tip on graylog2. it looks good at first glance. going over it now. if this works out I owe you donuts –  Vangel Jun 23 '11 at 5:02
    
I'm not experienced with Munin. I use Zenoss Core to poll our servers via SNMP. It generates alerts and provides trend graphs, however it doesn't store logs, so researching the cause of an alert often requires additional info. What are your doubts about rotation? –  Martijn Heemels Jul 5 '11 at 23:26
    
I like donuts, but an upvote would taste great as well. :) –  Martijn Heemels Jul 5 '11 at 23:28
    
yes graylog combined with AMQP did my work perfectly. Awesome suggestion and to anyone else who finds this QA helpful know this that I mark it as correct after implementing the suggestion in production. Martin: Will also check out your suggestion for SNMP. Rotation, havent worked this out on graylog yet. lets see when i get to that bridge :D –  Vangel Jul 26 '11 at 17:02
    
I updated my answer to add info on log stash, a tool I recently discovered. It seems to be excellent in combination with Graylog2, with Logstash doing the collecting and normalizing of logs and events, and Graylog2 as the search interface. Thanks for the feedback in the comments. –  Martijn Heemels Sep 3 '11 at 23:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.