0

I have a few servers running on AWS and have Nagios/Icinga doing the monitoring of all critical services.

We're trying to figure out the best way to monitor all logs - system, DB, PHP, Apache, etc - on the system so we know about issues (for e.g. that Apache reached the max_clients threshold yesterday) immediately via email. We only look at logs currently after a service goes down, not before, which is bad.

I'm new to Linux administration and I've identified the following options after a search online:

  1. Nagios scripts to monitor logs - The problem is most of them check one log file for one specific regex at a time. It's not scalable to install one service for each log file (I don't even know all the log files we have to monitor!)
  2. A service such as logrobot.com - I'm not sure how effective this is though.

Appreciate your advice on what's the best way to monitor all these logs on multiple servers with minimal configuration.

4 Answers 4

3

Don't know how much servers/logs you have to monitor but there are many solutions out there

small environment

Use rsyslog and a frontend you like (ex. LogAnalyzer http://loganalyzer.adiscon.com/)

bigger environment

We monitor our serverlogs from (+300 system) with beaver as logshipper, logstash as indexer and elasticsearch as backend. This solution scales up to [insert random number here] hosts ;)

2
  • Thanks for this. Do you also get alerts from any of those daemons on critical or other errors? If you could point me to a tutorial on setting these up, would be much obliged.
    – kouton
    May 10, 2014 at 12:48
  • We store our logs with custom severity in elasticsearch and use a (also custom) django/python webfrontend for monitoring and alerting. Tutorials for a setup of logstash+elasticsearch should be easily found with a search engine of your choice.
    – deagh
    May 10, 2014 at 13:00
1

Basically you should not (at least not only) read the logs on the same host but instead use some sort of logserver which would get all the logs of the servers centralised.

i used this setup to be sure the logs aren't altered after they were entered.

Additionally just use logcheck and let it check the logs for you.

Basically its a check for lines you find acceptable and can be ignored and only sends you the ones you did not tell logcheck to ignore beforehand.

you can easy install it on every server.

for a graphical version , counting how many severe log entries etc is

logzilla a nice option, thought not free anymore.

0

I spent a couple of days searching ("log management solutions"), I discovered just the tools I was looking for. The following 3 tools are Cloud based logging tools, are easy to setup and configure. They ship system logs and custom logs to their servers, store them, let you search and setup email/webhook alerts for regex patterns.

  1. Papertrail - the simplest/quickest interface by far (like tail -f on a terminal). Extremely affordable pricing as well. You'll have to spend some time configuring it for custom logging (apache, mysql, your application) though. Their log-shipper based out of Go (in Beta as of today) is very memory efficient and I can deploy the log files it has to monitor through a GIT repo.
  2. Log entries - also quite simple. Easiest for setting up custom logging through their 'le' daemon. It has quite a few features, and this made it seem bloated compared to papertrail. Their free plan's quite extensive for startups.
  3. Loggly - Offer everything the other two do, but it was quite complex to go through this. And their free plan doesn't offer alerts.
0

With regards to logrobot.com, there's now a free version of it that does exactly what you need and it can be downloaded here:

http://www.logxray.com/logxray.zip

To use it to address your concerns, you can run logxray this way:

./logxray localhost:emailing /apps/logxray autonda /var/log/messages 60m 'kernel|error|panic|fail' 'timed out' 1 2 -show error_check [email protected]

To monitor multiple logs or specific logs within a specific directory:

./logxray localhost /apps/logxray autoblz /var/log 60m ‘panic|error’ ‘.’ 1 1 directory_error_watch -ndfoundn

http://www.logXray.com (for more information or documentation on how to use the tool)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .