Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to figure out how other people implement their log management systems.

I have 20-30 Linux servers and a few Windows boxes (most of them virtualized). We utilize a lot of Perl and Bash scripts to do most of our automated jobs and I'm trying to standardize their logging.

I've been looking at log4perl and log4sh for logging of scripts and syslog-ng to get all the logs on a centralized logging server. I've also read up on splunk, even though is sounds like the enterprise edition is pretty pricey and I might go over the free license limit with all my servers.

I've seen other tools like swatch and logcheck, but I'm not quite sure how all these pieces fit together... Any recommendations would be greatly appreciated!

share|improve this question
    
add comment

7 Answers 7

I've got about 30 servers, and I just use straight up syslog to send all the logs to a single logging server. For backup, all of the machines are also configured to store their own logs locally for a few days, using logrotate to take care of the rotation and deletion of old logs.

Each of my application servers runs a small perl script to send their logs to syslog, which then forwards on to the loghost (perl script below).

Then on the loghost we have some custom scripts that are similar to logcheck that basically watch the incoming logs for anything suspicious.

We also have all of the email from every host going to one place, so that if any program complains that way, we get all the messages. This could theoretically go to a single mailbox that a program could act on and analyze.

Here is my logging perl script. It works by piping the program's output into it, and then it syslogs the output and spits it back out so you can send it elsewhere (I send to multilog). You can also give it the -q option to just go to syslog.

#!/usr/bin/perl

use Sys::Syslog;
use Getopt::Long;

$SERVER_NAME = `hostname`;
chomp $SERVER_NAME;
$FACILITY = 'local0';
$PRIORITY = 'info';

GetOptions ('s=s' => \$SERVER_NAME, 'f=s' => \$FACILITY, 'p=s' => \$PRIORITY, 'q+' => \$quiet);

#print "$SERVER_NAME\n$FACILITY\n$PRIORITY\n";

#Sys::Syslog::setlogsock('unix');
openlog ($SERVER_NAME,'ndelay',$FACILITY);

if (!($quiet)) {syslog($PRIORITY,"Logging Started -- Logger version 1.1");}

$| = 1;

while (<>) {
    if (!($quiet)) {print $_ unless $_ =~ /^\s+$/};
    chomp;
    syslog($PRIORITY,$_) if $_;
}

closelog;

$| = 0;
share|improve this answer
    
The script is pretty handy, but with syslog on clients and syslog-ng on a server (or even syslog-ng on clients too) you can get this functionality with more control over filtering the logs. –  thepocketwade Aug 6 '09 at 19:11
    
@thepocketwade: Very true. I just never needed the additional functionality. –  jedberg Aug 7 '09 at 16:37
add comment

Although I haven't implemented it yet, I'm planning on moving all of my log-generating machines to rsyslog, and implementing a bastion-type server which will function as the collector of syslogs. From there, I think the free version of Splunk can do everything I need to pull out information.

Now just to implement it...

share|improve this answer
add comment

I use a central syslog host. Each edge system sends *.debug to the central loghost. The central syslog host runs syslog-ng, and has rules to split logs so that each machine generates its own files named for that day. It also dumps everything into a single file, against which I run a descendant of logcheck.sh.

Once a day I run a log compacter, which zips up any logs older than 7 days, and deletes anything older than 28 days. Between the two, it gives logs an expected life of 35 days on the server, which means that all logs should make it to monthly backups, where they can be recovered for up to two years.

It's storage-intense, but seems to be the best way to assure coverage.

share|improve this answer
    
I've got a similar system, but my log server has predefined folders (mail, auth, catchall) that logs are filtered to. At one point I was looking into using splunk. I could forward data from the log server to the splunk server easily. –  thepocketwade Aug 6 '09 at 19:10
add comment

We use an appliance from LogLogic for our enterprise logging. It's based on syslog, so all *nix boxes have no problem using it; there is a small app that needs to be installed on windows servers. I can search on anything I want, including REGEX queries, and it seems to be able to handle quite a bit of load(our Active Directory setup alone generates a mind boggling amount of traffic).

share|improve this answer
1  
Just be careful evaluating their products... I got about 10 calls/emails from them, they are VERY persistent. –  Flamewires Dec 5 '11 at 23:22
    
I think this can be said for just about any vendor these days, and has no bearing on the actual product's functionality itself. You don't want to know how often DELL, EMC, etc. come knocking/calling around here.... –  Tatas Dec 7 '11 at 16:22
add comment

For the centralized logging server, you can take a look at my Octopussy project.

It's a lot of work at the begining, but after you can do a lot of things with these logs !

share|improve this answer
add comment

Here is a tutorial that I wrote that covers all of the aspects of centralized logging and analysis.

Link: http://crunchtools.com/centralizing-log-files/

share|improve this answer
    
I am also looking at log4sh for a project I have internally (eventually to be open sourced, but working now), called scriptlog, essentially you run it before commands that you care about the output of and it does some magical stuff like added a WARNING string or CRITICAL string, it also has a nagios plugin to monitor it. Will post when I get it out –  fatherlinux Sep 22 '10 at 19:07
add comment

For centralized logging, I would highly recommend LogZilla. We've been using it for over a year now and absolutely love it. The UI is extremely easy to learn and use and installation took me about an hour.

Even if you don't, you really should try to get away from script-based monitoring as that's exactly what you get...monitoring. What you should try to achieve is Management. Repairing problems on Top talkers, etc. will greatly reduce the amount of "fires" triggered by script-base monitoring. Here's a very good article on syslog management:

http://www.cisco.com/en/US/technologies/collateral/tk869/tk769/white_paper_c11-557812.html

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.