Linux: logwatch(8) is too noisy. How can I control the noise level?

Question

Our Linux systems run logwatch(8) utility by default. On a RedHat/CentOS/SL system, Logwatch is called by the /etc/cron.daily/ cronjob, which then sends a daily email with the results. These emails have a subject like:

Subject: Logwatch for $HOSTNAME

The problem is that by default these daily emails are too noisy and contain a lot of superfluous information (HTTP errors, daily disk usage, etc) which are already monitored by other services (Nagios, Cacti, central syslog, etc). For 100 systems, the email load is unbearable. People ignore the emails, which means that we may miss problems which are picked up by logwatch.

How can I reduce the amount of noise generated by logwatch, but still use logwatch to notify us of significant problems?

I'll post my own answer below, but I would like to see what others have done.

Note: I have a similar question regarding FreeBSD, at FreeBSD: periodic(8) is too noisy. How can I control the noise level?

Answer 1

Overall, the available documentation for Logwatch lacks adequate explanation and is often far too vague. I pieced together some useful examples, and have reduced the Logwatch noise by over 95%.

Here's what I have found.

Keep in mind that you can find some Logwatch documentation at /usr/share/doc/logwatch-*/HOWTO-Customize-LogWatch, and it contains a few useful examples.

1. On RHEL/CentOS/SL, the default logwatch configuration is under /usr/share/logwatch/default.conf/logwatch.conf

   These settings can be overriden by placing your local configuration under /etc/logwatch/conf/logwatch.conf. Place the following in that file to tell logwatch to completely ignore services like 'httpd' and the daily disk usage checks:

   # Don't spam about the following Services
   Service = "-http"
   Service = "-zz-disk_space"
   

2. Sometimes I don't want to completely disable logwatch for a specific service, I just want to fine tune the results to make them less noisy. /usr/share/logwatch/default.conf/services/*.conf contains the default configuration for the services. These parameters can be overridden by placing your local configuration under /etc/logwatch/conf/services/$SERVICE.conf. Unfortunately, logwatch's ability here is limited, and many of the logwatch executables are full of undocumented Perl. Your choice is to replace the executable with something else, or try to override some settings using /etc/logwatch/conf/services.

   For example, I have a security scanner which runs scans across the network. As the tests run, the security scanner generates many error messages in the application logs. I would like logwatch to ignore errors from my security scanners, but still notify me of attacks from other hosts. This is covered in more detail at Logwatch: Ignore certain IPs for SSH & PAM checks?. To do this, I place the following under /etc/logwatch/conf/services/sshd.conf:

   # Ignore these hosts
   *Remove = 192.168.100.1
   *Remove = X.Y.123.123
   # Ignore these usernames
   *Remove = testuser
   # Ignore other noise. Note that we need to escape the ()
   *Remove = "pam_succeed_if\(sshd:auth\): error retrieving information about user netscan.*
   

   "

3. logwatch also allows you to strip out output from the logwatch emails by placing regular expressions in /etc/logwatch/conf/ignore.conf. HOWTO-Customize-LogWatch says:

   ignore.conf: This file specifies regular expressions that, when matched by the output of logwatch, will suppress the matching line, regardless of which service is being executed.

   However, I haven't had much luck with this. My requirements need a conditional statement, which is something like 'If there are security warnings due to my security scanner, then don't print the output. But if there are security warnings from my security scanner and from some bad guys, then print the useful parts-- The header which says "Failed logins from:", the IPs of the bad hosts, but not the IPs of scanners.'

4. Nip it at the source (As suggested by @user48838). These messages are being generated by some application, and then Logwatch is happily spewing the results to you. In these cases, you can modify the application to log less.

   This isn't always desirable, because sometimes you want the full logs to be sent somewhere (to a Central syslog server, central IDS server, Splunk, Nagios, etc.), but you don't want logwatch to email you about this from every server, every day.

Answer 2

Yes - logwatch is often too noisy. You already mentioned disabling checks completely.

If you don`t want to do that you have to prevent certain events to appear. For instance - it is not interesting if nagios connects via ssh to a DMZ system. But is is interesting if there are other logins via ssh.

We use rsyslog instead of ksyslogd (first install rsyslog, then remove ksyslogd). With rsyslog you can fine-tune what goes to the logs and what not (e.g. build a expression that drops messages from sshd cointaining "nagios connected"). That way logwatch will report useful information only.

Another case might be xinetd - I don`t want to get informed about successful connects - this can be configured in xinetd itselv - without disabling the logwatch-check for xinetd.

Answer 3

As a point of interest I followed option 2 form the answer by Stefan Lasiewski but for my purposes I wanted to only include specific lines rather than exclude all the noise I didn't want.

I was configuring vsftpd so I created /etc/logwatch/conf/services/vsftpd.conf and instead of using something like *Remove = testuser which removes rows which include the text testuser I used the line *OnlyContains = "testuser" which only returns rows including that text.

These 2 scripts work very basically by using grep and grep -v.

The difference being that you can use *Remove as many times as you want but with *OnlyContains you have to use it once if you want something or something else or something else. So for multiple values you do *OnlyContains = "testuser|testuser2|testuser3"

Answer 4

Have you considered funneling the email status messages to a listserver, maybe one that can provide digests based on programmable attributes of size and/or duration/age? The approach does not reduce the amount of logging being emailed, but can control the amount of individual emails by batching to reduce the emailing frequency.

Answer 5

I recently needed to quiet down the output from the sshd service. Some of the sections were quite long and could not be controlled by setting the detail level.

It's not an ideal solution, but I wound up overriding the sshd script by copying it from here: /usr/share/logwatch/scripts/services/sshd to here: /etc/logwatch/scripts/services/sshd

Of course, you now have to keep up on any updates to that script file, but it does give you very fine control over what is output. Alternatively, I suppose you could pipe the output of logwatch through a tool like awk or sed to strip out what you don't want, but that seemed harder to me.

Answer 6

I had the same question on UNIX&Linux Stackexchange and here is the answer I that fixed it for me:

You can tell logwatch to look at 7 days instead of 1 day by changing the Range parameter in your logwatch.conf:

Range = between -7 days and -1 days

You can tell logwatch to run weekly instead of daily by moving it from the weekly cron directory to the daily cron directory:

mv /etc/cron.daily/00logwatch /etc/cron.weekly/

Thanks to @JeffSchaller