unable to filter badly-formatted messages in syslog-ng

Question

I am contacting you regarding an issue I have with syslog-NG. some of our devices (mainly HP switches and SANs) are sending syslog messages that do not respect the syslog RFC 5424 it seems.

to give you a few examples :

if I sniff the network interface of the server, we can see these "wrong" messages like that (as you can see, after the PRI, we can see the PROGRAM, then timezone and fields separated with commas. in short, mixed fields, missing fields etc. not a standard syslog message) :

<190>raslogd: 2017/03/08-16:03:20, [SEC-1203], 53642, WWN 10:00:50:eb:1a:6c:21:38 | FID 128, INFO, cswc-mo8x-SAN01, Login information: Login successful via TELNET/SSH/RSH. IP Addr: 1.2.3.4

therefore, I am currently unable to filter these messages. I tried to define a regular filter and try to print fields such as MSGHDR, MSG, PRI etc.. but couldn't see anything.

the ONLY WAY of having this message filtered is by just defining the external interface, and a file as output, without any template.

e.g if I do a filter like this :

destination d_INCOMING_ALL   { file("/app/syslog-ng/logs/incoming_all.log"); };

log {
  source(s_EXTERNAL);
  destination(d_INCOMING_ALL);
};

I can see messages in the log file, but formatted, somehow (I suppose syslog-NG reformats them) :

[root@xxxxxxxxxxxx logs]# grep -i cswc incoming_all.log
Mar  9 09:44:20 cswc-mo8x-hpsan01 raslogd: 2017/03/09-08:34:50, [SEC-1203], 53647, WWN 10:00:50:eb:1a:6c:21:38 | FID 128, INFO, cswc-mo8x-SAN01, Login information: Login successful via TELNET/SSH/RSH. IP Addr: 1.2.3.4
[root@xm1p1034vmo logs]#

the problem is that I cannot filter these messages like that (we receive logs from more than 1000 devices) there, i need to filter messages coming from these devices ! and the only way I can do it is on the hostname (cswc-) or program name (raslogd)

so I tried to display the fields by adding a template to that file, example :

destination d_test { 
    file ("/app/syslog-ng/logs/test_olivier.log" 
    template("pri=${priority} date=${ISODATE} host=${HOST} program=${PROGRAM} message=${MSG}\n") 
    ); 
};

but nothing works, nothing is displayed. I tried all fields, MSG, MESSAGE, MSGHDR etc.. can't manage do display ANYTHING. the only working thing is the parsing without filters or templates.

naturally, if I tred all kind of filters, like these below, it does not work (as fields are mixed) :

filter f_is_SAN     {
    host("cswc.*" flags(ignore-case));
};

same for :

filter f_is_SAN     {
    match(".*cswc.*" flags(ignore-case));
};

any hints on how to create filters for these messages coming from these devices (on hostname or programname) ?

thanks regards,

Answer 1

Actually, syslog-ng is not reformatting them, it is adding a proper syslog header:

Mar 9 09:44:20 cswc-mo8x-hpsan01 raslogd: 2017/03/09-08:34:50, [SEC-1203], 53647, WWN 10:00:50:eb:1a:6c:21:38 | FID 128, INFO, cswc-mo8x-SAN01, Login information: Login successful via TELNET/SSH/RSH. IP Addr: 1.2.3.4

The following is added by syslog:

Mar 9 09:44:20 cswc-mo8x-hpsan01

The rest is the original message:

raslogd: 2017/03/09-08:34:50, [SEC-1203], 53647, WWN 10:00:50:eb:1a:6c:21:38 | FID 128, INFO, cswc-mo8x-SAN01, Login information: Login successful via TELNET/SSH/RSH. IP Addr: 1.2.3.4

So syslog-ng is agreeing with you that these messages are not RFC-compliant.

In terms of setting up filters, this means you do not have a lot to play with.

I am personally more used to rsyslog (where you could inspect the message with regex, e.g. looking for raslogd - although that is not necessarily efficient).

However, the header syslog-ng is able to add gives you a couple of things to go on:

• it seems syslog-ng is able to report a hostname, which I would tend to assume is resolved from the source IP. So you at very least can filter on source ip, using either host or netmask:

  filter hp_hosts { (host(192.168.0.25)) };

• your current host filter is failing because there is no HOST field in the messages - the documents for the host filter state:

  That is, syslog-ng OSE will compare the filter expression to the content of the ${HOST} macro.

• you do however have a hostname (at least in your example) within the message, so you could consider using a message filter.

That being said - to me, if it was possible to get a list of the source IPs of all these devices (and I can only hope for your sake that is possible), then I would use the netmask filter - it seems likely to be more efficient than the other options.

Answer 2

mmh I found a solution which seem to be working

I am reusing the "local copy" destination as a source

e.g :

I have a regular log filter keeping a copy of everything :

destination d_INCOMING_ALL       { file("/app/syslog-ng/logs/incoming_all.log"); };
log {
  source(s_EXTERNAL);
  destination(d_INCOMING_ALL); 
};

the message is kept in that "local copy" being reformatted. I can then re-read from this file in a source and apply the filters to it. it's dirty, but it works :

source s_NON_RFC_COMPLIANT {   # new source
    file("/app/syslog-ng/logs/incoming_all.log" follow-freq(10));
};

....

destination d_INCOMING_ALL {
    file("/app/syslog-ng/logs/incoming_all.log");    # old destination where we keep a local copy of EVERYTHING
};

....

log {      # local copy of everything
  source(s_EXTERNAL);
  destination(d_INCOMING_ALL);
};

....

other filters here

....

filter f_is_Network     {
  host("cswc.*" flags(ignore-case))       # we want to filter on hosts called cswc-something or program raslogd
  or program("raslogd" flags(ignore-case));
};

log {    # new source for HP stuff here
  source(s_NON_RFC_COMPLIANT);
  filter(f_is_Network);              # add the filters you need
  destination(<your_final_destination_here>);
  flags(final);
};