1

We are getting syslog-ng feeds from a forwarder that we don't control.

The issue is that they are using an older version (RHEL 6), and while they have keep_hostnames set to 'yes' (chain_hostnames to 'no'), it's just not working. Their feed is still appending the relay's IP and timestamp to the message.

Other than create a global filter for this single input to rip off that leading IP and timestamp, I don't know any other way of doing it.

Anyone have any suggestions? I'm very surprised that I can't tell my syslog-ng to just ignore all appended relay chains.

  • I'd settle for being able to filter on the sender's port. All messages are inbound at 514, but the sender is utilizing localport. If I could filter by the sender's port, I would be golden. Would be easy to tag messages after that. – jasonmclose Feb 7 '17 at 19:31
  • Hi, try to play with the keep-hostname and chain-hostname options on the syslog-ng server, the documentation has a table about how they should work in a relay scenario: balabit.com/documents/syslog-ng-ose-latest-guides/en/… Also, check the values of the related macros on the server, like FULLHOST, – Robert Fekete Feb 8 '17 at 7:39
  • Thanks. I have it set correctly according to the table(s). The issue is really the upstream relay, as it keeps prepending its own IP and timestamp. I found a workaround, but it's not pretty. Will post below. – jasonmclose Feb 8 '17 at 13:25
  • Hi, on a second thought, you mention that the relay also appends a timestamp. Do you know why that happens? Is the relay configured to do so (using a template, for example), or is it possible that the message is not RFC-compliant, and syslog-ng cannot parse it properly (hence the IP and the timestamp)? – Robert Fekete Feb 8 '17 at 13:43
  • It's just prepending the extra source/hop like syslog-ng would do. spoof_source and keep_hostnames(yes) and chain_hostnames(no) should prevent this from happening, but I think that version has some bugs (older version of syslog-ng). – jasonmclose Feb 8 '17 at 17:28
1

My work around for this isn't sexy, but it does work. I'm all ears if someone else knows of a better way.

Here is what I did. So in this situation, I'm trying to filter out IIS logs (s_net is simply tcp/udp listening on port 514). So here is what is sitting in my iis.conf file, inside of conf.d

parser p_iis_pattern_db {
    db_parser(file("/etc/syslog-ng/patterndb.d/strip_header.xml"));
};

filter f_iis {
    match("iis", value(".classifier.class"));
};

rewrite r_iis {
    set("${real_sender}", value("HOST"));
    set("iis", value("PROGRAM"));
};

destination d_iis {
    file("/nfs/syslog/iis/$YEAR/$MONTH/$DAY/${real_sender}/$YEAR-$MONTH-$DAY-$HOUR-${real_sender}-$PROGRAM.log"
        template("${orig_sender}${msg}\n"));
};

log {
    source(s_net);
    parser(p_iis_pattern_db);
    filter(f_iis);
    rewrite(r_iis);
    destination(d_iis);
    flags(final);
};

Then, I have a pattern file in patterndb.d (strip_header.xml), which looks like this:

<patterndb version='3' pub_date='2017-02-07'>
  <ruleset name='strip_sender' id='strip_leading_sender'>
    <rules>
      <rule id='iis' class='iis'>
        <patterns>
          <pattern>@ESTRING:datestamp: @@IPv4:real_sender@ iis - - - @ANYSTRING:msg@</pattern>
        </patterns>
      </rule>
    </rules>
  </ruleset>
</patterndb>

I would prefer to be filtering based upon a MACRO value other than the message, as I think this is probably rather expensive; it's having to introspect the message of every package. Because data is coming from a relay running an older version of syslog-ng, I perform an outbound tag of data on that relay, and the keep-hostname/chain-hostname options aren't working.

My other worry is that this is going to get sloppy in a hurry as I start adding in non-relay sources. I might have to do the whole thing where I name the conf files in a certain way to assure they are performed in the correct order, in order to save CPU cycles (is it alphabetical?).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.