Extracting values from csv-parser in syslog-ng

Question

I've got a syslog feed in csv format being ingested by syslog-ng using the csv-parser and written to disk.

As I understand it, using the csv-parser is supposed to give syslog-ng context into the data coming in and what value means what. I'd like to use that context to make syslog-ng apply filters based on that logic.

For example, I'd like it to only log an event to disk if one of the csv columns match a specific value.

The csv-parser documentation seems to suggest this should be possible.

My config looks a little like:

parser p_my_app {
    csv-parser(
        columns("MY_APP.COLOUR","MY_APP.SIZE","MY_APP.SERIAL_NUMBER")
        delimiters(",")
        flags(escape-double-char)
    );
};

source s_my_app {
    syslog(ip(0.0.0.0) port(6514)
        transport("tcp")
    );
}; 

filter f_my_app { 
    match("123456" value("MY_APP.SERIAL_NUMBER") );
};

destination d_my_app { 
    file("/var/log/my_app.log"
        create_dirs(yes)
    ); 
};

log { 
    source(s_my_app);
    filter(f_my_app);
    parser(p_my_app);
    destination(d_my_app); 
};

However, it seems no matter how much I try, the match never matches. Here's the sylog-ng debug output with that config:

Incoming log entry; line='<14>1 2017-01-18T17:46:38+11:00 hostname  - - - red,large,123456
'
Filter rule evaluation begins; rule='f_my_app', location='/etc/syslog-ng/conf.d/my-app.conf:16:18'
Filter node evaluation result; result='not-match'
Filter rule evaluation result; result='not-match', rule='f_my_app', location='/etc/syslog-ng/conf.d/my-app.conf:16:18'

I (hope?) I'm missing something really obvious, but I can't find any complete examples online after hours of Googling, just extracts which never seem to work. Is anyone able to see what I'm doing wrong and/or supply a complete working example?

Answer 1

Well, as seems to always be the way, I spend a week on a problem, give up, decide to seek help -- then figure out the solution myself an hour later.

My issue was the order of the items in the log() statement. Specifically, the parser() line MUST be before the filter() statement.

Indeed, the documentation (which I must have read 10 times and missed) states that:

Note The order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.

So, the working code is to use:

log { 
    source(s_my_app);
    parser(p_my_app);
    filter(f_my_app);
    destination(d_my_app); 
};

Also, one other thing to note is that match() actually is supposed to be a regex, so for added clarity, I've also updated the filter to:

filter f_my_app { 
    match("123456" value("MY_APP.SERIAL_NUMBER") type("string"));
};

With syslog-ng now reporting:

Incoming log entry; line='<14>1 2017-01-18T17:46:38+11:00 hostname  - - - red,large,123456
'
Message parsing complete; result='1', rule='p_my_app', location='/etc/syslog-ng/conf.d/my-app.conf:14:2'
Filter rule evaluation begins; rule='f_my_app', location='/etc/syslog-ng/conf.d/my-app.conf:16:18'
Filter node evaluation result; result='match'
Filter rule evaluation result; result='match', rule='f_my_app', location='/etc/syslog-ng/conf.d/my-app.conf:16:18'