1

I've been googling around the last few days looking for a solid example of how to regex a log entry for desired data, which is then to be inserted into a database, but apparently my google-fu is lacking.

What I am trying to do is track when an email is sent, and then track the remote mta response, specifically the dsn code. At this point I have two templates setup for each situation:

# /etc/rsyslog.conf
...
$Template tpl_custom_header, "MPurcell: CUSTOM HEADER Template: %msg%\n"
$Template tpl_response_dsn, "MPurcell: RESPONSE DSN Template: %msg%\n"

# /etc/rsyslog.d/mail
if $programname == 'mail-myapp' then /var/log/mail/myapp.log
if ($programname == 'mail-myapp') and ($msg contains 'X-custom_header') then /var/log/mail/test.log;tpl_custom_header
if ($programname == 'mail-myapp') and ($msg contains 'dsn=') then /var/log/mail/test.log;tpl_response_dsn
& ~

Example log entries:

MPurcell: CUSTOM HEADER Template:  D921940A1A: prepend: header X-custom_header: 101 from localhost[127.0.0.1]; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<localhost>: headername: message-id

MPurcell: RESPONSE DSN Template:  D921940A1A: to=<[email protected]>, relay=gmail-smtp-in.l.google.com[2607:f8b0:400e:c02::1a]:25, delay=2, delays=0.12/0.01/0.82/1.1, dsn=2.0.0, status=sent (250 2.0.0 OK 1372378600 o4si2828280pac.279 - gsmtp)

From the CUSTOM HEADER Template I would like to extract: D921940A1A, and X-custom_header value; 101

From the RESPONSE DSN Template I would like to extract: D921940A1A, and "dsn=2.0.0"

1 Answer 1

0

In case anyone ends up facing the same situation, here's what I ended up doing:

# /etc/rsyslog.conf

# Not sure what R signifies but saw it in other examples
# ERE = extended regex
# 0 = The submatch we want
# DFLT = How should a non match be returned?
$Template tpl_custom_header, "%msg:R,ERE,0,DFLT:[^:]+--end% | %msg:R,ERE,2,DFLT:X-custom_header:( )([0-9]*)--end%\n"    
$Template tpl_response_dsn, "%msg:R,ERE,0,DFLT:[^:]+--end% | %msg:R,ERE,1,DFLT:dsn=([0-9][.][0-9][.][0-9])--end% \n"

To test your regexes, you should use: http://www.rsyslog.com/regex/, it's a bit hokey but gets the job done.

Example raw log entries, slightly changed from OP:

Jun 29 05:40:28 service1 mail-myapp/cleanup[22200]: 6F67240A1A: prepend: header X-custom_header: 136 from localhost[127.0.0.1]; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<localhost>: headername: message-id

Jun 29 05:40:30 service1 mail-myapp/smtp[22201]: 6F67240A1A: to=<[email protected]>, relay=gmail-smtp-in.l.google.com[2607:f8b0:400e:c01::1a]:25, delay=2, delays=0.09/0/0.82/1, dsn=2.0.0, status=sent (250 2.0.0 OK 1372485254 rs6si5760686pbc.32 - gsmtp)

What they look like after templates were applied:

 6F67240A1A | 136
 6F67240A1A | 2.0.0

When I insert into mysql, I am going to insert dsn as int vs string for better performance, so thinking of using this:

 insert into response_log_dsn set mail_id = '6F67240A1A', dsn = (select cast(replace('2.0.0', '.', '') as unsigned));
1
  • R in "%msg:R,...." signifies that Regex will be used, afterwards specified more precisely with ERE (extended regex) or BRE (basic regex). Otherwise fields could be used which would be signified with an F instead of an R.
    – cstoll
    Feb 21, 2020 at 14:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .