Extracting From Logs [closed]

Question

I want to grep following information from below raw logs:

2016-05-23 11:01:40 [1005583] 1b4ivg-004DZf-GX ** [email protected] F=<abbas@DomainName> P=<abbas@DomainName> R=dkim_lookuphost T=dkim_remote_smtp H=mx2.hotmail.com [65.54.188.72]:25 I=[IP Address]:56910 X=TLSv1.2:ECDHE-RSA-AES256-SHA384:256 CV=yes DN="/CN=*.hotmail.com": SMTP error from remote mail server after MAIL FROM:<abbas@DomainName> SIZE=275286: 421 RP-001 (BAY004-MC1F14) Unfortunately, messages from 16.23.21.111 weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.
2016-05-23 11:12:53 [1015989] 1b4j6h-004GIq-Ob ** [email protected] F=<corporate-kbl@DomainName> P=<corporate-kbl@DomainName> R=lookuphost T=remote_smtp H=mx3.hotmail.com [65.55.37.120]:25 I=[IP Address]:51605 X=TLSv1.2:ECDHE-RSA-AES256-SHA384:256 CV=yes DN="/CN=*.hotmail.com": SMTP error from remote mail server after MAIL FROM:<corporate-kbl@DomainName> SIZE=17484: 550 SC-001 (COL004-MC4F44) Unfortunately, messages from 16.23.21.111 weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.
2016-05-23 11:13:19 [1020551] 1b4j76-004HUH-Nr ** [email protected] (muhammad.yousuf@DomainName) <muhammad.yousuf@DomainName> F=<saeed.ahmed@DomainName> P=<saeed.ahmed@DomainName> R=dkim_lookuphost T=dkim_remote_smtp H=mx3.hotmail.com [134.170.2.199]:25 I=[IP Address]:55971 X=TLSv1.2:ECDHE-RSA-AES256-SHA384:256 CV=yes DN="/CN=*.hotmail.com": SMTP error from remote mail server after MAIL FROM:<saeed.ahmed@DomainName> SIZE=24006: 550 DY-001 (BLU004-MC1F21) Unfortunately, messages from 16.23.21.111 weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.

As i have following set of error codes, they may occur if they occur error field shows error:

421 RP-001
421 RP-002
421 RP-003
550 SC-001
550 SC-002
550 SC-003
550 SC-004
550 DY-001
550 DY-002
550 DY-001
550 OU-001
550 OU-002

As i have getting first three fields output from following command:

  echo "Timestamp            emailto:                  emailfrom:" && awk 'NF>6 { d=6 ; while ( ! ($d ~ /^F=/ ) ) d++ ; printf "%s\t%s\t%s\n",$1,$6,substr($d,4,length($d)-4) ;} ' logs | column -t

What i want to get:

  Timestamp:                    Email To:               Email From:            Messages From:       Error Codes:
 2016-05-23                [email protected]       abbas@DomainName          16.23.21.111         421 RP-001
 2016-05-23                [email protected]       corporate-kbl@DomainName    16.23.21.111         550 SC-001
 2016-05-23                [email protected]      saeed.ahmed@DomainName      16.23.21.111         550 DY-001  

Answer 1

You wouldn't use grep, you could use awk, but I prefer a nice regex with sed.

# <logs sed -nE 's,^([-0-9]{10})[^@]* ([^@]*@[^[:space:]]*)[^=]*F=<([^@]*@[^[:space:]]*)>.*SIZE=[^[:space:]]* (... ..-...) .*([[:digit:]]+\.[[:digit:]]+\.[[:digit:]]+\.[[:digit:]]+).*,\1 \2 \3 \5 \4,p'

Looks scary, but it captures the bits in parentheses as groups (\1 \2 etc), so the first is a date (10 digits or -), then it skips to the next @ sign ([^@] means anything not matching @), groups the email address, skips to the next =, back to the F, groups the F address, then skips ahead to SIZE, then grabs the error code (three of anything, space, two of anything, hyphen, three of anything, then skips to an IP address (exercise for the reader). The 'p' command makes sed print any substituted line.

It doesn't do much validation, for example 9.9.99.999 is a valid IP to it, but this is out of the scope of the task.

Does that help?

You can put tabs in instead of spaces in the last part for alignment.