We process our apache logs for some Google Shopping and Adwords stats. There's a command line that
cuts up the lines from the log to grab the IPs with specific things on the query string like
gclid. Today the script stopped working and when looking at the logs, I have found entries like this:
22.214.171.124 - - [20/Jun/2016:15:03:47 +0100] "GET /some-section/ HTTP/1.1" 200 13379 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, email@example.com)" E-SearchBox&FORM=IENTTR&conversationid=" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko" 126.96.36.199 - - [20/Jun/2016:15:03:48 +0100] "GET /some/page.php?action=setvalues%5B%5D=41&_=1466431513315 HTTP/1.1" 200 20 "http://www.website.co.uk/the-section/?gclid=COC5q9TAAs0CFdgaGwodLREAAA" "Moz
As you can see here, the second line (
E-SearchBox...) isn't a proper entry. this is a system with two load balanced web servers and all logs are sent to the same mounted location, which would probably describe the order being wrong (i.e. that line not being related to the previous one).
The weird thing is that this has only started happening in the past month. Last month the same report worked perfectly, but this month there are errors because of the corrupt format of the logs. And, running the exact same report for previous months now shows errors too where it hasn't previously.
Has anyone seen this behaviour before? Here is an example of how logging is setup:
CustomLog "|/usr/sbin/rotatelogs -l /var/www/vhosts/ombrelle/logs/website.co.uk/%Y%m%d_web.log 86400" combined env=!image
This hasn't changed for a few years, but the issue only appeared this month. I'm trying to figure out what happened to the files. Any suggestions?