We process our apache logs for some Google Shopping and Adwords stats. There's a command line that cuts up the lines from the log to grab the IPs with specific things on the query string like gclid. Today the script stopped working and when looking at the logs, I have found entries like this: - - [20/Jun/2016:15:03:47 +0100] "GET /some-section/ HTTP/1.1" 200 13379
"-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)"
E-SearchBox&FORM=IENTTR&conversationid=" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0)
like Gecko" - - [20/Jun/2016:15:03:48 +0100] "GET
HTTP/1.1" 200 20 "http://www.website.co.uk/the-section/?gclid=COC5q9TAAs0CFdgaGwodLREAAA"

As you can see here, the second line (E-SearchBox...) isn't a proper entry. this is a system with two load balanced web servers and all logs are sent to the same mounted location, which would probably describe the order being wrong (i.e. that line not being related to the previous one).

The weird thing is that this has only started happening in the past month. Last month the same report worked perfectly, but this month there are errors because of the corrupt format of the logs. And, running the exact same report for previous months now shows errors too where it hasn't previously.

Has anyone seen this behaviour before? Here is an example of how logging is setup:

CustomLog "|/usr/sbin/rotatelogs -l /var/www/vhosts/ombrelle/logs/website.co.uk/%Y%m%d_web.log 86400" combined env=!image

This hasn't changed for a few years, but the issue only appeared this month. I'm trying to figure out what happened to the files. Any suggestions?

  • Did you try to change the log location of one of the two services and see if the behaviour changes? – hub Jul 1 '16 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.