I am parsing my httpd access logs, to grab out which of our Google appliance's crawlers are bombarding my web servers, and when. If I type the command:

grep google /path/to/access_log | awk '{print $4, $14}'

I can get a very large (like I said, they're bombarding me) return set, at least 4 per second. I want to be able to group the above listed result set by the time stamp, and return a string inline with the number of hits there were per second. So ideally, I'd like to have something similar to

04/Aug/2011:15:56:16 Crawler1 6
04/Aug/2011:15:56:16 Crawler2 10
04/Aug/2011:15:56:17 Crawler1 8
04/Aug/2011:15:56:18 Crawler1 12

Where the first is the timestamp, second is the 14th field, the Google Crawler's ID, and third being the count. The order of the columns is irrelevant.

awk '/google/ { print substr($4, 2, length($4)-1), $14 }' access_log | sort -rn | uniq -c | awk '{ print $2"\t"$3"\t"$1 }'
  • Much appreciated! – Scott Aug 4 '11 at 16:48

This can be done in a single awk, counting hits in the same second using arrays, but it is hard to test without a sample input. Lets guess:

awk '/google/ {

  if (ts != lts) {
    for(c in count) {
      print lts " " c " " count[c]
      delete count[c]


  for(c in count) {
    print lts " " c " " count[c]
}' access_log

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.