Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am parsing my httpd access logs, to grab out which of our Google appliance's crawlers are bombarding my web servers, and when. If I type the command:

grep google /path/to/access_log | awk '{print $4, $14}'

I can get a very large (like I said, they're bombarding me) return set, at least 4 per second. I want to be able to group the above listed result set by the time stamp, and return a string inline with the number of hits there were per second. So ideally, I'd like to have something similar to

04/Aug/2011:15:56:16 Crawler1 6
04/Aug/2011:15:56:16 Crawler2 10
04/Aug/2011:15:56:17 Crawler1 8
04/Aug/2011:15:56:18 Crawler1 12

Where the first is the timestamp, second is the 14th field, the Google Crawler's ID, and third being the count. The order of the columns is irrelevant.

share|improve this question
add comment

2 Answers

up vote 0 down vote accepted
awk '/google/ { print substr($4, 2, length($4)-1), $14 }' access_log | sort -rn | uniq -c | awk '{ print $2"\t"$3"\t"$1 }'
share|improve this answer
    
Much appreciated! –  Scott Aug 4 '11 at 16:48
add comment

This can be done in a single awk, counting hits in the same second using arrays, but it is hard to test without a sample input. Lets guess:

awk '/google/ {
  ts=$4
  crawler=$14

  if (ts != lts) {
    for(c in count) {
      print lts " " c " " count[c]
      delete count[c]
    }

    lts=ts
  }

  count[crawler]++
}END{
  for(c in count) {
    print lts " " c " " count[c]
  }
}' access_log
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.