Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am parsing my httpd access logs, to grab out which of our Google appliance's crawlers are bombarding my web servers, and when. If I type the command:

grep google /path/to/access_log | awk '{print $4, $14}'

I can get a very large (like I said, they're bombarding me) return set, at least 4 per second. I want to be able to group the above listed result set by the time stamp, and return a string inline with the number of hits there were per second. So ideally, I'd like to have something similar to

04/Aug/2011:15:56:16 Crawler1 6
04/Aug/2011:15:56:16 Crawler2 10
04/Aug/2011:15:56:17 Crawler1 8
04/Aug/2011:15:56:18 Crawler1 12

Where the first is the timestamp, second is the 14th field, the Google Crawler's ID, and third being the count. The order of the columns is irrelevant.

share|improve this question

2 Answers 2

up vote 0 down vote accepted
awk '/google/ { print substr($4, 2, length($4)-1), $14 }' access_log | sort -rn | uniq -c | awk '{ print $2"\t"$3"\t"$1 }'
share|improve this answer
    
Much appreciated! –  Scott Aug 4 '11 at 16:48

This can be done in a single awk, counting hits in the same second using arrays, but it is hard to test without a sample input. Lets guess:

awk '/google/ {
  ts=$4
  crawler=$14

  if (ts != lts) {
    for(c in count) {
      print lts " " c " " count[c]
      delete count[c]
    }

    lts=ts
  }

  count[crawler]++
}END{
  for(c in count) {
    print lts " " c " " count[c]
  }
}' access_log
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.