Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am parsing my httpd access logs, to grab out which of our Google appliance's crawlers are bombarding my web servers, and when. If I type the command:

grep google /path/to/access_log | awk '{print $4, $14}'

I can get a very large (like I said, they're bombarding me) return set, at least 4 per second. I want to be able to group the above listed result set by the time stamp, and return a string inline with the number of hits there were per second. So ideally, I'd like to have something similar to

04/Aug/2011:15:56:16 Crawler1 6
04/Aug/2011:15:56:16 Crawler2 10
04/Aug/2011:15:56:17 Crawler1 8
04/Aug/2011:15:56:18 Crawler1 12

Where the first is the timestamp, second is the 14th field, the Google Crawler's ID, and third being the count. The order of the columns is irrelevant.

share|improve this question
up vote 0 down vote accepted
awk '/google/ { print substr($4, 2, length($4)-1), $14 }' access_log | sort -rn | uniq -c | awk '{ print $2"\t"$3"\t"$1 }'
share|improve this answer
Much appreciated! – Scott Aug 4 '11 at 16:48

This can be done in a single awk, counting hits in the same second using arrays, but it is hard to test without a sample input. Lets guess:

awk '/google/ {

  if (ts != lts) {
    for(c in count) {
      print lts " " c " " count[c]
      delete count[c]


  for(c in count) {
    print lts " " c " " count[c]
}' access_log
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.