Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I am using AWK to read through a custom log file I have. The format is something like this:

[12:08:00 +0000] 192.168.2.3 98374 "CONNECT 192.168.2.4:8091 HTTP/1.0" 200

Right now, I have AWK (from bash) set to read the whole log, analyze each line and grab each line that contains "CONNECT" which works, however, it does not help me discover unique clients.

The way to do this would be to somehow filter it so that it analyzed this part of each line: "CONNECT 192.168.2.4:8091 HTTP/1.0"

If there was a way to grab all those lines in a log file, then compare them all and only count similar lines as one. So let's say, for example:

 [12:08:00 +0000] 192.168.2.3 98374 "CONNECT 192.168.2.6:8091 HTTP/2.0" 200
 [12:08:00 +0000] 192.168.2.3 98374 "CONNECT 192.168.2.9:8091 HTTP/2.0" 200
 [12:08:00 +0000] 192.168.2.3 98374 "CONNECT 192.168.2.2:8091 HTTP/2.0" 200
 [12:08:00 +0000] 192.168.2.3 98374 "CONNECT 192.168.2.9:8091 HTTP/2.0" 200

In this case, the answer I need would be 3, not 4. Because 2 lines are the same, so there are only 3 unique lines. What I need is an automated way to accomplish this with AWK.

If anybody can lend a hand that would be great.

share|improve this question

3 Answers 3

You could let awk count unique instances like this:

awk -F\" '/CONNECT/ && !seen[$2] { seen[$2]++ } END { print length(seen) }' logfile

Output:

3

This collects the first double-quoted string from lines containing CONNECT in the seen hash array. When the end of input is reached, the number of elements in seen is printed.

share|improve this answer
    
+1 for a much more elegant answer than mine :) –  SmallClanger Mar 19 '13 at 15:54

sed -re 's/.*"([^"]*)".*/\1/' <logfile> |sort |uniq

Awk variant: awk -F'"' {print $2} <logfile> |sort |uniq

Add -c to uniq to get a count of each matching line, or |wc -l to get a count of the number of matching lines.

share|improve this answer

Running the log file through sort | uniq should filter out the duplicate lines, but i would question why you have those lines in there. Are they really duplicates?

If they are legitimate log entries and all you want is a unique list of clients (the 2nd field) for lines which are non-duplicates, then a simple modification of @Thor's script should get you what you want:

awk '
/CONNECT/ {
  if (seen[$0] == 0) {
    clients[$3]++
  }
  seen[$0]++ 
} 
END {
  for (i in clients) {
    print i
  }
}'

Which for the sample you've given results in:

192.168.2.3

This isn't as compact as Thor's script, but i usually find that as soon as i've written something like this, i want to do more with the lines themselves, so i've left the seen array (tracking the count of unique lines) in there.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.