6

I have nginx log file, and I want to find out market share for each major version of browsers. I am not interested in minor versions and operating systems. I would like to get something like this:

100 IE6
 99 IE7
 20 IE8
200 FF2
300 FF3

I know how to get the list of user agents from the file, but I want to aggregate the list to see only the major versions of the browsers. Is there a tool that does it?

19
awk -F'"' '/GET/ {print $6}' /var/log/nginx-access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn
  • awk(1) - selecting full User-Agent string of GET requests
  • cut(1) - using first word from it
  • sort(1) - sorting
  • uniq(1) - count
  • sort(1) - sorting by count, reversed

PS. Of course it can be replaced by one awk/sed/perl/python/etc script. I just wanted to show how rich unix-way is.

  • Thanks, this looks interesting. I will try it out immediately. :) – Željko Filipin Dec 2 '09 at 11:14
  • I am trying to get your code to work. Could you please provide more information about each command? (I will do some research too.) – Željko Filipin Dec 3 '09 at 10:54
  • you can use man grep, man awk etc. – SaveTheRbtz Dec 3 '09 at 15:55
  • Thanks, will do that. I was not explicit enough. The only thing confusing me so far is {print $6}, but I guess it represents the piece of data in a line. – Željko Filipin Dec 4 '09 at 9:16
  • 1
    Thanks a lot. I guess my log is not standard, but I just had to change $6 to $8 to get it working. – Željko Filipin Dec 31 '09 at 14:35
6

While the one liner by SaveTheRbtz does the job, it took several hours to parse my nginx access log.

Here is a faster version based on his, which takes less than 1 minute per 100MB of log file (corresponding to about 1 million lines):

sed -n 's!.* "GET.* "\([[:alnum:].]\+/*[[:digit:].]*\)[^"]*"$!\1!p' /var/log/nginx/access.log | sort | uniq -c | sort -rfg

It works with the default access log format of nginx, which is the same as the combined format of Apache's httpd and has the User-Agent as the last field, delimited by ".

3

This is a slight variation of the accepted answer, using fgrep and cut.

cat your_file.log | fgrep '"GET ' | cut -d'"' -f6 | cut -d' ' -f1 | sort | uniq -c | sort -rn

There is something appealing about using "weaker" commands when it is possible.

2

Awstats should do the trick, but will supply far more information. I hope this helps...

  • Thanks, Awstats looks similar to Webalizer. I will try it out. – Željko Filipin Dec 1 '09 at 14:23
  • Awstats was to complicated to install. – Željko Filipin Dec 3 '09 at 10:52
2

Webalizer can do it.

Example:

webalizer -o reports_folder -M 5 log_file
  • -o reports_folder specifies folder where report is generated
  • -M 5 displays only the browser name and the major version number
  • log_file specifies log file name
  • source: ftp://ftp.mrunix.net/pub/webalizer/README
0

I'd use shell script for that: cat, awk pipe, sort and uniq will do the job

  • Thanks. I know how to parse logs. I was looking for a tool that already knows how to do it, so I do not have to write a script. – Željko Filipin Dec 2 '09 at 11:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.