8

I have nginx log file, and I want to find out market share for each major version of browsers. I am not interested in minor versions and operating systems. I would like to get something like this:

100 IE6
 99 IE7
 20 IE8
200 FF2
300 FF3

I know how to get the list of user agents from the file, but I want to aggregate the list to see only the major versions of the browsers. Is there a tool that does it?

Improve this question

7 Answers 7

Reset to default
25 
awk -F'"' '/GET/ {print $6}' /var/log/nginx-access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn
  • awk(1) - selecting full User-Agent string of GET requests
  • cut(1) - using first word from it
  • sort(1) - sorting
  • uniq(1) - count
  • sort(1) - sorting by count, reversed

PS. Of course it can be replaced by one awk/sed/perl/python/etc script. I just wanted to show how rich unix-way is.

Improve this answer
6
  • Thanks, this looks interesting. I will try it out immediately. :)
    – Željko Filipin
    Dec 2, 2009 at 11:14
  • I am trying to get your code to work. Could you please provide more information about each command? (I will do some research too.)
    – Željko Filipin
    Dec 3, 2009 at 10:54
  • you can use man grep, man awk etc.
    – SaveTheRbtz
    Dec 3, 2009 at 15:55
  • Thanks, will do that. I was not explicit enough. The only thing confusing me so far is {print $6}, but I guess it represents the piece of data in a line.
    – Željko Filipin
    Dec 4, 2009 at 9:16
  • 1
    Thanks a lot. I guess my log is not standard, but I just had to change $6 to $8 to get it working.
    – Željko Filipin
    Dec 31, 2009 at 14:35
7

While the one liner by SaveTheRbtz does the job, it took several hours to parse my nginx access log.

Here is a faster version based on his, which takes less than 1 minute per 100MB of log file (corresponding to about 1 million lines):

sed -n 's!.* "GET.* "\([[:alnum:].]\+/*[[:digit:].]*\)[^"]*"$!\1!p' /var/log/nginx/access.log | sort | uniq -c | sort -rfg

It works with the default access log format of nginx, which is the same as the combined format of Apache's httpd and has the User-Agent as the last field, delimited by ".

Improve this answer
4

To get user agent

sudo awk -F"\"" '{print $6}' /var/log/nginx/access.log | sort | uniq -dc
Improve this answer
3

This is a slight variation of the accepted answer, using fgrep and cut.

cat your_file.log | fgrep '"GET ' | cut -d'"' -f6 | cut -d' ' -f1 | sort | uniq -c | sort -rn

There is something appealing about using "weaker" commands when it is possible.

Improve this answer
2

Awstats should do the trick, but will supply far more information. I hope this helps...

Improve this answer
2
  • Thanks, Awstats looks similar to Webalizer. I will try it out.
    – Željko Filipin
    Dec 1, 2009 at 14:23
  • Awstats was to complicated to install.
    – Željko Filipin
    Dec 3, 2009 at 10:52
2

Webalizer can do it.

Example:

webalizer -o reports_folder -M 5 log_file
  • -o reports_folder specifies folder where report is generated
  • -M 5 displays only the browser name and the major version number
  • log_file specifies log file name
  • source: ftp://ftp.mrunix.net/pub/webalizer/README
Improve this answer
0
0

I'd use shell script for that: cat, awk pipe, sort and uniq will do the job

Improve this answer
1
  • Thanks. I know how to parse logs. I was looking for a tool that already knows how to do it, so I do not have to write a script.
    – Željko Filipin
    Dec 2, 2009 at 11:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .