I have a bunch of Apache log files that I would like to analyze. I'm looking for a tool that doesn't require much setup; something that I can run a log through the command line, without messing around on our live web servers.
Any recommendations?
5
-
And the OS you're going to be using is...– IzzyMay 6, 2010 at 19:53
-
Linux or MacOSX...– mmattaxMay 6, 2010 at 20:07
-
1"Analyze" doesn't say much. What are you trying to do? Look for a string, crunch numbers?– DavidMay 6, 2010 at 20:37
-
1goaccess.prosoftcorp.com free and awesome, found it here: webmasters.stackexchange.com/questions/4852/…– jitbitDec 12, 2017 at 23:50
-
@jitbit cool tool but the ncurses UI is very hard to use– ychaoucheDec 15, 2020 at 8:50
Add a comment
|
7 Answers
While the tools above are all cool I think I know what the questioner was asking. It often pains me that I can't pull the information out of an access-log in the way I can with other files.
It's because of the dumb access log format:
127.0.0.1 - - [16/Aug/2014:20:47:29 +0100] "GET /manual/elisp/index.html HTTP/1.1" 200 37230 "http://testlocalhost/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0"
Why did they use [] for the date and "" for other things? did they think we wouldn't know a date was in field 4? It's incredibly frustrating.
The best tool right now for this is gawk:
gawk 'BEGIN { FPAT="([^ ]+)|(\"[^\"]+\")|(\\[[^\\]]+\\])" } { print $5 }'
on the data above this would give you:
"GET /manual/elisp/index.html HTTP/1.1"
In other words, the FPAT gives you the ability to pull out the fields of the apache-log as if they were actual fields instead of just space separated entities. This is always what I want. I can then parse that a bit more with a pipeline.
Making the FSPAT work is defined here: http://www.gnu.org/software/gawk/manual/html_node/Splitting-By-Content.html
You can therefore set up an alias to make a gawk that can parse apache logs:
alias apacheawk="gawk -vFPAT='([^ ]+)|(\"[^\"]+\")|(\\\\[[^\\\\]]+\\\\])' "
apacheawk '$6 ~ /200/ { print $5 } | sort | uniq
made this for me:
"GET / HTTP/1.1"
"GET /manual/elisp/index.html HTTP/1.1"
"GET /manual/elisp/Index.html HTTP/1.1"
"GET /scripts/app.js HTTP/1.1"
"GET /style.css HTTP/1.1"
and of course almost anything else is now possible.
Enjoy!
-
12 remarks: The date is not really in field 4 but in fields 4 + 5 ^^ (without the shift from GMT, the date has little value). And an access_log has most of the time 12 fields form (actually, there could be more than 12 fields, as the 12th is the http agent, which can contain many spaces in its name too.. the first 11 fields are easy to parse, and the 12th field (and maybe more) remaining should be the http agent). So you can just:
awk '($9 == 200) {print $6,$7,$8}'to display the same thing as in your example. No need to use FPAT there (even though this method can be usefull in other cases) Feb 10, 2015 at 13:50 -
I think you're over criticizing. The date is in field 4 if you consider the field to be bounded by []. Most of the time a log file is in one time zone, so the zone isn't necessary. The point of showing the example was not to show that something was exclusively possible this way, but to show the general trick. Feb 10, 2015 at 18:21
-
1I'm very surprised... I didn't "criticize" at all, just pointed 2 remarks (and said that indeed the method you used can be usefull in other cases, but here is just not needed)... Feb 12, 2015 at 6:38
wtop is cool. There's other utilities as well. Often, I'll parse logs using bash, sed, and awk.
-
wtop, and specially their log analyzer logrep are great, once you adapt the .conf to your log format it will provide a fast way to get what you need (top url, traffic, etc..)– asequesAug 26, 2013 at 11:05
apachetop is pretty cool; it prints live statistics. You run it with
apachetop -f /var/log/apache2/www.mysite.com.access.log
To install it in Debian/Ubuntu:
apt-get install apachetop
or from source: https://github.com/JeremyJones/Apachetop
What sort of output do you want?
If you are you just looking to count things then grep something logfile.txt | wc -l works great. If you want pretty graphs... not so much.
Instead of using a command line tool I would suggeset to try Apache Logs Viewer. It's a free tool which can monitor and analyze the Apache Log File. It can generate some pretty cool graphs and reports on the fly.
More info from http://www.apacheviewer.com
if you have a windows workstation that you can use then logparser is the tool of choice!
analog works well out of the box and doesn't require a lot of setup. logwrangler is a package that works with analog to generate nicer output and also requires little setup.
