Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Can anyone recommend a good tool to parse and analyze Nginx access logs which will group the urls based on the regex in my django files (or a config file generated from them)? It does not need to be real-time.

One of my primary concerns is looking at "request time" for various pages for which the url may contain slugs or uuids and may contain complex querystrings.
For example:

My core concern is that I be able to view aggregate statistics for all event detail pages. As an added bonus I would like to be able to see all event detail pages where "somethingelse" is in the querystring.

Other considerations: lightweight, open source, no database tables added to the django project if possible.

share|improve this question

closed as off-topic by dawud, Falcon Momot, Ward, mdpc, Dave M Oct 5 '13 at 15:47

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking product, service, or learning material recommendations are off-topic because they tend to become obsolete quickly. Instead, describe your situation and the specific problem you're trying to solve." – dawud, Falcon Momot, Ward, mdpc, Dave M
If this question can be reworded to fit the rules in the help center, please edit the question.

You can use bash to list all event_uuid from access log entry that contains word somethingelse:

grep '&somethingelse' /var/log/nginx/access.log | awk -F/ '{print $4}' 

Or just install the popular awstat to do general access log analysis and it gives you more than you expect.

share|improve this answer
I looked at awstats, and did't see in the docs how to define dynamic urls. In my example above the part of the url "my_event_uuid" is dynamic. Also note that the dynamic part of the url may not be the last part, but will be defined using regex. If that is possible with awstats can you please explain the configuration? – AgDude Oct 4 '13 at 14:01
The simplest way is providing awstat the log entries that contains those dynamic URLs only. Personally it would be easier to filter then extract what you want from access log entries than configuring any other log analysis tool. – shawnzhu Oct 4 '13 at 14:07
Thanks for your suggestion. However, that isn't practical for monitoring a production server with hundreds of urls. – AgDude Oct 4 '13 at 14:12
a cron job would be good enough for scale like hundreds of urls. Notice this is async operation and you didn't expect 'real-time' feedback. Or I would suggest you use a custom Django middleware to filter URIs instead of reading low level access log file. – shawnzhu Oct 4 '13 at 14:20

Not the answer you're looking for? Browse other questions tagged or ask your own question.