I've been working on a way to parse large multiple access logs (sometimes in double digit GB in size) for CMSs. Right now I just need WordPress and Joomla, but I know the types will grow over time. Currently I have two oneliners (these will eventually go into a BASH script so any scripting language can be used as long as BASH can interpret it):

echo -e "\n=== WordPress ===\n"; grep --no-messages wp-login.php /usr/local/apache/domlogs/* | grep POST | grep "$(date +"%d/%b/%Y")" | cut --fields=1 --delimiter=: | sort | uniq --count | sort --reverse --numeric-sort | head | sed 's/^ *//g'

echo -e "\n=== Joomla ===\n"; grep --no-messages '/administrator/index.php' /usr/local/apache/domlogs/* | grep POST | grep "$(date +"%d/%b/%Y")" | cut --fields=1 --delimiter=: | sort | uniq --count | sort --reverse --numeric-sort | head | sed 's/^ *//g'

I'm trying to find a way speed these up and combine them so I only need to make one pass and then put them into separate lists. So, for example, the output would look something like this:

=== WordPress ===

896 /usr/local/apache/domlogs/blogp.com
52 /usr/local/apache/domlogs/blogt.com
9 /usr/local/apache/domlogs/blog.com

=== Joomla ===

65778 /usr/local/apache/domlogs/bloge.com
478 /usr/local/apache/domlogs/blogq.com
83 /usr/local/apache/domlogs/blogh.com

I know there are faster parsing methods the just grep are out there, but my knowledge is very limited, so any help would be greatly appreciated.

  • This has all appearances of an X & Y problem. Could you clarify what your actual problem is, rather than how your trying to solve it? – HBruijn Dec 8 '14 at 13:23
  • The actual oneliners above work in their current form. What my question is how to merge them and make it more efficient. When running the first pass with the WordPress string, on a large number of access files, it takes a very long time to complete and then by adding another pass to search for the Joomla login it makes it go through that same long process again. So, I'm trying to find a way to combine them into one pass and then sort out the results (see results example above) it would save time and processing power. – tryingtofindtheway Dec 8 '14 at 13:56
  • My thought was more along the lines of conditional logging and completely avoiding the large scale parsing of log files. E.G. SetEnvIf Request_URI "wp-login.php$" wordpresslogin followed by for instance CustomLog "|/usr/bin/logger -p local1.info -t www_wordpresslogin" combined env=wordpresslogin. – HBruijn Dec 8 '14 at 14:43
  • I agree that using conditional logging would be a better option, however, in this case it wouldn't work since this would have to be done on thousands of existing servers. That's why I would need to stick with the standard Apache access file format. – tryingtofindtheway Dec 8 '14 at 15:02
  • The alternative is a scripting language that allows you to efficient if-then-elseif pattern matching on the Request field of you log file. Old-school awk and perl come to mind. - Or go the route of loading them in splunk or similar. – HBruijn Dec 8 '14 at 15:07

It depends a bit on how you configure Apache to format your access_log. If you're using the common "CLF" logformat:

LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog logs/access_log common

Then the 7th field contains the request URL. A quick bit of awk which is always present and fairly efficient at parsing text. (Linux almost always has the extended GNU awk as the default; on Solaris you'd better ensure gawk) That allows you to multiple things on a single pass:

awk '
   $7 ~ /wp-login.php/ {print $0 }
   $7 ~ /\/administrator\/index.php/ {print $0}
' access_log

will search the 7th field $7 with the regular expression ~ enclosed by / forward slashes. When a match is found the {action} enclosed by the braces is performed. A simple action is $print the whole line $0 that contains the match.

A slightly more complex action with updating counters in combination with an END block:

awk '
   $7 ~ /wp-login.php/ {++wp }
   $7 ~ /\/administrator\/index.php/ {++jl}
   END   { print wp , " WordPress Logins found.\n" , jl , " Joomla logins found.\n" }
' access_log

Note: Learn a proper scripting language. That is almost mandatory for a professional system administrator. The actual language is not that important, often a shop has an already established internal scripting standard and then you can have somebody mentor you.
A basic knowledge of bash sed and awk and the assorted commandline utilities in addition to grep will only get you so far. Supplement that by something like perl, python, ruby, even php or PowerShell in a Windows environment.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.