I've been working on a way to parse large multiple access logs (sometimes in double digit GB in size) for CMSs. Right now I just need WordPress and Joomla, but I know the types will grow over time. Currently I have two oneliners (these will eventually go into a BASH script so any scripting language can be used as long as BASH can interpret it):
echo -e "\n=== WordPress ===\n"; grep --no-messages wp-login.php /usr/local/apache/domlogs/* | grep POST | grep "$(date +"%d/%b/%Y")" | cut --fields=1 --delimiter=: | sort | uniq --count | sort --reverse --numeric-sort | head | sed 's/^ *//g' echo -e "\n=== Joomla ===\n"; grep --no-messages '/administrator/index.php' /usr/local/apache/domlogs/* | grep POST | grep "$(date +"%d/%b/%Y")" | cut --fields=1 --delimiter=: | sort | uniq --count | sort --reverse --numeric-sort | head | sed 's/^ *//g'
I'm trying to find a way speed these up and combine them so I only need to make one pass and then put them into separate lists. So, for example, the output would look something like this:
=== WordPress === 896 /usr/local/apache/domlogs/blogp.com 52 /usr/local/apache/domlogs/blogt.com 9 /usr/local/apache/domlogs/blog.com === Joomla === 65778 /usr/local/apache/domlogs/bloge.com 478 /usr/local/apache/domlogs/blogq.com 83 /usr/local/apache/domlogs/blogh.com
I know there are faster parsing methods the just grep are out there, but my knowledge is very limited, so any help would be greatly appreciated.