How can I use grep to find the most frequently recurring errors in my log file?

Question

I've got a PHP application that is generating large amounts of warnings and notices in the log file..

I'm fixing these one at a time.

I would like to be able to fix the most frequently occurring first in order to maximize the amount of effect my changes have on the size of the log file.

How can I use grep/sed/regex etc.find out which notices are most frequent?

I can't compare the whole line since the time stamp is at the beginning, but perhaps a listing/grouping/sorting of "on line ..."?

Answer 1

How about this?

grep 'on line' /var/log/httpd/error_log | cut -d' ' -f6- | sort | uniq -c | sort -nr

I.e.

1. Get the right lines from the logfile
2. Chop the date off the front
3. Sort them
4. Aggregate and count the duplicates
5. Sort by number of duplicates

Answer 2

I think this is more concise, so you really could avoid 'cut'.

  grep -o 'on line [[:digit:]]*' /var/log/nginx/error.log | sort | uniq -c | sort -nr