How can I grep only lines from a huge (120GB) httpd error_log based on a time range, say:

from 2011-11-15 11:30 pm
to   2011-11-16 01:30 am


  • Why didn't you use logrotate? – quanta Nov 16 '11 at 4:24
  • it's already rotated. too bad one of the apps generates tons of errors within a day and needed to be extracted for examination ... – ohho Nov 16 '11 at 8:02

You'll probably have to do some drill down, I'd start by getting the date range:

grep -e "2011\-11\-[15-16] " error_log > filtered
grep -v -e "2011\-11\-15 [0-10]:" | grep -v -e "2011\-11\-15 11:[0-29]" > filtered
grep -v -e "2011\-11\-16 [2-23]:" | grep -v -e "2011\-11\-16 01:[31-59]" > filtered

cat filtered

The most efficient way I can think of but haven't done is to find the start and end bytes of your date range and get that; (which is apparently possible with grep) but I dont know how to get a range of bytes from a file - probably takes some awk skills

Edit: Since this was an interesting question - I did some more digging:

You can get the first byte offset by doing:

 # Get first byte offset, leftmost number is the offset...
grep -m 1 -b "2011-11-15 11:3" error_log
 # Get last byte offset
grep -m 1 -b "2011-11-16 01:3" error_log

 #(Subtract first number from last number to get byte length) Then do:

dd if=error_log of=filtered bs=c skip=<first number> count=<last_byte#-first_byte#>
| improve this answer | |

awk '$3>"11:30:00" && $3<"13:30:00"' log_file | less

where $3 is the 3rd column of your logfile which is the timestamp, you can use any number as per your logfile.

| improve this answer | |
  • 3
    You should really read the older answers including the one that was accepted as it is much more thorough than the one you have provided. – Brent Pabst Oct 25 '12 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.