Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a 3GB log file, I need to extract the past 48 hours without downloading the entire 3GB file. How can I split the file up into the past 48 hours. So I can only download that single file?

I have full SSH access and I'm able to install additional tools.

share|improve this question
1  
what access do you have on the server ? Does it have to be now() - 48hrs or wiin from midnight yesterday do ? –  Iain Sep 29 '11 at 14:52
    
I have full access, it needs to be from now() - 48hrs –  Tom Sep 29 '11 at 14:58
add comment

2 Answers

up vote 9 down vote accepted

Assuming you have shell access to the server with the log file, try

egrep '2[789]/Sep/2011' logfile.big > /tmp/logfile.small

That will go back to the beginning of the 27th, which is a little over 48 hours, but I would expect that to be much smaller than the whole file, and it's quick to do. Don't forget to gzip the resulting file before you transfer it, that will speed things up even more.

share|improve this answer
    
That is awesome, thanks! –  Tom Sep 29 '11 at 15:09
add comment

You can make use of the Apache log parser - Analog. It works quickly because written in C. Analog allows specifying the time ranges with the command line options +F and +T which logically stand for 'FROM' and 'TO'.

share|improve this answer
    
MatHatter's grep is slick, but +1 for the general case suggestion too. I would have confused Analog with webalizer –  iPaulo Sep 29 '11 at 15:52
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.