Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Has anyone ever used/created a simple unix/linux log parser that can parse logs like the following:

timestamp log_message \n

Order the messages, parse the timestamp, and return:

  • All messages
  • Messages after a certain date (--since)
  • Messages before a certain date (--until)
  • Combination of --since, --until

I could write something like this, but wasn't sure if there was something canned. It would fit well in some automated reporting I'm planning on doing.

share|improve this question
1  
Out of curiosity, if you write such a program, what language would it be in? –  Stefan Lasiewski Jun 15 '10 at 19:46
    
Hi Stefan -- If I write it, it will be Perl, but I was hoping something existed for linux in C (speed) that I could do apt-get install logparse, or similar. I'm certainly going to look at Dennis' script before I write my own. –  dpb Jun 16 '10 at 2:41

3 Answers 3

up vote 3 down vote accepted

Take a look at this Python program I wrote to see if it comes close to what you're looking for or can be adapted to your needs.

share|improve this answer
    
I will go look at this. From appearances, it looks like exactly what I need. I'll probably have to tweak a bit, but that isn't a problem. Thanks! –  dpb Jun 16 '10 at 2:38

Even if you find one I don't know if I would trust it. For example, since the timestamp isn't known ahead of time, the only what it could distinguish DD-MM-YYYY and MM-DD-YYYY would be to read ahead until either xx or zz with xx-zz-YYYY is greater than 12. I am sure there are other issues.

Writing your own would be much easy and more reliable I think as you can use your language's standard string to datetime library and specify the date format specifiers explicitly.

For example with Python:

    import re
    from datetime import datetime
    line_regex = re.compile(
        r'''
        \[
        (?P<day> \d{1,2} )
        /
        (?P<month> \w{3} )
        / 
        (?P<year> \d{4} )
        :
        (?P<hour> \d{2} )
        : 
        (?P<minute> \d{2} )
        : 
        (?P<second> \d{2} )
        \s
        (?P<timezone> -?\d{4} )
        \]
        ''', re.VERBOSE)


   new_entry['time'] = datetime.strptime(
        parsed_line['day'] +
        parsed_line['month'] +
        parsed_line['year'] +
        parsed_line['hour'] +
        parsed_line['minute'] +
        parsed_line['second']
        , "%d%b%Y%H%M%S"
   )

If you do want a module / library that will try to figure out the format they do exist, one option for Perl is Date::Parse.

share|improve this answer
    
I know the format up front, thanks for the tips. Also, I have used Date::Parse, it's a bit slow, but it is almost magic. :) –  dpb Jun 16 '10 at 2:36

The only solution I've ever found for this which was even halfway decent was syslog-ng logging to a database (at which point it's reduced to simple SQL queries) -- Assuming you're logging everything centrally this doesn't add too much additional pain.

(Insert obvious benefits & obvious caveats here)

share|improve this answer
    
Good suggestion, not an option in this environment. –  dpb Jun 16 '10 at 2:35

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.