(I asked this question on stackoverflow but it might be better off here...)

I need to read through some gigantic log files on a Linux system. There's a lot of clutter in the logs. At the moment I'm doing something like this:

cat logfile.txt | grep -v "IgnoreThis\|IgnoreThat" | less

But it's cumbersome -- every time I want to add another filter, I need to quit less and edit the command line. Some of the filters are relatively complicated and may be multi-line.

I'd like some way to apply filters as I am reading through the log, and a way to save these filters somewhere.

Is there a tool that can do this for me? I can't install new software so hopefully it's something that would already be installed -- e.g., less, vi, something in a Python or Perl lib, etc.

Changing the code that generates the log to generate less is not an option.

  • Dan, does it have to be a tool that works at the command line? – William Feb 26 '10 at 1:45
  • @William: No, but that would be preferred. My biggest criteria is that it's something I already have installed -- of course you have no way of knowing that... I was hoping for some way vi could do this. Someone suggested multitail on SO, looks good but I don't have it installed. – Dan Feb 26 '10 at 1:49
  1. Reading through large log files is not a good idea
    • you need to first filter them to the necessary lines and then look at them
    • even then, if possible (and if the filtered lines are a lot too),
      you should run secondary filters and counting scripts to analyze them
  2. When you have a large static file and know the filters to eliminate lines from it,
    it is always a good idea to store the filtered output (rather than trying to look at it immediately).
    • This way you can run secondary filters on the filtered output and they will not need to run on the entire thing again
    • so, in your example, store the first output to a new file, and when you know another filter apply it on that stored file
    • of course, this implies some storage space for the filtered file.
      It works better if your filters will reduce the actual file by a large amount
  3. Regular linux tool like 'grep', 'sed', 'AWK' usually suffice to process text log file very nicely.
    I have processed log files in the order of 10GB frequently with these things.
    You can make your own tools with these things in 'bash scripts'.
  4. Don't underestimate 'vim', it can handle large files too (but will take time, so give it filtered files)

You certainly won't have this installed, but if this is going to be a regular thing you have to do it might be worth looking into Splunk. Splunk exists to index large datasets like this to help you find what you're looking for.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.