42

I want to tail -f my logs. However, I want to filter out everything that has the words:

"ELB", "Pingdom", "Health"

1

4 Answers 4

63

I don't know about using awk instead of grep, but this works for me:

tail -f file.log | grep -Ev '(ELB|Pingdom|Health)'

EDIT: As dmourati and Caleb pointed out, you could also use egrep instead of grep -E for convenience. On some systems this this will be an link to the same binary, in others a copy of it supplied by the grep package. Either way it lives as an alternative to the -E switch. However, according to the GNU grep man page:

[…]two variant programs egrep and fgrep are available. egrep is the same as grep -E. fgrep is the same as grep -F. Direct invocation as either egrep or fgrep is deprecated, but is provided to allow historical applications that rely on them to run unmodified.

Since they are synonymous commands, it comes down to preference unless you don't have egrep at all. However for forward compatibility it is recommended to use the grep -E syntax since the other method is officially deprecated.

10
  • Does using grep -E instead of egrep warrant a duplicate answer?
    – Caleb
    May 18, 2011 at 9:10
  • @Caleb I don't see why not. There's more than one way to skin a cat, and this site allows for the ability to list all May 18, 2011 at 11:40
  • 1
    Yes, but they're merely symlinks to each other, not two different programs with some functional overlap. So wouldnt that be more of an 'addendum' (read: comment), not a full fledged answer? I've had downvotes for lesser offences...
    – Marcin
    May 18, 2011 at 12:00
  • @DTest: By rights dmourati actually beat me to the punch by a couple of seconds and although he didn't explain the reasoning he deserves some credit here. You were distinctly late to the party since we both had at least two upvotes before you came in. Changing the syntax from a symlinked binary to an argument is usually something you would use a comment for, not a separate answer. If you want to skin the cat differently use sed, awk, perl, multitail or ninja_foo.
    – Caleb
    May 18, 2011 at 18:59
  • 1
    @DTest: I took the liberty of making a significant factual edit to your answer to add some original sources. It looks like -E is recommended for future usage, so I marked that, but removed your note about distros that don't have egrep. The distros lain mentioned do have egrep, it's just a separate binary instead of a symlink.
    – Caleb
    May 18, 2011 at 20:21
23

Try piping it to egrep with a pipe separated lists of words you want to filter out:

tail -f log_file | egrep -v 'ELB|Pingdom|Health'

Note that using parenthesis around the list of matches is optional. Since the | is treated as a logical OR operator by grep whether it occurs as part of a sub-group or not. '(ELB|Pingdom|Health)' would function exactly the same. For some, the syntax may be more obvious; I find it easier to type without since I can switch from a single match to a list of possible matches without going back to add the parenthesis.

For extra credit, it's worth mentioning that multitail does ninja foo when it comes to filtering output. For example you could filter for your words like this:

multitail -e ELB -e Pingdom -e Health -f log_file

You could also use it to color or otherwise highlight the output instead of just filtering it.

EDit: See DTests answer and the comments for the full explanation of how egrep is just a deprecated alternate way to fire off grep -E.

5
  • 2
    shouldn't expression end with ('), not with (")?
    – bbaja42
    May 18, 2011 at 6:27
  • Yes thanks that was a typo. For future referance since stack exchange sites function like wikis, that is the kind of thing you can just fix.
    – Caleb
    May 18, 2011 at 8:09
  • I thought the edit had to be more than 6 chars ?
    – Sirex
    May 18, 2011 at 8:25
  • If you don't have high rep, yes there is a 6 character minimum, but in this case the 1 character is super important. You can force the change through by adding an HTML comment to the body. The characters will count towards the limit and you can note why you are making the change.
    – Caleb
    May 18, 2011 at 9:09
  • @Caleb thanks so much for the multitail suggestion it is awesome!! Can't believe I went for so long in my life without it. Mar 29, 2017 at 23:22
6
tail -f /path/to/log | egrep -v 'ELB|Pingdom|Health'
3

Why do you want to log this information?

  • Is it strictly for archival?
  • Do you want to conditionally execute different scripts depending on different keywords or patterns in the log files?

If you want to have scripted behavior depending on the content of the log files, you may wish to do your filtering using Expect. ( http://en.wikipedia.org/wiki/Expect ) Expect is a Tcl extension but There is also a Python version of Expect.

Expect gives you this powerful flexible switch like statement that lets you specify different behaviors conditionally depending on the states, or patterns present in your input stream. For example:

expect {  
    "password:" {  
        send "password\r"  
    } 
    "yes/no)?" {  
        send "yes\r"  
        set timeout -1  
    }  
    timeout {  
        exit  
    }   
    -re . {  
        exp_continue  
    }  
    eof {  
        exit  
    }  
}

So you specify patterns in the expect statement, and you specify different behaviors, and you can wrap the whole thing in a loop, and you can easily write very powerful filters that also write portions of your input to different files, or drop it altogether, or take actions and run other scripts depending on what is in your input.

So, it comes down to why are you trying to filter your log files, to take action on log input, or just for archival reasons?

1
  • Plus one for the reference to Expect, which I used quite a long time ago and had completely forgotten about.
    – MPi
    May 18, 2011 at 8:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .