Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I want to tail -f my logs. However, I want to filter out everything that has the words:

"ELB", "Pingdom", "Health"

share|improve this question
2  
With vanheusden.com/multitail you can color important things or filter out things you dont want to see, example config pantz.org/software/multitail/multitailconfig.html –  oluies May 18 '11 at 8:33
add comment

4 Answers

I don't know about using awk instead of grep, but this works for me:

tail -f file.log | grep -Ev '(ELB|Pingdom|Health)'

EDIT: As dmourati and Caleb pointed out, you could also use egrep instead of grep -E for convenience. On some systems this this will be an link to the same binary, in others a copy of it supplied by the grep package. Either way it lives as an alternative to the -E switch. However, according to the GNU grep man page:

[…]two variant programs egrep and fgrep are available. egrep is the same as grep -E. fgrep is the same as grep -F. Direct invocation as either egrep or fgrep is deprecated, but is provided to allow historical applications that rely on them to run unmodified.

Since they are synonymous commands, it comes down to preference unless you don't have egrep at all. However for forward compatibility it is recommended to use the grep -E syntax since the other method is officially deprecated.

share|improve this answer
    
Does using grep -E instead of egrep warrant a duplicate answer? –  Caleb May 18 '11 at 9:10
    
@Caleb I don't see why not. There's more than one way to skin a cat, and this site allows for the ability to list all –  DTest May 18 '11 at 11:40
1  
Yes, but they're merely symlinks to each other, not two different programs with some functional overlap. So wouldnt that be more of an 'addendum' (read: comment), not a full fledged answer? I've had downvotes for lesser offences... –  Marcin May 18 '11 at 12:00
    
@DTest: By rights dmourati actually beat me to the punch by a couple of seconds and although he didn't explain the reasoning he deserves some credit here. You were distinctly late to the party since we both had at least two upvotes before you came in. Changing the syntax from a symlinked binary to an argument is usually something you would use a comment for, not a separate answer. If you want to skin the cat differently use sed, awk, perl, multitail or ninja_foo. –  Caleb May 18 '11 at 18:59
1  
@DTest: I took the liberty of making a significant factual edit to your answer to add some original sources. It looks like -E is recommended for future usage, so I marked that, but removed your note about distros that don't have egrep. The distros lain mentioned do have egrep, it's just a separate binary instead of a symlink. –  Caleb May 18 '11 at 20:21
show 5 more comments

Try piping it to egrep with a pipe separated lists of words you want to filter out:

tail -f log_file | egrep -v 'ELB|Pingdom|Health'

Note that using parenthesis around the list of matches is optional. Since the | is treated as a logical OR operator by grep whether it occurs as part of a sub-group or not. '(ELB|Pingdom|Health)' would function exactly the same. For some, the syntax may be more obvious; I find it easier to type without since I can switch from a single match to a list of possible matches without going back to add the parenthesis.

For extra credit, it's worth mentioning that multitail does ninja foo when it comes to filtering output. For example you could filter for your words like this:

multitail -e ELB -e Pingdom -e Health -f log_file

You could also use it to color or otherwise highlight the output instead of just filtering it.

EDit: See DTests answer and the comments for the full explanation of how egrep is just a deprecated alternate way to fire off grep -E.

share|improve this answer
2  
shouldn't expression end with ('), not with (")? –  bbaja42 May 18 '11 at 6:27
    
Yes thanks that was a typo. For future referance since stack exchange sites function like wikis, that is the kind of thing you can just fix. –  Caleb May 18 '11 at 8:09
    
I thought the edit had to be more than 6 chars ? –  Sirex May 18 '11 at 8:25
    
If you don't have high rep, yes there is a 6 character minimum, but in this case the 1 character is super important. You can force the change through by adding an HTML comment to the body. The characters will count towards the limit and you can note why you are making the change. –  Caleb May 18 '11 at 9:09
add comment
tail -f /path/to/log | egrep -v 'ELB|Pingdom|Health'
share|improve this answer
add comment

Why do you want to log this information?

  • Is it strictly for archival?
  • Do you want to conditionally execute different scripts depending on different keywords or patterns in the log files?

If you want to have scripted behavior depending on the content of the log files, you may wish to do your filtering using Expect. ( http://en.wikipedia.org/wiki/Expect ) Expect is a Tcl extension but There is also a Python version of Expect.

Expect gives you this powerful flexible switch like statement that lets you specify different behaviors conditionally depending on the states, or patterns present in your input stream. For example:

expect {  
    "password:" {  
        send "password\r"  
    } 
    "yes/no)?" {  
        send "yes\r"  
        set timeout -1  
    }  
    timeout {  
        exit  
    }   
    -re . {  
        exp_continue  
    }  
    eof {  
        exit  
    }  
}

So you specify patterns in the expect statement, and you specify different behaviors, and you can wrap the whole thing in a loop, and you can easily write very powerful filters that also write portions of your input to different files, or drop it altogether, or take actions and run other scripts depending on what is in your input.

So, it comes down to why are you trying to filter your log files, to take action on log input, or just for archival reasons?

share|improve this answer
    
Plus one for the reference to Expect, which I used quite a long time ago and had completely forgotten about. –  MPi May 18 '11 at 8:35
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.