Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Is there a solution for a distributed grep?

here's the story: I have a bunch of web servers and want to query their application logs (I'm using tomcat, if it matters). I don't want to have to copy the files to a common storage, they are too big, network is espensive and storage is too expensive so I want to keep them on the same web servers. So even haddop+hive or similar solutions won't fly.
So what I'm looking for is perhaps a local agent for which I can send the grep request to all hosts and get the results back.
Anything like that exists?

  • No file copying over the network
  • No common storage
  • Can't install hadoop on the web servers... I just can't...
  • grepping text files

Thanks!

BTW, I asked exactly the same question on http://scale.metaoptimize.com/questions/87/distributed-grep but for some reason the server keeps serving me 500s when I try to view this question, so I'm reposting here.

share

locked by HopelessN00b Feb 20 at 22:15

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as off-topic by HopelessN00b Feb 20 at 22:15

This question appears to be off-topic. The users who voted to close gave this specific reason:

If this question can be reworded to fit the rules in the help center, please edit the question.

    
Is there any way to consolidate your logs via syslog or similar, instead of trying to duplicate your log files? –  mfinni Mar 31 '11 at 17:49

6 Answers 6

For simple command runs, such as greping for a specific string on the server's logs, use a parallel SSH client like pssh or dsh.

For more complicated tasks you might want to take a look at MCollective.

share

Might be fun to try this with gnu parallel, maybe something like this:

Put the servers in a file servers.txt. Then:

parallel --sshlogins servers.txt "grep foo logfile"

I haven't tried myself.

share

I use GNU Parallel for that kind of tasks.

echo "fgrep -Rl <MyQuery> /var/log/" | parallel --onall --slf servers.txt
share

Why not simply run something like a cron job on your webserver running something like

grep something /path/to/log | ssh -C LogAnalyser "tee >> log_from_host X"

If you want to be able to know when the transfert from all host is finished on LogAnalyser you can start by creating a lock file and delete after tee.

share
    
distribution isn't the goal, it's a constraint. The log files are already on the web servers, now I need to grep them. I can't afford to copy them all to a single host or NAS, so I have to keep them in place and now I want to query them... –  Ran Jul 19 '10 at 6:58
    
Can't you just grep locally ?? –  radius Jul 19 '10 at 7:01
    
Ok I guess I misunderstood... you want to fetch grep result of local file from X server and not to send file over the network to X server and get the result back... I will update my answer –  radius Jul 19 '10 at 7:05
    
if I use the grep and cron technique I need to ssh first to all machines and set up the cron, it's tedious. I want to run ad hoc greps on various files (at the same location on all servers) –  Ran Jul 19 '10 at 17:48

You can try MCollective http://puppetlabs.com/mcollective

The description for its website is:

The Marionette Collective AKA MCollective is a framework to build server orchestration or parallel job execution systems. Primarily we’ll use it as a means of programmatic execution of Systems Administration actions on clusters of servers. In this regard we operate in the same space as tools like Func, Fabric or Capistrano. We’ve attempted to think out of the box a bit designing this system by not relying on central inventories and tools like SSH, we’re not simply a fancy SSH “for loop.”

It's an high complex orchestration system, if you have only to execute some simple logs grep it may exceed your needs.

share

Perhaps something along the following bash loop?

export FQDNS="hostname.domain.tld another.domain.tld"
for host in $FQDNS
do
  ssh $host 'grep "andol was here" /var/log/syslog'
done
share
1  
This will only serially ssh into the remote hosts and run the grep command. You usually want to do that in parallel. –  joschi Jul 19 '10 at 6:33
    
True, so I guess it all depends on what number "a bunch" actually translates into. –  andol Jul 19 '10 at 6:36
    
And how /var/log/syslog comes here ? –  radius Jul 19 '10 at 6:45
    
"a bunch" ~= 50 to 100 –  Ran Jul 19 '10 at 6:59
1  
radius: /var/log/syslog is as much a placeholder as "andol was here". I hardly think this question is about what specific grep command to run at which log file? –  andol Jul 19 '10 at 7:01

Not the answer you're looking for? Browse other questions tagged or ask your own question.