Distributed grep [closed]

Question

Is there a solution for a distributed grep?

here's the story: I have a bunch of web servers and want to query their application logs (I'm using tomcat, if it matters). I don't want to have to copy the files to a common storage, they are too big, network is espensive and storage is too expensive so I want to keep them on the same web servers. So even haddop+hive or similar solutions won't fly.
So what I'm looking for is perhaps a local agent for which I can send the grep request to all hosts and get the results back.
Anything like that exists?

• No file copying over the network
• No common storage
• Can't install hadoop on the web servers... I just can't...
• grepping text files

Thanks!

BTW, I asked exactly the same question on http://scale.metaoptimize.com/questions/87/distributed-grep but for some reason the server keeps serving me 500s when I try to view this question, so I'm reposting here.

Answer 1

For simple command runs, such as greping for a specific string on the server's logs, use a parallel SSH client like pssh or dsh.

For more complicated tasks you might want to take a look at MCollective.

Answer 2

Might be fun to try this with gnu parallel, maybe something like this:

Put the servers in a file servers.txt. Then:

parallel --sshlogins servers.txt "grep foo logfile"

I haven't tried myself.

Answer 3

I use GNU Parallel for that kind of tasks.

echo "fgrep -Rl <MyQuery> /var/log/" | parallel --onall --slf servers.txt

Answer 4

Why not simply run something like a cron job on your webserver running something like

grep something /path/to/log | ssh -C LogAnalyser "tee >> log_from_host X"

If you want to be able to know when the transfert from all host is finished on LogAnalyser you can start by creating a lock file and delete after tee.

Answer 5

You can try MCollective http://puppetlabs.com/mcollective

The description for its website is:

The Marionette Collective AKA MCollective is a framework to build server orchestration or parallel job execution systems. Primarily we’ll use it as a means of programmatic execution of Systems Administration actions on clusters of servers. In this regard we operate in the same space as tools like Func, Fabric or Capistrano. We’ve attempted to think out of the box a bit designing this system by not relying on central inventories and tools like SSH, we’re not simply a fancy SSH “for loop.”

It's an high complex orchestration system, if you have only to execute some simple logs grep it may exceed your needs.

Answer 6

Perhaps something along the following bash loop?

export FQDNS="hostname.domain.tld another.domain.tld"
for host in $FQDNS
do
  ssh $host 'grep "andol was here" /var/log/syslog'
done