up vote 2 down vote favorite
1
share [g+] share [fb]

Is there a solution for a distributed grep?

here's the story: I have a bunch of web servers and want to query their application logs (I'm using tomcat, if it matters). I don't want to have to copy the files to a common storage, they are too big, network is espensive and storage is too expensive so I want to keep them on the same web servers. So even haddop+hive or similar solutions won't fly.
So what I'm looking for is perhaps a local agent for which I can send the grep request to all hosts and get the results back.
Anything like that exists?

  • No file copying over the network
  • No common storage
  • Can't install hadoop on the web servers... I just can't...
  • grepping text files

Thanks!

BTW, I asked exactly the same question on http://scale.metaoptimize.com/questions/87/distributed-grep but for some reason the server keeps serving me 500s when I try to view this question, so I'm reposting here.

link|improve this question
Is there any way to consolidate your logs via syslog or similar, instead of trying to duplicate your log files? – mfinni Mar 31 '11 at 17:49
feedback

4 Answers

up vote 3 down vote

For simple command runs, such as greping for a specific string on the server's logs, use a parallel SSH client like pssh or dsh.

For more complicated tasks you might want to take a look at MCollective.

link|improve this answer
feedback
up vote 1 down vote

Might be fun to try this with gnu parallel, maybe something like this:

Put the servers in a file servers.txt. Then:

parallel --sshlogins servers.txt "grep foo logfile"

I haven't tried myself.

link|improve this answer
feedback
up vote 0 down vote

Why not simply run something like a cron job on your webserver running something like

grep something /path/to/log | ssh -C LogAnalyser "tee >> log_from_host X"

If you want to be able to know when the transfert from all host is finished on LogAnalyser you can start by creating a lock file and delete after tee.

link|improve this answer
distribution isn't the goal, it's a constraint. The log files are already on the web servers, now I need to grep them. I can't afford to copy them all to a single host or NAS, so I have to keep them in place and now I want to query them... – Ran Jul 19 '10 at 6:58
Can't you just grep locally ?? – radius Jul 19 '10 at 7:01
Ok I guess I misunderstood... you want to fetch grep result of local file from X server and not to send file over the network to X server and get the result back... I will update my answer – radius Jul 19 '10 at 7:05
if I use the grep and cron technique I need to ssh first to all machines and set up the cron, it's tedious. I want to run ad hoc greps on various files (at the same location on all servers) – Ran Jul 19 '10 at 17:48
feedback
up vote -2 down vote

Perhaps something along the following bash loop?

export FQDNS="hostname.domain.tld another.domain.tld"
for host in $FQDNS
do
  ssh $host 'grep "andol was here" /var/log/syslog'
done
link|improve this answer
1  
This will only serially ssh into the remote hosts and run the grep command. You usually want to do that in parallel. – joschi Jul 19 '10 at 6:33
True, so I guess it all depends on what number "a bunch" actually translates into. – andol Jul 19 '10 at 6:36
And how /var/log/syslog comes here ? – radius Jul 19 '10 at 6:45
"a bunch" ~= 50 to 100 – Ran Jul 19 '10 at 6:59
1  
radius: /var/log/syslog is as much a placeholder as "andol was here". I hardly think this question is about what specific grep command to run at which log file? – andol Jul 19 '10 at 7:01
show 2 more comments
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.