I have heard that
clusterssh are two popular ones, but I thought I would open it to discussion here and see what the community's experiences with these tools were? What are the gotchas? Any decent hacks or use cases?
Its important to know what you want to do. If you want to run 'apt-get update' on lots of servers, clusterssh is reportedly an easy-to use and effective tool. See Kyle Rankin's article on clusterssh at the Linux Journal.
If your work on different servers will require logic loops, putting, getting or otherwise interacting with each host (or possibly having to interact with a host because of an error condition) a tool such as Fabric (which has just hit its 1.0 release) is invaluable. You'll probably enjoy most using Fabric if you are a pythonista.
In either case one is probably expecting to do fairly simple tasks on the remote servers affected, and dropping in specially on any problem hosts. However Fabric provides more options for dealing with variation.
One thing that is generally not handled well is dealing with multiple hosts behind a gateway host, requiring one to connect first to the gateway and then port forwarding from there. As ssh itself does this quite simply, and one can set one's
.ssh/config file to something like
Host gateway Port 9000 User remoteadmin HostName datacentre1.com # mail LocalForward 9100 192.168.9.11:22 # phones LocalForward 9101 192.168.9.12:22 Host dc1mail Port 9100 Host 127.0.0.1 User localadmin Host dc1phones Port 9201 Host 127.0.0.1 User localadmin
In order to do the equivalent of
ssh dc1mail 'cmd' &&
ssh dc1phones 'cmd' one has to first bring up
ssh gateway. It would be nice if tools like clusterssh and Fabric would permit this to be done as part of their tools. Naturally, you can do this yourself by hand.
The SSH Power Tool (sshpt) was designed for parallel SSH without requiring that the user setup pre-shared SSH keys. It is superior to pssh and clusterssh in that it supports executions via sudo and can also copy files and execute them afterwards (optionally, via sudo as well). By default it outputs results in CSV format but sshpt.py doubles as an importable Python module so you can use it in your own programs (I used to use it as a back-end behind a a custom-built web-based reporting tool at my former employer).
Mussh is a good alternative, it is already included in many Linux distros.
Mussh is a shell script that allows you to execute a command or script over ssh on multiple hosts with one command. When possible mussh will use ssh-agent and RSA/DSA keys to minimize the need to enter your password more than once.
I use cssh and it works like a charm. You can edit the same file on several servers easily and then check only one window and change for instance value of particular variable. Repeat that for every window, hit :wq in vim and voila! Work is done ;)
It is also excellent tool for making aptitude upgrade.
There is also dsh for parallel ssh runs.
My favorite parallel SSH tool is PuSSH ... IMO the syntax is more intuitive than any other similar packages available, and it doesn't actually need installation as an RPM, compilation or anything else. It just downloads as a couple of scripts, and just works.