Execute multiple ssh commands in the same shell (programatically)

Question

NOTE: I'm really looking for an SSH-y solution here, more than bash-y one.

I need to execute a series of commands over ssh, programatically. There are many ways to do this, but I can't find any that works with my constraints:

• some of the commands set variables in the shell. Therefore I dont think I can make this work using multiple shells.

• some of these variables are functions, not normal values, so I dont think I can simply serialize the output of env to "restore" the old environment.

• I need to access the stdout, stderr and exit code of each command separately. This makes it difficult to chain the commands using && or ;. I believe this also makes it difficult to use screen, because we cant reliably tell when a command finishes.

• I need real-time access to the stdout, stderr, etc so we can record when the output occurs. (I use "real-time" to mean "nowish", not "hard real-time" or anything like that).

Update

Why do I need a single bash session? Because commands may store arbitrary variables and functions in the environment. For example, it would be common to run rvm, which is a bash function added by a previously run command.

Why can't I use && or ;? Because I need the stdout and stderr or each command individually, and this would mix them.

Why can't I use screen? Because we want to run commands automatically, not as part of an interactive session. Consider the difficulty of this: I'd have to run the command with input form redirection, output via redirection, and then scp the results back. I'd need to echo $? to get the exit code. I wouldn't be able to timestamp the stderr/stdout. I wouldn't be able to timeout the command. And it would be incredibly buggy.

Answer 1

If you need something to be able to capture output, interact with a shell session and/or interact with pty applications like screen you could try writing in a higher-level language's SSH library, like Ruby's Net:SSH. It has a bit of a learning curve, but would accomplish everything you need, I believe.

Answer 2

I need to access the stdout, stderr and exit code of each command separately. This makes it difficult to chain the commands using && or ;.

Well, I don't know if I really recommend this, but you could write a shell function like this one:

function prefix ()
(
  PREFIX="$1"
  OUT=''
  while read -r ; do
    OUT="${OUT}${PREFIX}${REPLY}"$'\n'
  done
  echo "$OUT"
)

function run_cmd ()
{
  echo "Running command << $@ >> . . ."
  (
    ( ( "$@" ; echo "\$?: $?" >&3 ) | prefix 'STDOUT: ' >&3 ) 2>&1 \
    | prefix 'STDERR: '
  ) 3>&1
}

Then if ./foo.sh bar prints out foo\nbar\n to standard output and baz\nbip\n to standard error, and exits with code 1, then run_cmd ./foo.sh bar will print something like:

Running command << ./foo.sh bar >> . . .
$?: 0
STDOUT: foo
STDOUT: bar
STDERR: baz
STDERR: bip

That should let you chain a sequence of commands, provided you use run_cmd for all of the ones that you need STDOUT/STDERR/exit-codes for. (But since the command itself is run in five-layers-deep of subshells, you can't use run_cmd to run any command that needs to set environment variables for use by other commands. Hopefully you don't have any commands that both give output and set variables?)

Answer 3

Did you already take a look at pssh? It covers the most of your requirements, and with some creativity and imagination it may well help you:

• Can send the commands to multiple servers at the same time
• Is able to output in real time
• stdout and stderr can be saved to separate files
• reports you if command X at server Y succeeded or not
• can receive input from stdin and send that to servers
• has configurable timeout

And so on...

Answer 4

I also believe Kyle's approach would be the most promising. If that doesn't work out, you could write your main script in a way that it goes in an endless loop where it looks for command files which it then sources and makes stdout, stderr and exit code available in a form usable for your purpose.

You would then insert the command files via an alternate channel, likely just scp  or something similar.