3

TL;DR

I want to remotely execute a script that begins synchronously (so that the local ssh command fails if the preparatory commands fail) and then goes asynchronous, thus terminating the local ssh command.

Details

I have a script on a remote server that slowly downloads a multi-gigabyte file. I would like to launch this script via SSH from the local server with ssh remote-server 'nohup /path/to/script arguments' but kill the SSH connection when I know that the script has been successfully started the download. Once it is launched, the SSH connection doesn't serve any useful purpose, systematically fails somewhere during the downloads, and blocks the execution on the local server.

I can't just do ssh -f or ssh & because I need the command to fail on the local server server if the remote script doesn't launch, fails on the first commands before the download, or if the remote server is not reachable.

I have tried various nohup and screen tricks. The closest I got was the following:

  • Launched by the local server:

    ssh -t me@remote-server 'screen -S long-download /path/to/download.sh'
    

    or

    ssh -t me@remote-server 'nohup screen -S long-download /path/to/download.sh'
    
  • In download.sh launched on the remote server:

    preliminary-instructions
    # if anything fails so far then the local server's ssh command should fail synchronously
    download-command &
    some-more-checking
    screen -d long-download # we can now safely end the ssh session
    

But somehow screen still gets killed…

Sample reproduction code

  • On a remote server, in /path/to/test.sh:

    #!/bin/bash
    # uncomment this line to validate synchronous failure
    sleep 10 && echo foo >/tmp/bar &
    screen -d long-download
    
  • Locally:

    ssh -t me@remote-server 'nohup screen -S long-download /path/to/test.sh'
    

2 Answers 2

2

Look at ssh subsystems. I do something similar, but not with a script (i.e. I use a.out/elf compiled programs.) But I just tested with a bash script and it works.

From the ssh man page:

     -s      May be used to request invocation of a subsystem on the remote
         system.  Subsystems are a feature of the SSH2 protocol which
         facilitate the use of SSH as a secure transport for other appli-
         cations (eg. sftp(1)).  The subsystem is specified as the remote
         command.

Use -s (lower case) on the client to specify the subsystem

Add another Subsystem entry to sshd_config on the server which points to your script.

    # override default of no subsystems
Subsystem       logger  /usr/local/libexec/my-logger.sh
Subsystem       sftp    /usr/libexec/sftp-server

Your script should be started just as any other file with execute permissions. My example:

#! /usr/local/bin/bash
logger "testing 1.2.3"
echo "return text from my-logger"
exec 0>&- # close stdin
exec 0<&- 
exec 1>&- # close stdout
exec 1<&- 
exec 2>&- # close stderr 
exec 2<&- 
logger "subsystem: nohup new script"
nohup /path/to/another/script/logger2.sh &  
logger "subsystem: the subsystem started script is now done"
exit 0  

You should be able to return useful error text for success or failure. The ssh connection (and thus stdin, stdout and stderr) is up until the subsystem ends it. The subsystem can continue to run or not.

Have your other script (logger2.sh above) sleep for 5-10 minutes so you can use netstat to see the ssh connection gone, ssh client back to the shell and e.g. ps ax show that the script is still running in the background. This script may also need to close fd's etc. It's just a quick example. Good luck.

4
  • Thanks for your answer. I can't seem to find much documentation on this subsytem feature. Would you have an example of how the subsystem can close the ssh connection yet keep running ? Jul 22, 2013 at 8:59
  • I do this using C/C++. My program just closes stdin, stdout and stderr (e.g. the socket connection to ssh client.) The program doesn't exit however, it does it's thing. You should be able to translate this to your script language. HTH Jul 23, 2013 at 23:37
  • I've updated my prior answer with a quick test that works for me using bash. Jul 24, 2013 at 0:50
  • Successfully reproduced without the need subsystems. First script that closes stdin, stdout and stderr + launches second script/command via nohup … & is sufficient. The second script should also close FDs if it is launched before they are closed in the first one. See combined example. In the meantime we have found another solution that uses 2 scripts and screen -d -m. I will be posting it here for the sake of exhaustivity. I'm accepting your answer though, I like its elegance. Thanks again. Jul 24, 2013 at 9:47
1

Aleternative solution to BostonDriver's answer: screen -d -m.

In async-launcher.sh:

#!/bin/bash
echo connected &&
#       preliminary-command &&
        screen -S index-fetch -d -m $(cd ${0%/*} && pwd -P)/long-download.sh

sleep 5    # ensure long-download.sh didn't immediately fail
if ((`screen -ls | grep index-fetch | wc -l`)); then
        echo ok
else
        echo ko
        exit 1
fi

And long-download.sh can contain all the asynchronous commands.

Launch with a simple ssh me@remote-server /path/to/async-launcher.sh. Also has an added benefit: the screen can be reattached at a later time.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .