How to make SCP on Linux output "success" to stdout instead of stderr?

Question

We have a cron job that sends stuff to another company using SCP. The command look like this:

 10 0 * * * ssh USER@LOCAL-SERVER 'scp -q ~/some/dir/* REMOTE_SERVER:/other/dir/' >> ~/log/some.log

This was setup by someone else. Now my problem is that my boss is on the cron mailing list, and despite the ">> ~/log/some.log" cron still sends an email every day, because even if the transfer was successful, it still outputs:

Connection to REMOTE_SERVER 12345 port [tcp/*] succeeded!

And my boss doesn't want to get that email unless an error happens.

The command is really inside a script, which I have omitted to make the make the question simpler. It runs on a RHEL 6 server. But since it's the only line where we talk to REMOTE_SERVER, then it must come from there. And the reason we perform the command on LOCAL-SERVER (Ubuntu 10.04.4 LTS) is because we are on the DB server, which cannot cross the firewall, so we first copy the data to another local server, and upload it from there.

When used without the -q, I also get the following output:

SSH Server supporting SFTP and SCP

I don't want to redirect all stderr output, because I still want to get an email when there is an error.

Answer 1

I don't think you can redirect STDERR output selectively without rewriting the program to write those particular messages to STDOUT instead of STDERR. What you can do, however, is write a wrapper script that filters out the unwanted message, and run that script in the cron job.

#!/bin/bash

(ssh USER@LOCAL-SERVER ... 3>&1 1>&2- 2>&3-) 2>> ~/log/some.log \
  | grep -v "Connection .* succeeded" 1>&2

exit $?

Out of curiosity, why aren't you running a cron job with the scp command directly on LOCAL-SERVER?

Edit: You want to filter out some of the STDERR output while redirecting all STDOUT output to a file, but the pipe can connect STDOUT of one process to STDIN of another process. |& or 2>&1 | won't work, because STDOUT is already is redirected to ~/log/some.log, so there'd be nothing going into the pipe, and a lot of unwanted output in your log file.

However, it's possible to swap the descriptors, so that STDOUT actually goes to STDERR and STDERR actually goes to STDOUT. Then you can redirect all STDERR output to your log file and filter the STDOUT output via grep. Someone over at stackoverflow posted this neat little trick. The 1>&2 returns the filtered output back to STDERR.

Answer 2

Assuming you are using BASH:

Swap stdout and stderr by putting the duplication first:

2>&1 >> ~/log/some.log ssh USER@LOCAL-SERVER 'scp -q ~/some/dir/* REMOTE_SERVER:/other/dir/'

Then you set up a pipe to grab whatever is going to stderr:

2>&1 >> ~/log/some.log ssh USER@LOCAL-SERVER 'scp -q ~/some/dir/* REMOTE_SERVER:/other/dir/' |

And grep only for failed messages:

2>&1 >> ~/log/some.log ssh USER@LOCAL-SERVER 'scp -q ~/some/dir/* REMOTE_SERVER:/other/dir/' | grep -E "failure message"

This will produce the output that cron grabs and includes in its MAIL message. All other "stderr" output will disappear (be lost). If for some reason you'd like to save a copy of all stderr, then use the tee command first:

2>&1 >> ~/log/some.log ssh USER@LOCAL-SERVER 'scp -q ~/some/dir/* REMOTE_SERVER:/other/dir/' | tee -a ~/log/some_other.log | grep -E "failure message"

which will put a copy of ALL stderr into the new log file, and then only the failure messages will make their way into the CRON email.