We have a cron job that sends stuff to another company using SCP. The command look like this:

 10 0 * * * ssh USER@LOCAL-SERVER 'scp -q ~/some/dir/* REMOTE_SERVER:/other/dir/' >> ~/log/some.log

This was setup by someone else. Now my problem is that my boss is on the cron mailing list, and despite the ">> ~/log/some.log" cron still sends an email every day, because even if the transfer was successful, it still outputs:

Connection to REMOTE_SERVER 12345 port [tcp/*] succeeded!

And my boss doesn't want to get that email unless an error happens.

The command is really inside a script, which I have omitted to make the make the question simpler. It runs on a RHEL 6 server. But since it's the only line where we talk to REMOTE_SERVER, then it must come from there. And the reason we perform the command on LOCAL-SERVER (Ubuntu 10.04.4 LTS) is because we are on the DB server, which cannot cross the firewall, so we first copy the data to another local server, and upload it from there.

When used without the -q, I also get the following output:

SSH Server supporting SFTP and SCP

I don't want to redirect all stderr output, because I still want to get an email when there is an error.


I don't think you can redirect STDERR output selectively without rewriting the program to write those particular messages to STDOUT instead of STDERR. What you can do, however, is write a wrapper script that filters out the unwanted message, and run that script in the cron job.


(ssh USER@LOCAL-SERVER ... 3>&1 1>&2- 2>&3-) 2>> ~/log/some.log \
  | grep -v "Connection .* succeeded" 1>&2

exit $?

Out of curiosity, why aren't you running a cron job with the scp command directly on LOCAL-SERVER?

Edit: You want to filter out some of the STDERR output while redirecting all STDOUT output to a file, but the pipe can connect STDOUT of one process to STDIN of another process. |& or 2>&1 | won't work, because STDOUT is already is redirected to ~/log/some.log, so there'd be nothing going into the pipe, and a lot of unwanted output in your log file.

However, it's possible to swap the descriptors, so that STDOUT actually goes to STDERR and STDERR actually goes to STDOUT. Then you can redirect all STDERR output to your log file and filter the STDOUT output via grep. Someone over at stackoverflow posted this neat little trick. The 1>&2 returns the filtered output back to STDERR.

  • It's a "security thing"; the DB server can connect out to LOCAL-SERVER, but not the other way around. So I have to push the files, instead of pulling them.
    – monster
    Sep 3 '12 at 15:15
  • Actually, I think there is a 2>&1 missing just before the pipe. The file redirection already swallows all stdout output, so all that is left is stderr, but | does not care about it, so we have to move stderr to stdout, before sending it through the pipe.
    – monster
    Sep 3 '12 at 15:25
  • Actually, that won't work either, because it would redirect all STDERR output to STDOUT, which is already redirected to ~/log/some.log, so the entire output would go to your log. You'll have to swap the file descriptors and redirect STDERR to your log file for this to work. See updated answer. Sep 3 '12 at 17:24

Assuming you are using BASH:

Swap stdout and stderr by putting the duplication first:

2>&1 >> ~/log/some.log ssh USER@LOCAL-SERVER 'scp -q ~/some/dir/* REMOTE_SERVER:/other/dir/'

Then you set up a pipe to grab whatever is going to stderr:

2>&1 >> ~/log/some.log ssh USER@LOCAL-SERVER 'scp -q ~/some/dir/* REMOTE_SERVER:/other/dir/' |

And grep only for failed messages:

2>&1 >> ~/log/some.log ssh USER@LOCAL-SERVER 'scp -q ~/some/dir/* REMOTE_SERVER:/other/dir/' | grep -E "failure message"

This will produce the output that cron grabs and includes in its MAIL message. All other "stderr" output will disappear (be lost). If for some reason you'd like to save a copy of all stderr, then use the tee command first:

2>&1 >> ~/log/some.log ssh USER@LOCAL-SERVER 'scp -q ~/some/dir/* REMOTE_SERVER:/other/dir/' | tee -a ~/log/some_other.log | grep -E "failure message"

which will put a copy of ALL stderr into the new log file, and then only the failure messages will make their way into the CRON email.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.