6

Simple script:

#!/bin/bash
remote_ssh_account="depesz@localhost"
directory_to_tar=pgdata
exec nice tar cf - "$directory_to_tar" | \
    tee >(
        md5sum - | \
        ssh "$remote_ssh_account" 'cat - > /tmp/h3po4-MD5-2012-03-13.tar'
    ) | \
    ssh "$remote_ssh_account" 'cat - > /tmp/h3po4-data-2012-03-13.tar'

Theoretically it should deliver the data and checksum, to remote machine.

But somehow the tee fails with:

tee: standard output: Resource temporarily unavailable

Did strace, but nothing came out of it. I see both ssh started, and tee writing to both of them, but only the pipe to ( md5sum | ssh ) gets data - strace of the ssh "data" doesn't get any data, and after 5 seconds tee shows the error.

Aside from this all works. 2 connections are established, tar works, md5sum and its delivery works.

Improve this question
  • On the strace front, try strace -fF. Works like a champ for me. – BMDan Mar 14 '12 at 20:14
  • -fF is not in manual, but there is -f, -F and -ff. I used strace -ff -o strace.log -s 512 ./z.sh – user13185 Mar 14 '12 at 20:25
  • -fF is the same as -f -F. The -F is actually unneeded on most of the systems I work on, but it (tries to) follow vforks on older versions of strace, and has no effect on newer ones, but it doesn't hurt. -ff ends up writing a bunch of files (one per PID) that you then have to collate back into a sensible timeline, so I avoid it in the vast majority of situations. That said, -ff should work, but the output files will be oddly-named. – BMDan Mar 15 '12 at 12:55
3

Try this, an alternative way of doing the pipe that breaks:

#!/bin/bash
remote_ssh_account="depesz@localhost"
directory_to_tar=pgdata
nice tar cf - "$directory_to_tar" | \
    tee >(
        md5sum | \
        ssh "$remote_ssh_account" 'cat > /tmp/h3po4-MD5-2012-03-13.sum'
    ) > >(
        ssh "$remote_ssh_account" 'cat > /tmp/h3po4-data-2012-03-13.tar'
    )
Improve this answer
  • 1
    Seems to be working. I still would love to know why the | ssh thing breaks. – user13185 Mar 14 '12 at 19:51
5

Issue found. Here's the relevant portion of the strace:

[pid 10243] write(1, "pFl\r\347\345]\244Hi\336\253,-\231\247\344\234\241\332\302\252\315\243G\234\225+\241\323\316s"..., 4096 <unfinished ...>
[pid 10247] select(7, [3 4], [3], NULL, {10, 0} <unfinished ...>
[pid 10243] <... write resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
[pid 10247] <... select resumed> )      = 1 (out [3], left {10, 0})
[pid 10243] write(2, "tee: ", 5tee:  <unfinished ...>
(...)
[pid 10243] write(2, "standard output", 15standard output <unfinished ...>
(...)
[pid 10243] write(2, ": Resource temporarily unavailab"..., 34: Resource temporarily unavailable) = 34

So, what's happening is that the remote ssh isn't yet ready for the write to continue. Most programs handle this correctly, but tee decides to die in a pile. See http://lists.freebsd.org/pipermail/freebsd-bugs/2012-February/047528.html for one reference to this sort of behavior. There are a couple others that I found too in a brief search for "EAGAIN tee".

The solution that lhunath found works because it effectively forces bash to handle the EAGAIN. Elegant.

Improve this answer
  • Same error - you can also easily test it yourself - just change directory to tar (it shouldn't be too small), and remote_ssh_account - to something that you have passwordless access to. – user13185 Mar 14 '12 at 19:32
  • Changed; please try again. Yes, I know that doesn't do exactly what you want, but I just want to try it. Please be sure you're running it the same way (i.e., as a script, not directly on the command line). – BMDan Mar 14 '12 at 19:43
  • This works, and delivers the file. While I do appreciate the help - tests like this can be ran everywhere - did my original script not fail for you? – user13185 Mar 14 '12 at 19:49
  • 1
    I can get the problem to reliably recur with dd if=/dev/urandom bs=1M count=20 | tee >(md5sum - | ssh user@host 'cat - > /tmp/foo.md5') | ssh user@host 'cat > /tmp/foo.tar'. Working on debugging why now. – BMDan Mar 14 '12 at 19:54
  • It also happens with the order reversed (that is, with the cat>tar inside the >()). Neat. Adding the -i flag to tee (on a blind hunch) didn't help. – BMDan Mar 14 '12 at 19:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy