6

I want to do something like gzip a file, then immediately rsync it to another server so that the created gzip file will not be written on the local server. I was thinking of doing something like this... but obviously this fails.

gzip file | rsync -auv another_server

Anybody has any idea on this? Or is this even possible?

1
  • 1
    Why do you want/need rsync for this?
    – Zoredache
    Jul 15, 2009 at 23:15

5 Answers 5

8

My usual approach to this would be to use gzip and ssh as follows:

gzip -c file | ssh [email protected] "cat > destination.gz"

This will send the zip output over ssh straight into the file "destination.gz" on the remote host. If you're doing this manually, just enter your ssh password; if you want to do it automatically, you'll need to set up ssh keys, per something like this:

http://pkeck.myweb.uga.edu/ssh/

5
  • so, instead of ssh... why can't I use rsync... since from what I understand rsync doesn't do encryption unlike ssh... so, it will be less resource intensive. Jul 16, 2009 at 0:29
  • 1
    rsync uses ssh as the transport, meaning it DOES do encryption, but ALSO adds overhead. If you want it to be less resource-intensive, change to a different cipher like blowfish.
    – Dan Udey
    Jul 16, 2009 at 1:20
  • @bichonfrise74, do you actually have an rsync daemon setup on the server, or are you using SSH for the transport?
    – Zoredache
    Jul 16, 2009 at 6:11
  • My similar approach. this will unpack it on the other end. tar -zcf - tmp | ssh [email protected] tar -C /home/username -zxf -
    – egorgry
    Jul 16, 2009 at 13:12
  • If you know you're in a secure-enough environment, especially a high-speed one (between two machines in the same datacentre for example) my netcat example has the smallest amount of protocol overhead possible - no encryption at all. Jul 16, 2009 at 17:10
5

Don't take this personally, but from your question and comments it seems like you don't understand what rsync is actually for. It's not for moving files from one place to another – it's for synchronizing files between two places. If you don't want to keep a copy on the local machine, then you're not synchronizing, and rsync isn't the tool you want.

You're trying to transfer a file. For that, you can put it on disk and use ssh, ftp, sftp, ftps, and so on, or you can pipe it through ssh as Glenn suggested, which is the right answer and the only right way to do what you want. If you're worried about resource utilization (and for high-bandwidth connections this can make a difference), then change ciphers:

gzip -c somefile | ssh -c blowfish user@host 'cat > someotherfile'

If you try to do something (like pipe to rsync) and it doesn't work, it's possible you're doing it wrong, or it's possible you're trying to do the wrong thing. It's a question of tool for the job, and it's important not to rule out that whatever it is you're trying to do, there's a proper tool for it.

Understanding and keeping in mind the purpose of the sundry different tools available in a UNIX environment will let you use and manipulate that environment more effectively. A lot of tools have subtle differences or overlap in functionality, but unless you pay attention to them all, you'll miss out on a lot of the best ways to accomplish tasks.

3

Rsync will likely provide no benefits in this instance, since the content it is sending will be unique.

Compressing a file and sending the output to STDOUT is simple enough:

gzip -c file > another-file

If you are looking for speed, and have control over both ends, netcat has almost no overhead.

Setup the the destination machine to listen, on port 1234

user@bar$ nc -l -p 1234 > file.gz

Send, 'file.gz', to machine bar:1234

user@foo$ gzip -c file | nc bar 1234
3
  • So, why can't gzip pipe its output so that rsync can transfer it? I saw the nc thingy as well, but I'm still wondering why rsync cannot be used. Jul 16, 2009 at 0:28
  • 2
    @bichonfrise74: Because if you don't have a copy of the file on the destination, rsync is less efficient than a straight netcat or scp.
    – womble
    Jul 16, 2009 at 3:38
  • @bichonfrise74 gzip is perfectly capable of piping it's output. rsync is not capable of "transferring" it. rsync needs a source file and a destination file and synchronises them, sending only the differences between the two.
    – rjmunro
    Jul 2, 2012 at 23:32
2

rsync has a -z option that will transfer files compressed. If you want to do this in order to speed up transfer, you should use -z rather than mess around with gzip.

1
  • 1
    There is an option gzip --rsyncable that makes the output file more rsync friendly (at a tiny increase in size), but it's rarely used. Jul 18, 2012 at 15:26
0

Combining gzip and rsync doesn't work.

If you send the uncompressed file, then rsync works by sending only the bits that have changed. But when you gzip a file, the resulting file is going to be significantly different each time. Piping it to rsync would at best achieve a similar result to ssh, and could even be worse.

The reason a compressed file is different each time is because it profiles the data before compressing. That means even a small change in the file can mean in a change in the profile, affecting every byte that's compressed.

2
  • 6
    Take a look at gzip --rsyncable, the compressed file doesn't have to be significantly different with slightly different input.
    – womble
    Jul 16, 2009 at 3:39
  • @womble: WOW, that's really cool. I had no idea gzip could do that. Thanks! Jul 16, 2009 at 22:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .