Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I want to do something like gzip a file, then immediately rsync it to another server so that the created gzip file will not be written on the local server. I was thinking of doing something like this... but obviously this fails.

gzip file | rsync -auv another_server

Anybody has any idea on this? Or is this even possible?

share|improve this question
1  
Why do you want/need rsync for this? –  Zoredache Jul 15 '09 at 23:15

5 Answers 5

My usual approach to this would be to use gzip and ssh as follows:

gzip -c file | ssh user@remote.host "cat > destination.gz"

This will send the zip output over ssh straight into the file "destination.gz" on the remote host. If you're doing this manually, just enter your ssh password; if you want to do it automatically, you'll need to set up ssh keys, per something like this:

http://pkeck.myweb.uga.edu/ssh/

share|improve this answer
    
so, instead of ssh... why can't I use rsync... since from what I understand rsync doesn't do encryption unlike ssh... so, it will be less resource intensive. –  bichonfrise74 Jul 16 '09 at 0:29
1  
rsync uses ssh as the transport, meaning it DOES do encryption, but ALSO adds overhead. If you want it to be less resource-intensive, change to a different cipher like blowfish. –  Dan Udey Jul 16 '09 at 1:20
    
@bichonfrise74, do you actually have an rsync daemon setup on the server, or are you using SSH for the transport? –  Zoredache Jul 16 '09 at 6:11
    
My similar approach. this will unpack it on the other end. tar -zcf - tmp | ssh username@servername.com tar -C /home/username -zxf - –  egorgry Jul 16 '09 at 13:12
    
If you know you're in a secure-enough environment, especially a high-speed one (between two machines in the same datacentre for example) my netcat example has the smallest amount of protocol overhead possible - no encryption at all. –  Alister Bulman Jul 16 '09 at 17:10

Don't take this personally, but from your question and comments it seems like you don't understand what rsync is actually for. It's not for moving files from one place to another – it's for synchronizing files between two places. If you don't want to keep a copy on the local machine, then you're not synchronizing, and rsync isn't the tool you want.

You're trying to transfer a file. For that, you can put it on disk and use ssh, ftp, sftp, ftps, and so on, or you can pipe it through ssh as Glenn suggested, which is the right answer and the only right way to do what you want. If you're worried about resource utilization (and for high-bandwidth connections this can make a difference), then change ciphers:

gzip -c somefile | ssh -c blowfish user@host 'cat > someotherfile'

If you try to do something (like pipe to rsync) and it doesn't work, it's possible you're doing it wrong, or it's possible you're trying to do the wrong thing. It's a question of tool for the job, and it's important not to rule out that whatever it is you're trying to do, there's a proper tool for it.

Understanding and keeping in mind the purpose of the sundry different tools available in a UNIX environment will let you use and manipulate that environment more effectively. A lot of tools have subtle differences or overlap in functionality, but unless you pay attention to them all, you'll miss out on a lot of the best ways to accomplish tasks.

share|improve this answer

Rsync will likely provide no benefits in this instance, since the content it is sending will be unique.

Compressing a file and sending the output to STDOUT is simple enough:

gzip -c file > another-file

If you are looking for speed, and have control over both ends, netcat has almost no overhead.

Setup the the destination machine to listen, on port 1234

user@bar$ nc -l -p 1234 > file.gz

Send, 'file.gz', to machine bar:1234

user@foo$ gzip -c file | nc bar 1234
share|improve this answer
    
So, why can't gzip pipe its output so that rsync can transfer it? I saw the nc thingy as well, but I'm still wondering why rsync cannot be used. –  bichonfrise74 Jul 16 '09 at 0:28
2  
@bichonfrise74: Because if you don't have a copy of the file on the destination, rsync is less efficient than a straight netcat or scp. –  womble Jul 16 '09 at 3:38
    
@bichonfrise74 gzip is perfectly capable of piping it's output. rsync is not capable of "transferring" it. rsync needs a source file and a destination file and synchronises them, sending only the differences between the two. –  rjmunro Jul 2 '12 at 23:32

rsync has a -z option that will transfer files compressed. If you want to do this in order to speed up transfer, you should use -z rather than mess around with gzip.

share|improve this answer
1  
There is an option gzip --rsyncable that makes the output file more rsync friendly (at a tiny increase in size), but it's rarely used. –  Alister Bulman Jul 18 '12 at 15:26

Combining gzip and rsync doesn't work.

If you send the uncompressed file, then rsync works by sending only the bits that have changed. But when you gzip a file, the resulting file is going to be significantly different each time. Piping it to rsync would at best achieve a similar result to ssh, and could even be worse.

The reason a compressed file is different each time is because it profiles the data before compressing. That means even a small change in the file can mean in a change in the profile, affecting every byte that's compressed.

share|improve this answer
4  
Take a look at gzip --rsyncable, the compressed file doesn't have to be significantly different with slightly different input. –  womble Jul 16 '09 at 3:39
    
@womble: WOW, that's really cool. I had no idea gzip could do that. Thanks! –  Glenn Willen Jul 16 '09 at 22:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.