Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I take backups of a server with rsync, but those backups are quite large, and I'd like to compress them. Is there a way to have some sort of wrapper around rsync to gunzip the file in the backup, rsync the changes from the live file, then re-gzip the file as soon as it is copied?

I.e. if the source files on the live are foo, bar and baz, the backup has foo.gz, bar.gz and baz.gz.

To restate: I want compressed files at one end and uncompressed files at the other end. I don't want to compress before rsyncing, because even with --rsyncable, it makes rsync less efficient. I know about the -z option to rsync. I don't have space on the backup machine to store all the files uncompressed.

share|improve this question
add comment

4 Answers

up vote 5 down vote accepted

I think that rsyncing to a fuseCompress mounted filesystem might be the best option. The project's history specifically mentions them making improvements to rsync's performance with it.

http://code.google.com/p/fusecompress/

https://github.com/tex/fusecompress/

share|improve this answer
add comment

You can write a wrapper to rsync. If you 'pull' data, there's nothing special to do. If you want to 'push' data from the original machines to the backup server it's more delicate to make it work:

First you have to be sure the wrapper is invoked with the same name, path and parameters as the original rsync, at least for the user under which you run the backups.

Second, be sure not to write anything to stdout, because it would mangle the communication channel.

And third, be sure to pass all the parameters you get, just modifying as appropriate for the newly uncompressed destination.

share|improve this answer
add comment

He's probably looking for data transmission compression via rsync, not file compression at the other end.

Check out: http://jimmyg.org/blog/2007/rsync-basics.html

Compression parameters from the man page:

   -z, --compress              compress file data during the transfer

        --compress-level=NUM    explicitly set compression level

By default ssh does its own compression to an extent. I don't know how much better rsync's is over native ssh. If the majority of your files are already compressed into gzip files, there probably wont be much that rsync's compression mechanism can do for you. You can only squeeze a file so much.

share|improve this answer
    
No, I'm looking for file compression at one end only. rsyncing already compressed files is inefficient, so I'd like the rsync part to happen with uncompressed data. –  rjmunro Aug 11 '09 at 14:37
add comment

I'm not sure I understand what you're trying to do, but you could create gzipped backups on the main server making sure that they're readily rsyncable and not bother decompressing anything in transit.

gzip(1)

--rsyncable
       While compressing, synchronize the output occasionally based on
       the  input.   This  increases  size by less than 1 percent most
       cases, but means that the rsync(1) program can much more  effi‐
       ciently  synchronize  files  compressed with this flag.  gunzip
       cannot tell the difference between a  compressed  file  created
       with this option, and one created without it.
share|improve this answer
    
Nice. I didn't know about the rsyncable flag. That's handy. I wonder why that isn't the default for a 1% loss –  Matt Simmons Aug 11 '09 at 15:52
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.