13

I take backups of a server with rsync, but those backups are quite large, and I'd like to compress them. Is there a way to have some sort of wrapper around rsync to gunzip the file in the backup, rsync the changes from the live file, then re-gzip the file as soon as it is copied?

I.e. if the source files on the live are foo, bar and baz, the backup has foo.gz, bar.gz and baz.gz.

To restate: I want compressed files at one end and uncompressed files at the other end. I don't want to compress before rsyncing, because even with --rsyncable, it makes rsync less efficient. I know about the -z option to rsync. I don't have space on the backup machine to store all the files uncompressed.

Improve this question

4 Answers 4

Reset to default
12

I'm not sure I understand what you're trying to do, but you could create gzipped backups on the main server making sure that they're readily rsyncable and not bother decompressing anything in transit.

gzip(1)

--rsyncable
       While compressing, synchronize the output occasionally based on
       the  input.   This  increases  size by less than 1 percent most
       cases, but means that the rsync(1) program can much more  effi‐
       ciently  synchronize  files  compressed with this flag.  gunzip
       cannot tell the difference between a  compressed  file  created
       with this option, and one created without it.
Improve this answer
2
  • Nice. I didn't know about the rsyncable flag. That's handy. I wonder why that isn't the default for a 1% loss
    – Matt Simmons
    Aug 11, 2009 at 15:52
  • NB: the --rsyncable option is not available on all versions of gzip, and also it comes from a commonly applied patch rather than the core gzip code, so version numbers are not enough to know if you have it.
    – mc0e
    Mar 26, 2015 at 17:29
6

I think that rsyncing to a fuseCompress mounted filesystem might be the best option. The project's history specifically mentions them making improvements to rsync's performance with it.

http://code.google.com/p/fusecompress/

https://github.com/tex/fusecompress/

Improve this answer
-1

You can write a wrapper to rsync. If you 'pull' data, there's nothing special to do. If you want to 'push' data from the original machines to the backup server it's more delicate to make it work:

First you have to be sure the wrapper is invoked with the same name, path and parameters as the original rsync, at least for the user under which you run the backups.

Second, be sure not to write anything to stdout, because it would mangle the communication channel.

And third, be sure to pass all the parameters you get, just modifying as appropriate for the newly uncompressed destination.

Improve this answer
-2

He's probably looking for data transmission compression via rsync, not file compression at the other end.

Check out: http://jimmyg.org/blog/2007/rsync-basics.html

Compression parameters from the man page:

   -z, --compress              compress file data during the transfer

        --compress-level=NUM    explicitly set compression level

By default ssh does its own compression to an extent. I don't know how much better rsync's is over native ssh. If the majority of your files are already compressed into gzip files, there probably wont be much that rsync's compression mechanism can do for you. You can only squeeze a file so much.

Improve this answer
1
  • No, I'm looking for file compression at one end only. rsyncing already compressed files is inefficient, so I'd like the rsync part to happen with uncompressed data.
    – rjmunro
    Aug 11, 2009 at 14:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .