Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm putting together a deployment script which tars up a directory of my code, names the tar file after the current date and time, pushes that up to the server, untars it in a directory of the same name and then swaps a "current" symlink to point at the new directory. This means my older deployments stay around in timestamped directories (at least until I delete them).

The tar file is around 5MB and it takes nearly a minute to transfer. I'd like to speed this up.

I assume each new tarball is pretty similar in structure to the previous tarball (since I'm often only changing a few lines of source code in between deployments). Is there a way to take advantage of this fact to speed up my uploads using rsync?

Ideally I'd like to say "hey rsync, upload this local file called 2009-10-28-222403.tar.gz to my server, but it's only a tiny bit different from the file 2009-10-27-101155.tar.gz which is already up there, so try to just send over the differences". Is this possible, or is there another tool I should be looking at?

share|improve this question
    
Thanks for all the responses - I'll have a more detailed think about what I'm trying to achieve and pick the most appropriate solution once I've updated my initial assumptions. –  Simon Willison Oct 29 '09 at 5:15

10 Answers 10

up vote 6 down vote accepted

I'm putting together a deployment script which tars up a directory of my code, names the tar file after the current date and time, pushes that up to the server, untars it in a directory of the same name and then swaps a "current" symlink to point at the new directory.

Personally, I think you should skip using tar, and instead look at using the --link-dest or --copy-dest feature of rsync. The link-dest function is pretty cool it will know to look at the previous sync of the directory, and if the files where identical it will hardlink them together skipping the need to retransfer the file each time.

mkdir -p /srv/codebackup/2009-10-12 \
         /srv/codebackup/2009-10-13

# first backup on 10-12
rsync -a sourcehost:/sourcepath/ \
         /srv/codebackup/2009-10-12/

# second backup made on 10-13
rsync -a --link-dest=/srv/codebackup/2009-10-12/
         sourcehost:/sourcepath/ \
         /srv/codebackup/2009-10-13/

Your second run of rsync will only transfer changed files. Identical files will be hard linked together. You can delete the older tree and the new backup will still be 100% complete. You will save a lot of storage space since you will not be keeping multiple copies of identical files.

share|improve this answer
2  
I'm now using this trick and my deployments to a remote server have gone down from 60+ seconds to around 3 seconds! Thank you very much. –  Simon Willison Nov 8 '09 at 9:35
2  
For anyone else who wants to try this, be sure to include the trailing slashes on ALL of the directory arguments - they are significant, if you leave them off you end up with a different directory structure being created. –  Simon Willison Nov 8 '09 at 9:36

You might look into rsync's fuzzy mode (activated with the --fuzzy switch)

This allows rsync to select a file on the destination system that is similar to the file being transferred, and use that file as the base upon which to apply its delta uploading algorithm. It's a bit memory and I/O hungry, particularly if you have a large directory on the destination side, however it should give you the upload improvements you're looking for without having to rejigger your approach as other answers have suggested.

share|improve this answer

Ok, I haven't tried this, but it'd be interesting to see how it works in your case.

You'll want to minimise the changes on each invocation of tar. Would it help to make sure that the files are always in the same order in each instance. You can then compress with the --rsyncable option.

Can you order the files by last modified date? That way the files that don't change are always in the same order, and at the beginning, and the files that change are at the end, so when they change length they don't break the blocking algorithm.

tar cvf - -T `find . -type f | xargs ls --sort=time -r` | gzip -9 --rsyncable

Another thing to consider is that tar supports blocking, and will pad out each file with nulls to a block offset. Check out block sizes. You could set this to the rsync block size (ah, that depends on the size of the file, erm how about 8k?). Which will help the algorithm when a single file is reordered. Now, drop the gzip on each end (gzip the last-but-one on the server if you're worried about disk space), and I think you might get the speed up you want.

I'm not that impressed with the --rsyncable option. I'm using it on daily postgres dumps, and find that, although only a small amount of the dump changes each day, rsync uses about half the bandwidth of just copying the .gz around. I might ask a question about this actually.

I think you'll be best off with the efficient rsync of individual files included in other answers, and then generating the .tar.gz from the resulting directory on the server (or the client if that's where you want to keep your archive). What's wrong with your version control system, as a record of what you deployed when? You're not deploying uncommitted code are you?

share|improve this answer

The other solutions ignore the reason you wanted to use rsync in the first place, that is, only sending the files that have changed. How about approaching it slightly differently, eschewing tarballs in the process, but keeping the benefits of rsync and rollbacks.

First, on your remote host create a recent directory for rsync:

mkdir /var/www/recent

Then create a symlink to point to this directory:

ln -s /var/www/recent /var/www/active

Configure Apache to serve up files under /var/www/active

Then, rsync your local folder to your remote host:

rsync -v -r --delete ~/Sites/Foo/ foo.org:/var/www/recent

Then back up the remote directory, remotely:

ssh foo.org cp -R /var/www/current /var/www/`date +%Y%m%d%H%M%S`

Now, after a while, your remote /var/www directory should look something like:

/var/www/200909041234
/var/www/200910121712
/var/www/200911030446
/var/www/active
/var/www/recent

If you need to rollback, you switch the symlink:

ssh foo.org ln -s /var/www/200911030446 /var/www/active

Easy peasy!

For bonus points:

  1. Open up Automator
  2. Create a new service
  3. Create new action to run a shell script
  4. Plop the rysnc command and the remote directory copy commands into this window
  5. Save the workflow as "Publish Foo Site"
  6. Go to System Preferences
  7. Go to Keyboard Preferences
  8. Go to Services
  9. Find your Publish Foo Site service and bind to a key shortcut

Automated publishing from any application in OS X!

share|improve this answer

use hardlinks for copying and transfer only the diffs; example : cp -lr old_date_dir/ new_date_dir/ (this is on the "server") rsync -ax --numeric-ids code server:/path/new_date_dir/

this will work because rsync unlinks before transferring the diffs.

share|improve this answer

Simon: repeating the same question mentioned above...any reason why needing to tar in the first place?

share|improve this answer
    
Mainly just that I quite like the idea of having a bunch of .tar.gz files lying around representing older deployments (I can delete the older deployed directories and just leave the tarballs, which feels easier to manage than lots of folders full of hundreds of files). Not a very pressing reason though - could just as easily transfer up whole directories with rsync and create a tar on the server of each old version as and when I want to archive it. –  Simon Willison Oct 29 '09 at 5:14

I think using tar here is the wrong answer. What I would do, for this particular case is, cp -rp your "current" code on the server to a dated directory. Then rsync your local code checkout against "current". So basically this:

  1. ssh user@host cp -rp /path/to/current /path/to/2009-10-28/

  2. rsync /local/copy user@host:/path/to/current

This gives you the backup copy you want, syncs your changes, and will be much faster than tar+scp+untar.

Hope that helps!

share|improve this answer

This isn't directly related, as it doesn't address the rsync solution, but it might help a bit with the filesize: have you tried usin bzip2 compression instead of gzip?

Instead of tar czvf blah.tar.gz files, you can do tar cjvf blah.tar.bz2 files and get some better compression (assuming you have bzip2 installed, of course).

share|improve this answer
    
bzip2 is very outdated---it's slow compared to other modern compressors like lzma, which offer similar/better compression. Also, since the file being transferred is 5 MB, it's not reasonable to expect bzip2/lzma/whatever to compress files that much more to actually be significant. –  Samat Jain Oct 28 '09 at 23:22
    
I'm not sure that's an accurate statement. In most cases bzip2 is faster, and there times when the size-savinsg are greater with bzip2 than with lzma. I will, however, agree that the savings would be minimal using another compression algorithm (gzip being the fastest, but with the largest file-sizes). –  bchang Oct 28 '09 at 23:36

rsync AFAIK can't do this directly, but you can structure your tarballs to make them transfer faster, taking advantage of the fact that they're similar.

Check out gzip's --resyncable flag. From the manual:

While compressing, synchronize the output occasionally based on the input. This increases size by less than 1 percent most cases, but means that the rsync(1) program can much more efficiently synchronize files compressed with this flag. gunzip cannot tell the difference between a compressed file created with this option, and one created without it.

This will make your similar tarballs actually more similar such that rsync will be able to recognize them.

You'd probably have to modify your deployment scripts a little to reduce the amount of transfer, because I don't think rsync can be told to "look at another file"... what I'd do is always rsync something called current.tar.gz (compressed with gzip and the above flag), and then rename it for archival purposes on the server. That, or rename an old tarball on the server to the name of the tarball that is about to be uploaded, so that rsync can use it.

share|improve this answer

What does it have to be a tar file? Why not rsync the code to your deploy directory and use the tar as backup?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.