Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Is there a command, such as rsync, which can synchronise huge, sparse, files from one linux server to another?

It is very important that the destination file remains sparse. It may be longer (but not bigger) than the drive which contains it. Only changed blocks should be sent across the wire.

I have tried rsync, but got no joy. groups.google.com/group/mailing.unix.rsync/browse_thread/thread/94f39271980513d3

If I write a programme to do this, am I just reinventing the wheel? http://www.finalcog.com/synchronise-block-devices

Thanks,

Chris.

share|improve this question

8 Answers 8

up vote 2 down vote accepted

I ended up writing software to do this:

http://www.virtsync.com

This is commercial software costing $49 per physical server.

I can now replicate a 50GB sparse file (which has 3GB of content) in under 3 minutes across residential broadband.

chris@server:~$ time virtsync -v /var/lib/libvirt/images/vsws.img backup.barricane.com:/home/chris/
syncing /var/lib/libvirt/images/vsws.img to backup.barricane.com:/home/chris/vsws.img (dot = 1 GiB)
[........>.........................................]
done - 53687091200 bytes compared, 4096 bytes transferred.

real    2m47.201s
user    0m48.821s
sys     0m43.915s 
share|improve this answer
1  
TBH, the stating timing at which you can sync is pretty meaningless because it obviously depends on the amount of data changed. What would be more accurate to say is that it takes your software 3 minutes to figure out which blocks have changed, and even that speed probably depends on your disk i/o and maybe CPU cycles available. –  Reality Extractor May 3 '13 at 7:43
1  
You should disclose that this is commercial software costing $98 or more for network functionality. –  Reid May 13 at 20:46

Take a look at Zumastor Linux Storage Project it implements "snapshot" backup using binary "rsync" via the ddsnap tool.

From the man-page:

ddsnap provides block device replication given a block level snapshot facility capable of holding multiple simultaneous snapshots efficiently. ddsnap can generate a list of snapshot chunks that differ between two snapshots, then send that difference over the wire. On a downstream server, write the updated data to a snapshotted block device.

share|improve this answer

lvmsync does this.

Here's a usage transcript. It creates an LVM snapshot on the source, transfers the logical partition. You can transfer incremental updates of the changes since snapshot creation as often as you like.

share|improve this answer
    
I have tried it, but it doesn't work, and author is not willing to support –  user1007727 Jun 4 '13 at 19:58
1  
@user1007727 not willing to support, or not willing to support for free? –  chrisdew Jul 31 at 9:24

Maybe a bit strange here, but I found out recently that NFS handles this fine.

So you export a directory on one machine then mount it on the other and you just copy the files with basic utils like cp. (Some old/ancient utilities can have problem with sparse files.)

I found rsync especially inefficient in transferring sparse files.

share|improve this answer
rsync --ignore-existing --sparse ...

To create new files in sparse mode

Followed by

rsync --inplace ...

To update all existing files (including the previously created sparse ones) inplace.

share|improve this answer
    
Reverse it to have rsync --existing --inplace and then rsync --ignore-existing --sparse to have a sync speedup –  Mike Jan 16 at 15:54

Could replicating the whole file system be a solution? DRBD? http://www.drbd.org/

share|improve this answer
    
I don't think drbd is a good solution here, but the idea of rsyncing --inplace the whole fs, rather than the disk-image-files, is interesting. I'm not sure whether rsync allows this - I'll give it a try and report back... –  chrisdew Oct 21 '09 at 12:15

Rsync only transfers changes to each file and with --inplace should only rewrite the blocks that changed without recreating the file. From their features page.

rsync is a file transfer program for Unix systems. rsync uses the "rsync algorithm" which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand.

Using --inplace should work for you. This will show you progress, compress the transfer (at the default compression level), transfer the contents of the local storage directory recursively (that first trailing slash matters), make the changes to the files in place and use ssh for the transport.

rsync -v -z -r --inplace --progress -e ssh /path/to/local/storage/ \
user@remote.machine:/path/to/remote/storage/

I often use the -a flag as well which does a few more things. It's equivalent to -rlptgoD I'll leave the exact behavior for you to look up in the man page.

share|improve this answer
1  
The '-S' is for sparse files, not 'chops long lines'. From man page: -S, --sparse handle sparse files efficiently. I'll give this a try, thanks. –  chrisdew Sep 18 '09 at 6:56
    
Thanks I fixed that - I Was going off of something that was said in the link you gave. –  reconbot Sep 18 '09 at 14:40
    
No, unfortunately this does not solve the problem. It does sync the file, but it turns the sparse file at the far end into a non-sparse file. I am using ssh/rsync which comes with Ubuntu 9.04. –  chrisdew Sep 26 '09 at 6:09
    
My above comment was incorrect. The problem was that rsync creates non-sparse files on its first copy. The --inplace rsync does work correctly, provided that the destination file already exists and is as long (not big) as the origin file. I now have a solution, but it requires me to check whether each file already exists on the target server. If it does, I do an --inplace, if it doesn't, I use --sparse. This is not ideal, but it works. –  chrisdew Sep 26 '09 at 7:19

I'm not aware of such a utility, only of the system calls that can handle it, so if you write such a utility, it might be rather helpful.

what you actually can do is use qemu-img convert to copy the files, but it will only work if the destination FS supports sparse files

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.