26

Is there a command, such as rsync, which can synchronise huge, sparse, files from one linux server to another?

It is very important that the destination file remains sparse. It may be longer (but not bigger) than the drive which contains it. Only changed blocks should be sent across the wire.

I have tried rsync, but got no joy. https://groups.google.com/forum/#!topic/mailing.unix.rsync/lPOScZgFE9M

If I write a programme to do this, am I just reinventing the wheel? http://www.finalcog.com/synchronise-block-devices

Thanks,

Chris.

2
  • 1
    rsync is hugely inefficient with huge files. Even with --inplace it will first read the whole file on the target host and THEN begin reading the file on the local host and transfer the differences (just run dstat or similar while running rsync and observe)
    – ndemou
    Aug 23, 2018 at 14:48
  • 1
    Even if the file is not sparse, rsync will struggle with it and the bdsync suggestion below is much better. for a 60 GB disk image with 100 MB of changes rsync took 2.5 hours and generated a 700 MB patch file. bdsync took under 20 minutes and the patch file was around 200 MB.
    – Origin
    Sep 12, 2020 at 12:57

9 Answers 9

24
rsync --ignore-existing --sparse ...

To create new files in sparse mode

Followed by

rsync --inplace ...

To update all existing files (including the previously created sparse ones) inplace.

6
  • 4
    Reverse it to have rsync --existing --inplace and then rsync --ignore-existing --sparse to have a sync speedup
    – Mike
    Jan 16, 2014 at 15:54
  • 3
    Can anyone explain Mikes comment and how this should speed up the sync?
    – Preexo
    Apr 17, 2015 at 4:23
  • I think Mike means first inplace change and then add new, so that the new ones do not need to be --inplace again due to the time difference between the first and second call. It is only true if you rsync directly off the datastore and VMs are running. Unless he means something else?
    – Yuan
    Nov 29, 2015 at 3:54
  • I agree with Yuan. Steves second command will rsync the new files again, you can safe that by using Mikes command sequence.
    – falstaff
    Apr 22, 2018 at 16:10
  • rsync is hugely inefficient with huge files. See my comment on the question.
    – ndemou
    Sep 26, 2019 at 19:40
7

To sync huge files or block-devices with low to moderate differences you can either do a plain copy or use bdsync, rsync is absolutely not fit for this particular case*.

bdsync worked for me, seems mature enough, it's history of bugs is encouraging (little issues, prompt resolution). In my tests it's speed was close to the theoretical maximum you could get** (that is you can sync in about the time you need to read the file). Finally it's open source and costs nothing.

bdsync reads the files from both hosts and exchanges check-sums to compare them and detect differences. All these at the same time. It finally creates a compressed patch file on the source host. Then you move that file to the destination host and run bdsync a second time to patch the destination file.

When using it over a rather fast link (e.g. 100Mbit Ethernet) and for files with small differences (as is most often the case on VM disks) it reduces the time to sync to the time you need to read the file. Over a slow link you need a bit more time because you have to copy the compressed changes from one host to the other (it seems you can save time using a nice trick but haven't tested). For files with many changes the time to write the patch file to disk should also be taken into account (and you need enough free space in both hosts to hold it).

Here's how I typically use bdsync. These commands are run on $LOCAL_HOST to "copy" $LOCAL_FILE to $REMOTE_FILE on $REMOTE_HOST. I use pigz (a faster gzip) to compress the changes, ssh to run bdsync on the remote host and rsync/ssh to copy the changes. Do note that I'm checking whether the patch has been applied successfully but I only print "Update successful" when it does. You may wish to do something more clevel in case of failure.

REMOTE_HOST=1.2.3.4
LOCAL_FILE=/path/to/source/file
REMOTE_FILE=/path/to/destination/file
PATCH=a_file_name
LOC_TMPDIR=/tmp/
REM_TMPDIR=/tmp/
# if you do use /tmp/ make sure it fits large patch files

# find changes and create a compressed patch file
bdsync "ssh $REMOTE_HOST bdsync --server" "$LOCAL_FILE" "$REMOTE_FILE" --diffsize=resize | pigz > "$LOC_TMPDIR/$PATCH"

# move patch file to remote host
rsync "$LOC_TMPDIR/$PATCH" $REMOTE_HOST:$REM_TMPDIR/$PATCH

# apply patch to remote file
(
    ssh -T $REMOTE_HOST  <<ENDSSH
    pigz -d < $REM_TMPDIR/$PATCH | bdsync --patch="$REMOTE_FILE" --diffsize=resize && echo "ALL-DONE"
    rm $REM_TMPDIR/$PATCH
ENDSSH
) | grep -q "ALL-DONE" && echo "Update succesful"  && rm "$LOC_TMPDIR/$PATCH"

# (optional) update remote file timestamp to match local file
MTIME=`stat "$LOCAL_$FILE" -c %Y`
ssh $REMOTE_HOST touch -c -d @"$MTIME_0" "$REMOTE_FILE" </dev/null

*: rsync is hugely inefficient with huge files. Even with --inplace it will first read the whole file on the destination host, AFTERWARDS begin reading the file on the source host and finally transfer the differences (just run dstat or similar while running rsync and observe). The result is that even for files with small differences it takes about double the time you need to read the file in order to sync it.

**: Under the assumption that you have no other way to tell what parts of the files have changed. LVM snapshots use bitmaps to record the changed blocks so they can be extremely faster (The readme of lvmsync has more info).

5

Rsync only transfers changes to each file and with --inplace should only rewrite the blocks that changed without recreating the file. From their features page.

rsync is a file transfer program for Unix systems. rsync uses the "rsync algorithm" which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand.

Using --inplace should work for you. This will show you progress, compress the transfer (at the default compression level), transfer the contents of the local storage directory recursively (that first trailing slash matters), make the changes to the files in place and use ssh for the transport.

rsync -v -z -r --inplace --progress -e ssh /path/to/local/storage/ \
[email protected]:/path/to/remote/storage/ 

I often use the -a flag as well which does a few more things. It's equivalent to -rlptgoD I'll leave the exact behavior for you to look up in the man page.

5
  • 1
    The '-S' is for sparse files, not 'chops long lines'. From man page: -S, --sparse handle sparse files efficiently. I'll give this a try, thanks.
    – fadedbee
    Sep 18, 2009 at 6:56
  • Thanks I fixed that - I Was going off of something that was said in the link you gave.
    – reconbot
    Sep 18, 2009 at 14:40
  • No, unfortunately this does not solve the problem. It does sync the file, but it turns the sparse file at the far end into a non-sparse file. I am using ssh/rsync which comes with Ubuntu 9.04.
    – fadedbee
    Sep 26, 2009 at 6:09
  • My above comment was incorrect. The problem was that rsync creates non-sparse files on its first copy. The --inplace rsync does work correctly, provided that the destination file already exists and is as long (not big) as the origin file. I now have a solution, but it requires me to check whether each file already exists on the target server. If it does, I do an --inplace, if it doesn't, I use --sparse. This is not ideal, but it works.
    – fadedbee
    Sep 26, 2009 at 7:19
  • rsync is hugely inefficient with huge files. See my comment on the question
    – ndemou
    Sep 26, 2019 at 19:41
4

Take a look at Zumastor Linux Storage Project it implements "snapshot" backup using binary "rsync" via the ddsnap tool.

From the man-page:

ddsnap provides block device replication given a block level snapshot facility capable of holding multiple simultaneous snapshots efficiently. ddsnap can generate a list of snapshot chunks that differ between two snapshots, then send that difference over the wire. On a downstream server, write the updated data to a snapshotted block device.

4

I ended up writing software to do this:

http://www.virtsync.com

This is commercial software costing $49 per physical server.

I can now replicate a 50GB sparse file (which has 3GB of content) in under 3 minutes across residential broadband.

chris@server:~$ time virtsync -v /var/lib/libvirt/images/vsws.img backup.barricane.com:/home/chris/
syncing /var/lib/libvirt/images/vsws.img to backup.barricane.com:/home/chris/vsws.img (dot = 1 GiB)
[........>.........................................]
done - 53687091200 bytes compared, 4096 bytes transferred.

real    2m47.201s
user    0m48.821s
sys     0m43.915s 
4
  • 4
    TBH, the stating timing at which you can sync is pretty meaningless because it obviously depends on the amount of data changed. What would be more accurate to say is that it takes your software 3 minutes to figure out which blocks have changed, and even that speed probably depends on your disk i/o and maybe CPU cycles available. May 3, 2013 at 7:43
  • 7
    You should disclose that this is commercial software costing $98 or more for network functionality.
    – Reid
    May 13, 2014 at 20:46
  • Thank you for pointing us at a software that worked well for you, which people can now consider and use, or not use as they need. Not thank you for the other two people for contribution nothing new. May 19, 2016 at 0:00
  • 1
    Link's gone....
    – bishop
    Dec 19, 2019 at 18:08
2

lvmsync does this.

Here's a usage transcript. It creates an LVM snapshot on the source, transfers the logical partition. You can transfer incremental updates of the changes since snapshot creation as often as you like.

3
  • I have tried it, but it doesn't work, and author is not willing to support Jun 4, 2013 at 19:58
  • 2
    @user1007727 not willing to support, or not willing to support for free?
    – fadedbee
    Jul 31, 2014 at 9:24
  • I used lvmsync in the past, it worked but it's not "prod grade" software imo. :-) May 19, 2016 at 0:04
1

Could replicating the whole file system be a solution? DRBD? http://www.drbd.org/

1
  • I don't think drbd is a good solution here, but the idea of rsyncing --inplace the whole fs, rather than the disk-image-files, is interesting. I'm not sure whether rsync allows this - I'll give it a try and report back...
    – fadedbee
    Oct 21, 2009 at 12:15
1

Maybe a bit strange here, but I found out recently that NFS handles this fine.

So you export a directory on one machine then mount it on the other and you just copy the files with basic utils like cp. (Some old/ancient utilities can have problem with sparse files.)

I found rsync especially inefficient in transferring sparse files.

0

I'm not aware of such a utility, only of the system calls that can handle it, so if you write such a utility, it might be rather helpful.

what you actually can do is use qemu-img convert to copy the files, but it will only work if the destination FS supports sparse files

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .