Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I want to compare two large binary files which are stored on different Linux machines with limited bandwidth between them and then back up only the blocks which differ – on the command line. To simplify the task, we can assume the files are not going to change during the comparison process, and the files are the same size.

This is almost like what I believe rsync does, only I don't want to modify the target file – I want to keep diffs which I can apply to the base image so I can recreate a copy at various points (ie when the diffs are taken).

I'm also aware of xdelta, but that appears to only compare files on the same machine.

The "process" I roughly envisage (hopefully all done by a script/program) might be -

  1. (On each machine) produce a list of hashes for each block.
  2. Compare the 2 sets of hashes.
  3. Produce a file which pulls only the changed blocks in the source in such a way as they can be "merged" back with the target file.

Is anyone aware of a program, script or elegant method to do this without me having to cut code?

share|improve this question
    
Server Fault is for questions in a professional workplace setting only. Is that the case for you? –  slhck Mar 11 '13 at 19:53
    
Yes, definitely. (I believe that looking at dedup type solutions puts this question fairly in the realm of server administration - and I am a system administrator by trade). The answer to this question could be particularly useful to system administrators running VM infrastructure - as I am doing. –  davidgo Mar 11 '13 at 19:58
add comment

migrated from superuser.com Mar 11 '13 at 20:26

This question came from our site for computer enthusiasts and power users.

2 Answers

up vote 1 down vote accepted

I recommend examining rsync's batch mode. The --only-write-batch option in particular seems to accomplish your goal.

share|improve this answer
    
Thank you for this - I was unaware of that option and it can probably be shoe-horned into what I'm trying to achive. –  davidgo Mar 11 '13 at 21:44
add comment

Efficient comparison usually requires comparing files on the same machine because if you want to account for offsets, you need to do lots of range-checking in the process. For example, if I added one character in a text file, everything after that character might need to be transmitted as "new" in a simple check.

One such example of a very simple checking method is implemented by Bit Torrent. Each block has a checksum, and each file is made of a series of blocks. Blocks might span the end / start of two or more files, but block verification will also check those spans. Only the blocks that don't match the description of the file will be changed. Thus is you start a client with some of the files accurately written and some differing (by corruption or change), only the necessary blocks to fix the difference will be transferred. Block size is configurable per torrent description file by powers of 2, and there are tons of open source clients you can grab this code from.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.