Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a filesystem that is changed on two servers and also needs to be replicated to Amazon S3.

Until recently, syncing the filesystem between the two servers using Unison, and then copying to S3 with s3sync.rb has been a fine solution.

Now that the filesystem is nearly 50GB, s3sync.rb has become the bottleneck, as it needs to check each file for freshness (we use the --no-md5 flag).

So I now have a script that expects a list of files, and it would update these and only these using s3cmd.rb

I'd expected that I could use the unison.log file to get a canonical list of files to pass, but the format of it varies depending on the operation that occurred to a file (new file, copy from local alternative, rename etc).

Is unison able to generate a log or list of files that have been changed other than that left in unison.log?

At the moment this is how I'm extracting the list of file from the unison.log (I'm deliberately ignoring deletes)

# Ignore deletes and get the list of new & changed files
grep -v '\[END\] Deleting ' /tmp/unison.log | grep '\[END\]' $unisonlog | sed -re 's/\[END\] (Copying|Updating file) //' > /tmp/changed-files.log

# Files that unison lists as shortcuts are harder as it doesn't always prefix them with their full path
# so before adding them to the log, find the files in the relevant directory
grep 'Shortcut: copying ' /tmp/unison.log | sed -re 's/Shortcut: copying (.*)+ from local file.*/\1/' | while read file
do
  echo "Having to look for $file in source directory"
  find /ebs/src -wholename "*$file" >> /tmp/changed-files.log
done
share|improve this question
    
It has been a while since I've used unison, maybe the dumparchives option might be what you're looking for? –  NickW Feb 27 '13 at 17:29
    
Thanks @NickW but the unison.dump file looks even harder to parse than the unison.log. Looks like I'll stick with my bash script for the moment –  jaygooby Mar 5 '13 at 12:12
    
Yeah, the logs tend to be really verbose.. –  NickW Mar 5 '13 at 12:18

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.