Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am using inotify to watch a directory and sync files between servers using rsync. Syncing works perfectly, and memory usage is mostly not an issue. However, recently a large number of files were added (350k) and this has impacted performance, specifically on CPU. Now when rsync runs, CPU usage spikes to 90%/100% and rsync takes long to complete, there are 650k files being watched/synced.

Is there any way to speed up rsync and only rsync the directory that has been changed? Or alternatively to set up multiple inotifywaits on separate directories. Script being used is below.

UPDATE: I have added the --update flag and usage seems mostly unchanged

#! /bin/bash

EVENTS="CREATE,DELETE,MODIFY,MOVED_FROM,MOVED_TO"

inotifywait -e "$EVENTS" -m -r --format '%:e %f' /var/www/ --exclude '/var/www/.*cache.*' | (
WAITING="";
while true; do
    LINE="";
    read -t 1 LINE;
    if test -z "$LINE"; then
        if test ! -z "$WAITING"; then
                echo "CHANGE";
                WAITING="";
                rsync --update -alvzr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/
        fi;
    else
        WAITING=1;
    fi;
done)
share|improve this question
add comment

3 Answers

up vote 0 down vote accepted

I ended up removing the compression option (z) and upping the WAITING var to 10 (seconds). This seems to have helped, rsync still spikes CPU load but it is shorter lived. Credit goes to an answer on unix stackexchange

share|improve this answer
add comment

You could also look into lsyncd which has more options. You can use lua scripting, so then say how many concurrent processes it needs to use and perhaps more things that might be of interest.

This uses rsync and ionotify too.

share|improve this answer
add comment

You may also consider to disable SSH compression and using a less demanding encryption algorithm, example:

rsync -e "ssh -c arcfour -x -o Compression=no" ....

One other measure can be to setup your ssh tunnel explicitly. You need to setup an Rsync daemon for that purpose though. Check the shell script at our FAQ. It should be usable for your case as well.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.