I am using inotify to watch a directory and sync files between servers using rsync. Syncing works perfectly, and memory usage is mostly not an issue. However, recently a large number of files were added (350k) and this has impacted performance, specifically on CPU. Now when rsync runs, CPU usage spikes to 90%/100% and rsync takes long to complete, there are 650k files being watched/synced.
Is there any way to speed up rsync and only rsync the directory that has been changed? Or alternatively to set up multiple inotifywaits on separate directories. Script being used is below.
UPDATE: I have added the --update flag and usage seems mostly unchanged
#! /bin/bash EVENTS="CREATE,DELETE,MODIFY,MOVED_FROM,MOVED_TO" inotifywait -e "$EVENTS" -m -r --format '%:e %f' /var/www/ --exclude '/var/www/.*cache.*' | ( WAITING=""; while true; do LINE=""; read -t 1 LINE; if test -z "$LINE"; then if test ! -z "$WAITING"; then echo "CHANGE"; WAITING=""; rsync --update -alvzr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/ fi; else WAITING=1; fi; done)