7

Assume I have 1 million files, the file list would be something like 20MB. If I add just one file, inotify will tell rsync to transfer the file list and the new file. My network is not good and I might have to limit the bandwidth (--bwlimit).

Multiple files might be added at once and, given the network transfer speed, there may be multiple rsyncs running at once.

Is it worth it? Is there a better way?

12

Your case sounds exactly what lsyncd was made for.
It watches the directory with inotify and spawns an rsync with only the changed file(s).
It aggregates changes for a few seconds.
So when you make 5 changes in one second it will only spawn one new sync process to the destination server(s).

3
  • Yeah, I've tried lsyncd and it suits me. – seanlook May 13 '15 at 1:49
  • @seanlook how is performance with 1 million files? – gcb Mar 2 '16 at 8:22
  • 2
    @gcb I got a setup with a pretty decent server and >1 million folders(not that lsyncd watches folders, not files!) and it took ~2 days to even do the full startup sync, the low single-threaded inotify_add_watch performance seeming to be the limiting factor. – Izzy Jan 15 '17 at 22:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.