Using rsync as a queue

Question

Is it possible to use rsync (or similar) in a way that lets me queue requests?

I have a script that processes photos (creates jpegs of various sizes) then uploads them to a server. Version 1 of the script did all the processing then rsynced everything at the end. This wasn't very efficient so I'm trying to parallelise it.

So I have a version 2 script that processes a single photo then rsyncs just the jpegs it has created in the background. The problem with that script is that it is spawning a new rsync process for each photo it has processed. It's fine if the processing takes the same time as an upload but if the upload takes longer I'll end up with dozens of rsync processes slowing to a crawl.

Is there a way to pass the upload requests to as single rsync process so it can queue them somehow? Or is there another application that can do similar?

I'm running this from a OS X client and uploading to a Linux server. My script that controls it all is written in Ruby.

Thanks all.

Answer 1

Use two threads:

• thread A just processes your photos. send a signal when finished.
• thread B does rsync in a loop, maybe waiting a few seconds (or a few photos) between finishing one and starting the next. Don't start two rsync's in parallel. Finish the loop after the first rsync that started after the signal from A was received.

This way you're getting advantage of the fact that rsync transfers only changes. All but the last iterations will send at least one partial file; but it's no permanent damage, because it will be completed on the next iteration.

Be sure that the last rsync iteration is started after the last photo was processed.

Answer 2

I'd be tempted to split the script up such that the jpeg-generating code is separate from the rsync-calling code. That way you can then schedule/batch up the rsyncing as you see fit. You might have rsync run via cron if periodic uploads are good enough or, if you're happy with OSX-specific code, use launchd's "watched folder" feature to trigger an rsync as and when a new jpeg is ready.

With either of those approaches you'll still have to make sure that you don't have too many rsyncs running at once. A simple solution would be to wrap the rsync call in code that sets and checks for the presence of a lock file (or equivalent).

Hope that helps