Is it possible to use rsync (or similar) in a way that lets me queue requests?
I have a script that processes photos (creates jpegs of various sizes) then uploads them to a server. Version 1 of the script did all the processing then rsynced everything at the end. This wasn't very efficient so I'm trying to parallelise it.
So I have a version 2 script that processes a single photo then rsyncs just the jpegs it has created in the background. The problem with that script is that it is spawning a new rsync process for each photo it has processed. It's fine if the processing takes the same time as an upload but if the upload takes longer I'll end up with dozens of rsync processes slowing to a crawl.
Is there a way to pass the upload requests to as single rsync process so it can queue them somehow? Or is there another application that can do similar?
I'm running this from a OS X client and uploading to a Linux server. My script that controls it all is written in Ruby.