Namely, I need to rotate backup folders. I have many machines on a schedule to rsync to a single backup machine. Although I schedule the backups to begin in the late evening, and schedule a folder rotation (to make folder day0 become day1, starting with the oldest) in the late morning the next day (providing, say, 10 hours for the backups to complete), I'd like to be able to assure all backups have completed before allowing the rotation to begin rather than making an assumption (because, if I rotate the folders while a backup is in progress, my backup is inaccurate).
This would be trivial for a single machine, but for several, I'm hoping someone knows the best method... I can think of a few but would prefer not to have to 'experiment' on running systems:
Have each backup create a completion stamp, and run the rotation script every few minutes after a certain time, checking that it hasn't already run successfully and that all stamps are current (older than last rotation script stamp)?
Have each backup mv their previous rsync to an in-progress folder, rsync, then mv back to day0 so the rotation just skips that backup if it's not complete?
Just live with potentially inaccurate backups?