13

I'm using rsync to backup some data from one server to another. All works fine, but it might take longer to finish depending on how much data there is to transfer.

Is there any gauranteed way to ensure that an rsync command doesn't start before the previous one finished using a cronjob?

For example, every hour I run the rsync command, but its possible the transfer takes more than 1 hour to complete, so the next one would start before the previous one finishes.

1
  • If the job potentially takes longer than one hour to complete, and you are scheduling it closer than the duration, then you are mis-scheduling the job. Either figure out how to cut down on the time or increase the interval between jobs. If you are continually doing remote backups, you might want to consider a new disaster recovery plan.
    – vgoff
    Dec 28, 2012 at 0:31

8 Answers 8

13

You can implement some kind of locking. This will print the number of rsync processes still running:

pgrep -cx rsync

And this will run the rsync only if no other rsync process exists:

pgrep -cx rsync || rsync ...

Using -x will prevent from accidentally matching unwanted names (for example "foobarsynchronizator" or "not_an_rsync_totally" - it works just like pgrep -c ^rsync$)

1
  • In case it is not obvious. -c counts the number of processes that have the name rsync. If this is not 0 then the shell interprets the result as true(not false). The || "or lines" see the first item is true and do not bother to run the second item, rsync.
    – rob
    Nov 6, 2017 at 10:34
17

You can use the flock command to help you do this e.g. In this case flock -n is probably what you want as it will cause an immediate failure of the command if it can't obtain the lock e.g.

30 * * * *  /usr/bin/flock -n /tmp/myRsyncJob.lck /path/to/your/rsyncScript 
2
  • In general, predictable file names in /tmp are often dangerous due to race conditions and broad access to the /tmp directory. Is it safe in this case?
    – mc0e
    Sep 24, 2013 at 2:43
  • 1
    In this case a predictable name is not only safe, it is necessary; that is what makes the lock (noun) lock (verb). In other words, the state of the lock is based specifically and solely on the existence of a file with a specific, predictable, name. If the file name was unpredictable, or if it changed dynamically, then flock would allow rsync to run over itself, defeating the purpose. However, you can ease your concerns, and imo be a bit more "correct", by putting the lock file somewhere such as /var/run instead. Jun 10, 2016 at 17:43
3

If you're willing to consider other tools, you could also have a look at rdiff-backup. It uses librsync to do backups, and saves a configurable number of deltas/increments. It also locks so that only one rdiff-backup process can run at any given time.

1
  • I use rdiff-backup aswell. But you need to be careful in this setup as rdiff-backup takes more time to complete than rsync alone.
    – mgabriel
    Dec 29, 2012 at 9:26
3

Here is what I'd do. Create a wrapper script around the rsync to create a lock file.

script 1
- create lock file
- rsync
- remove lock file

script 2 (running later then script 1)
- check if lock file is there
    - if not run
    - if it is there wait 10 minutes in a loop. break out of lopp when the lock file is gone
- continue to run script
1
  • 2
    Just be sure to also remove the lock file after a reboot, otherwise you could end up with a process that never runs again. Dec 28, 2012 at 2:07
2

My Answer is somewhat same what Mike told.

In the script, you should put something like this:

  • create a lock file
  • Check for the existence of the lock file when you run it next time.

But there is one very important thing you should be doing. and that to implement a trap system.

So, with that, what you can do, is that even if somehow your script is killed or someone killed it, then you can trap that signal and remove the lock file, so that you don't have a stale lock file.

You can read how to implement that over here.

Just one small thing, you can't trap signal 9, I mean if someone do kill -9, you can't trap that as that signal directly interacts with the kernel and there is no way to trap that.

Also, as suggested by John, you need to remove the lock file everytime your system reboots, just to make sure that there is no stale file left.

That you can easily do by putting a small rm -f <FILE> command in /etc/rc.local

1

Have a look at anacron (anachronistic cron) with the -s (serialize) switch. Serialize ensures that the command won't be called again if the previous one is still running.

4
  • You may have misunderstood the question. Dec 29, 2012 at 9:53
  • I don't think so. The question is "Is there any gauranteed way to ensure that an rsync command doesn't start before the previous one finished using a cronjob?" Anacron runs cronjobs with extra/different functionality. Serialize ensures that any command you call doesn't start until the previous one finishes. Dec 31, 2012 at 4:59
  • My apologies. It was me that misread the question. Dec 31, 2012 at 6:01
  • @tu-ReinstateMonica-dorduh, could you elaborate your answer a little bit? How would this setup look like?
    – moi
    Jun 1, 2022 at 9:45
0

Use hatools (http://www.fatalmind.com/software/hatools/) to lock rsync cron in waiting mode.

1
  • Isn't this simply a wrapper around flock(1) Oct 19, 2013 at 5:56
0

I couldn't get mgabriel's solution to work on OSX as the OSX version of pgrep does not seem to have the -c option (I assume this is for count). Instead I used the following:

[ $(pgrep ping | wc -l) -eq 0 ] && ping multiplay.co.uk || echo "Sorry, ping already in progress"

I used ping as an example command.

Hope this helps.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .