11

I'm using rsync to backup some data from one server to another. All works fine, but it might take longer to finish depending on how much data there is to transfer.

Is there any gauranteed way to ensure that an rsync command doesn't start before the previous one finished using a cronjob?

For example, every hour I run the rsync command, but its possible the transfer takes more than 1 hour to complete, so the next one would start before the previous one finishes.

  • If the job potentially takes longer than one hour to complete, and you are scheduling it closer than the duration, then you are mis-scheduling the job. Either figure out how to cut down on the time or increase the interval between jobs. If you are continually doing remote backups, you might want to consider a new disaster recovery plan. – vgoff Dec 28 '12 at 0:31
11

You can implement some kind of locking. This will print the number of rsync processes still running:

pgrep -cx rsync

And this will run the rsync only if no other rsync process exists:

pgrep -cx rsync || rsync ...

Using -x will prevent from accidentally matching unwanted names (for example "foobarsynchronizator" or "not_an_rsync_totally" - it works just like pgrep -c ^rsync$)

| improve this answer | |
  • In case it is not obvious. -c counts the number of processes that have the name rsync. If this is not 0 then the shell interprets the result as true(not false). The || "or lines" see the first item is true and do not bother to run the second item, rsync. – rob Nov 6 '17 at 10:34
13

You can use the flock command to help you do this e.g. In this case flock -n is probably what you want as it will cause an immediate failure of the command if it can't obtain the lock e.g.

30 * * * *  /usr/bin/flock -n /tmp/myRsyncJob.lck /path/to/your/rsyncScript 
| improve this answer | |
  • In general, predictable file names in /tmp are often dangerous due to race conditions and broad access to the /tmp directory. Is it safe in this case? – mc0e Sep 24 '13 at 2:43
  • In this case a predictable name is not only safe, it is necessary; that is what makes the lock (noun) lock (verb). In other words, the state of the lock is based specifically and solely on the existence of a file with a specific, predictable, name. If the file name was unpredictable, or if it changed dynamically, then flock would allow rsync to run over itself, defeating the purpose. However, you can ease your concerns, and imo be a bit more "correct", by putting the lock file somewhere such as /var/run instead. – Evan de la Cruz Jun 10 '16 at 17:43
3

If you're willing to consider other tools, you could also have a look at rdiff-backup. It uses librsync to do backups, and saves a configurable number of deltas/increments. It also locks so that only one rdiff-backup process can run at any given time.

| improve this answer | |
  • I use rdiff-backup aswell. But you need to be careful in this setup as rdiff-backup takes more time to complete than rsync alone. – mgabriel Dec 29 '12 at 9:26
3

Here is what I'd do. Create a wrapper script around the rsync to create a lock file.

script 1
- create lock file
- rsync
- remove lock file

script 2 (running later then script 1)
- check if lock file is there
    - if not run
    - if it is there wait 10 minutes in a loop. break out of lopp when the lock file is gone
- continue to run script
| improve this answer | |
  • 2
    Just be sure to also remove the lock file after a reboot, otherwise you could end up with a process that never runs again. – John Gardeniers Dec 28 '12 at 2:07
2

My Answer is somewhat same what Mike told.

In the script, you should put something like this:

  • create a lock file
  • Check for the existence of the lock file when you run it next time.

But there is one very important thing you should be doing. and that to implement a trap system.

So, with that, what you can do, is that even if somehow your script is killed or someone killed it, then you can trap that signal and remove the lock file, so that you don't have a stale lock file.

You can read how to implement that over here.

Just one small thing, you can't trap signal 9, I mean if someone do kill -9, you can't trap that as that signal directly interacts with the kernel and there is no way to trap that.

Also, as suggested by John, you need to remove the lock file everytime your system reboots, just to make sure that there is no stale file left.

That you can easily do by putting a small rm -f <FILE> command in /etc/rc.local

| improve this answer | |
1

Have a look at anacron (anachronistic cron) with the -s (serialize) switch. Serialize ensures that the command won't be called again if the previous one is still running.

| improve this answer | |
  • You may have misunderstood the question. – John Gardeniers Dec 29 '12 at 9:53
  • I don't think so. The question is "Is there any gauranteed way to ensure that an rsync command doesn't start before the previous one finished using a cronjob?" Anacron runs cronjobs with extra/different functionality. Serialize ensures that any command you call doesn't start until the previous one finishes. – tu-Reinstate Monica-dor duh Dec 31 '12 at 4:59
  • My apologies. It was me that misread the question. – John Gardeniers Dec 31 '12 at 6:01
0

Use hatools (http://www.fatalmind.com/software/hatools/) to lock rsync cron in waiting mode.

| improve this answer | |
  • Isn't this simply a wrapper around flock(1) – WarmWaffles Oct 19 '13 at 5:56
0

I couldn't get mgabriel's solution to work on OSX as the OSX version of pgrep does not seem to have the -c option (I assume this is for count). Instead I used the following:

[ $(pgrep ping | wc -l) -eq 0 ] && ping multiplay.co.uk || echo "Sorry, ping already in progress"

I used ping as an example command.

Hope this helps.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.