3

I'm running a bunch of cron jobs (set up using sudo crontab -e) on Ubuntu, and recently (though I don't know when to be sure) I'm suddenly seeing the same jobs appear to be run multiple times in the same minute. You don't need to see the full crontab in order to trust me that they are absolutely not listed twice. But for a flavor here is a snippet:

*/2 * * * * /usr/bin/wget --no-check-certificate 'https://myserver.net/someuri/pdm/33?embed_in_page=xyz'
* * * * * /usr/bin/wget --no-check-certificate 'https://myserver.net/someuri/pdm/77'
* * * * * /usr/bin/wget --no-check-certificate 'https://myserver.net/someuri/pdm/20?blah=blah'
* * * * * echo "`date` Running now" >> /home/somewhere/croncheck

I've also added that simple echo to the end to run every minute, and that never appears to run more than once.

So, for the echo I only see it happen once a minute. But intermittently, especially under load, the server seems to fire the wget URL requests multiple times in succession (I know by looking at the webserver log that they are coming from the same place at roughly the same time).

If I run ps -A |grep cron

I'll see a dozen or more entries looking like:

28055 ?        00:00:00 cron

They do not seem to go away.

If I run ps aux then I only see the one entry as I expected.

So, my guess is that there is a nasty interaction between wget and cron, and maybe something is failing in some way (even though the server, and interactive calls to the same webserver through a web browser seem largely unnaffected, just slowed by the unnecessary work). But the reality is that I don't know. I'm looking for any ideas you may have as to the cause, and possible solutions to the problem.

1
  • I'm testing flock on each entry with a range of lock files based on the category of job being run, to see if that can help me avoid the problem cascading into chaos. I will report back if anybody else is watching this...
    – phil_ayres
    Apr 18, 2013 at 22:51

2 Answers 2

1

Although I'm not entirely sure why this is happening, it seems to be that long running jobs inside cron cause an odd behaviour. In the original post have a series of jobs that are fired every five minutes and some one minute after each other. If the first job takes too long (10 minutes for example) my guess is that the forked process of the first job is hanging around for ten minutes and fires subsequent jobs in the list. But the primary cron process is also firing these jobs correctly, so duplicates occur and the whole issue cascades.

Enough of the guesswork. To fix, here is a snippet of my new crontab, using flock to block subsequent jobs that could cascade incorrectly.

# every 5 minutes
*/5 * * * * flock -w 2000 /tmp/cnsd.lockfile.pdm -c "/usr/bin/longrunningjob1"
# every 5 minutes
*/5 * * * * flock -w 2000 /tmp/cnsd.lockfile.pdm -c "/usr/bin/longrunningjob2"
# every 15 minutes past the hour
15 * * * * flock -w 2000 /tmp/cnsd.lockfile.pdm -c "/usr/bin/longrunningjob3"

At least this way, when the system is overloaded it doesn't cause a cascade of jobs that should never have been fired, instead giving everything a chance to recover. I did play with different classes of jobs using different flock lockfiles, but in my case I only had limited resources so queuing one behind another was the best way to keep the system running.

0

Verify that the web logs are showing the exact same entry multiple times. If so verify they are at the exact same time. Cron is NOT absolute and as was stated sometimes the system is taxed. Cron will "try" to run the jobs at the time stated but it's not absolute and sometimes a job will run later than expected and it may be at the time of the same job and then you see two entries.

Another issue maybe that a job takes longer than expected and is still running when the next iteration of it starts. I would start with elimination. Comment out all but one and let it run and monitor, then add the next, and the next etc to see what happens.

3
  • The multiple jobs are running with a second of each other when this happens. All are happening at 1 second past the top of the minute they are supposed to, so I don't think its a really scary lag that is causing this. Could it be that one job is holding up cron for more than a minute, and this forked cron that is waiting on the slow job goes on to trigger other jobs at the appropriate time, in addition to the main cron process?
    – phil_ayres
    Apr 18, 2013 at 20:54
  • hello phil_ayres, will you able to find it out why it is happening like that?? if yes please share your answer, we are also facing the same problem on centos 6 Nov 27, 2014 at 11:16
  • I ended up using flock on each entry. I'll take a moment to post my answer / workaround in a moment
    – phil_ayres
    Dec 15, 2014 at 16:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .