I'm running a bunch of cron jobs (set up using sudo crontab -e
) on Ubuntu, and recently (though I don't know when to be sure) I'm suddenly seeing the same jobs appear to be run multiple times in the same minute. You don't need to see the full crontab in order to trust me that they are absolutely not listed twice. But for a flavor here is a snippet:
*/2 * * * * /usr/bin/wget --no-check-certificate 'https://myserver.net/someuri/pdm/33?embed_in_page=xyz'
* * * * * /usr/bin/wget --no-check-certificate 'https://myserver.net/someuri/pdm/77'
* * * * * /usr/bin/wget --no-check-certificate 'https://myserver.net/someuri/pdm/20?blah=blah'
* * * * * echo "`date` Running now" >> /home/somewhere/croncheck
I've also added that simple echo to the end to run every minute, and that never appears to run more than once.
So, for the echo I only see it happen once a minute. But intermittently, especially under load, the server seems to fire the wget URL requests multiple times in succession (I know by looking at the webserver log that they are coming from the same place at roughly the same time).
If I run ps -A |grep cron
I'll see a dozen or more entries looking like:
28055 ? 00:00:00 cron
They do not seem to go away.
If I run ps aux
then I only see the one entry as I expected.
So, my guess is that there is a nasty interaction between wget and cron, and maybe something is failing in some way (even though the server, and interactive calls to the same webserver through a web browser seem largely unnaffected, just slowed by the unnecessary work). But the reality is that I don't know. I'm looking for any ideas you may have as to the cause, and possible solutions to the problem.
flock
on each entry with a range of lock files based on the category of job being run, to see if that can help me avoid the problem cascading into chaos. I will report back if anybody else is watching this...