cron job occasionally not running

Question

I have a CentOS 6.6 server with the following packages installed:

crontabs-1.10-33.el6.noarch
cronie-1.4.4-12.el6.x86_64
cronie-anacron-1.4.4-12.el6.x86_64
kernel-2.6.32-504.3.3.el6.x86_64

Sometimes, one of the backup jobs that is scheduled to run daily simply does not run. The script is not even called according to /var/log/cron.log. Interesting to mention that other jobs scheduled to run exactly at the same time run without any issues.

I can't reproduce the problem and haven't spotted any patterns on it. If I do nothing, then the job runs correctly the next day as expected.

crond simply ignores just one of the multiple jobs that are supposed to run at a particular time. This only happens sporadically.

I read in a few other places people talking about adding an empty line at the end of the crontab file. The job that's occasionally failing to run is indeed at the last line of my crontab file. I could not find any confirmation this is a real or known bug.

# tail -2 /var/spool/cron/postgres
*  * * * * OTHERJOB
0 21 * * * /pg_backup.sh

This is all I have in my /var/log/cron.log

Mar 31 21:00:02 SERVERNAME [cron.info] CROND[19394]: (root) CMD (OTHERJOB)
Mar 31 21:00:02 SERVERNAME [cron.info] CROND[19418]: (postgres) CMD (/pg_backup.sh)
Mar 31 21:01:02 SERVERNAME [cron.info] CROND[20062]: (root) CMD (OTHERJOB)

Apr  1 21:00:02 SERVERNAME [cron.info] CROND[31349]: (root) CMD (OTHERJOB)
Apr  1 21:01:01 SERVERNAME [cron.info] CROND[32080]: (root) CMD (OTHERJOB)

See how OTHERJOB always run while on Apr 1 pg_backup.sh was not even executed.

I've already tried restarting crond but this keeps happening. This is affecting multiple servers with the same version of OS, kernel and cron RPMs.

There is a newer version of cronie (1.4.12), however upgrading it is not an option as we're already using the latest available version for Centos 6.6

I went through the changelog for all cronie versions after mine (1.4.4) and haven't seem any fix to this particular problem. Also checked all commit messages.

Answer 1

The original cron required each entry to end with a newline so yes sometimes you do need a blank line or something at the end.

   Although cron requires that each entry in a crontab end  in  a  newline
   character,  neither the crontab command nor the cron daemon will detect
   this error. Instead, the crontab will appear to load normally. However,
   the  command  will  never  run.  The best choice is to ensure that your
   crontab has a blank line at the end.

   4th Berkeley Distribution      29 December 1993               CRONTAB(1)

Some versions have it fixed or emit a warning for example Ubuntu Maverik (10.10) : crontab look at the diagnostics section at the bottom which states a warning will be written to syslog.

DIAGNOSTICS
       cron requires that each entry in a crontab end in a newline  character.
       If  the last entry in a crontab is missing a newline (ie, terminated by
       EOF), cron will consider the crontab (at  least  partially)  broken.  A
       warning will be written to syslog. 

Answer 2

This is the first answer that comes up with the search text cron error getpwname failed so I thought I would post the cause of my issue:

I was using /etc/crontab but had forgotten to put the user in front of the command.

i.e.,

*/5   *  *  *  * /bin/bash <filename>

Instead of

 */5   *  *  *  * root /bin/bash <filename>

It gave the same error, go figure.

Answer 3

we use sssd for remote authentication. crond has to check for available users ahead of running jobs and it does this every 60 seconds. sssd default client_idle_timeout is 60 seconds. so we had a race condition between sssd and crond

We only got to the bottom of this problem because on version 1.4.4-14 crond started being a bit more verbose about some errors.

* Thu Feb  5 12:00:00 2015 Tomáš Mráz <[email protected]> - 1.4.4-14
- add log message when getpwnam fails

After updating to that version we started seeing the error below at the same time a job would not run:

[cron.err] crond[8654]: (user) ERROR (getpwnam() failed): Broken pipe

that brought us to this: https://bugzilla.redhat.com/show_bug.cgi?id=1209600#c2

and finally to this: https://access.redhat.com/solutions/1125133

Issue: sssd_be terminated with SIGKILL due to getpwnam() returning EPIPE (ie. broken pipe) can cause crond to silently skip cron job entries.

The suggest solution on the link above was add the line below to /etc/sssd/sssd.conf:

client_idle_timeout = 75

The change above has fixed the problem for us and cron no longer skips jobs.