Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

As of about a week ago, my cron daemon refuses to stay running. I'm using Debian 6 x64 on an OpenVZ virtual machine. Running something like pgrep cron shows that the daemon isn't running. I start the service with service cron start or /etc/init.d/cron start and it launches, but it disappears from the running process list after a few minutes (varying anywhere between 1 - 30 minutes before the process is killed again).

Using strace -f service cron start, I can see that the process is being killed for some reason:

nanosleep({60, 0},  <unfinished ...>
+++ killed by SIGKILL +++

There's nothing relevant in /var/log/syslog, /var/log/messages, /var/log/auth.log, or /var/log/kern.log to explain why the the process is dying. The system has at least 800 MB of free memory, and cat /proc/loadavg returns 0.22 0.13 0.04 so resources shouldn't be the issue. With cron running, free -m reports:

             total       used       free     shared    buffers     cached
Mem:          1024        211        812          0          0          0
-/+ buffers/cache:        211        812
Swap:            0          0          0

I also tried removing and reinstalling the cron package using apt-get.

Update: I initially thought the problem was a resource issues. I erased my entire VPS and started from a fresh Debian image. There is now nothing else running on the system, but even from a clean install my cron daemon is still being killed at random.

What else should I check? How do I find out what's killing my crond?

share|improve this question
    
For completeness, what's the output of free -m (concerned that your 500MB free is happening after it's killed and you have no swap) and is this a physical or virtual machine? If virtual, what hypervisor? –  Jay Jun 29 '12 at 0:09
    
Ah, good point. It's an OpenVZ VM. I updated the question. –  user113215 Jun 29 '12 at 0:23
    
I'm having the exactly same issue after reinstalling a VPS under OpenVZ with vePortal, Debian 5 x86_64 and DirectAdmin. In my case I have no control of the host because it's on a VPS hosting provider. The counters at /proc/user_beancounters doesn't reset after virtual host reboots. –  Martín Claro Jul 13 '12 at 3:57

2 Answers 2

Look at /proc/user_beancounters, more specifically, at the failcnt column.

For all the non-zero entries, you'll need to increase the barrier/limit accordingly, it's probably just OpenVZ killing your processes for hitting them.

Here is a description of each column: http://wiki.openvz.org/Proc/user_beancounters

For accountable parameters, the field held shows the current counter for the container (resource “usage”), and the field maxheld shows the counter's maximum for the last accounting period. The accounting period is usually the lifetime of the container.

The field failcnt shows the number of refused “resource allocations” for the whole lifetime of the process group.

The barrier and limit fields are resource control settings. For some parameters, only one of them may be used, for some parameters — both. These fields may specify limits or guarantees, and the exact meaning of them is parameter-specific. Description of each parameter in UBC parameters contains information about the difference between the barrier and the limit for the parameter.

share|improve this answer
    
Thanks! This is very useful. Most things look in line to me, except I see privvmpages has failed about 4 million times since the start of the accounting period. :-) I stopped and restarted my VM and the failcounts didn't reset -- so I don't know how long my accounting period is (since the VM was created?). However, I'm not sure if this is the cause of my cron mystery. Shouldn't I see one of the failcounts increase after each death of crond? –  user113215 Jun 29 '12 at 1:02
    
If it answers your question, please click the green hollow tick on the left of my answer to accept it :-) –  Jay Jun 29 '12 at 1:04
    
Sorry, I forgot I was in a comment and was premature in my use of the Enter key. :-) –  user113215 Jun 29 '12 at 1:06
    
Heh, fair enough :-) In my experience failcnts definitely reset after a VM restarts, so I'm not sure why yours didn't. failcnt measures the number of times the allocations were refused (e.g. if you write a C program to malloc() memory, if that calls fails, then only failcnt will increment). OpenVZ may have killed your application because it needed to maintain a different one (e.g. it may judge that killing crond would allow all other applications to function correctly, so it's most optimal to kill it). –  Jay Jun 29 '12 at 1:14
    
Ah, okay, so it may have sent the kill signal to crond in order to avoid "failure" (because killing it would allow another process's memory allocation request to proceed)? Good thing OpenVZ is there to save the day ;-) –  user113215 Jun 29 '12 at 1:22
up vote 0 down vote accepted

After much trial and error, I have stumbled on a workaround. For some reason, cron is only being killed (presumably by the host) if it's running in daemon mode. If launched with cron -f, the process persists. So, I created a simple script to launch it in the foreground (and continually relaunch it in the unlikely event that it does get killed):

#!/bin/bash -u
while [ 1 ]; do
        logger -i -t cronrestart -p cron.warn "Launching cron daemon"
        cron -f
        logger -i -t cronrestart -p cron.warn "Cron daemon killed"
done
logger -i -t cronrestart -p cron.warn "Quitting"

Then I start this launcher script via nohup cronrestart >/dev/null & so it runs in the background. Cron has been running for a week so far this way without being killed.

I suppose the next step would be to try making the launcher script start itself in the background to better simulate the daemon mode of cron.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.