Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm using Red Hat Linux (RHEL5) on a (VMWare) VM. I've written a daemon which should stay running all the time and automatically run on boot.

Last night the VM host had an unrecoverable hardware problem and the VM abruptly halted. When it came back, my daemon didn't start because the pidfile still existed.

Apparently this is called The Stale pidfile Syndrome but I'm not sure what's the best long-term approach for mitigating it. I'm thinking that the startup script in /etc/rc.d* should delete the pidfile before starting the daemon, but the service management script in /etc/init.d should remain the same so things like service mydaemon start doesn't clobber the pidfile.

/etc/rc.d/rc6.d just has a symlink to the script in /etc/init.d/, so how should I change how it behaves only on boot? I can make an additional script with higher precedence in the rc.d dirs, but it seems hacky. Someone also suggested adding logic like "if uptime is less than 1 minute, delete the pidfile" but that seems hacky too.

Any thoughts or solutions or best practices?

share|improve this question
add comment

3 Answers

Use daemontools and see Process Management.

share|improve this answer
    
Very good link, the kill -0 technique is very clever. –  coredump Feb 23 '11 at 19:34
    
But as the "The Stale pidfile Syndrome" I referenced noted, how do I know, say after a hard boot like today, that the process at that pid is my daemon and not something else that responds to kill -0? Also, daemontools' supervise addresses a different problem. My daemon is very stable and runs for months, but it didn't start this morning after the hard reboot because the pidfile from last run still existed. –  Nathan Feb 23 '11 at 19:58
    
@Nathan: pid=$(cat pidfile); process=$(ps -p $pid o cmd=); if [ "$process" != "me" ]; then echo "that's definitely not me"; else "maybe that's me"; fi and you can perform some additional sanity checks before deleting the pidfile and continuing. Do grep -l stale /etc/init.d/* | xargs less -p stale (press Alt-n to jump to successive occurrences of the search pattern) and see how some of the other daemons are doing it. Also, look at /lib/lsb/init-functions for some useful stuff (you can do the search above substituting the names of some of the function for "stale" to see how they're used). –  Dennis Williamson Feb 23 '11 at 20:23
add comment

The script is the same, the startup process just executes the 'start' action on the sysvinit scripts.

Why don't you check if the pid on the pid file is right, and if don't delete it and create a new one with the right pid?

EDIT: You can grep ps with the pidfile to see if the process still exists. Or do the other way around. Check RedHat initscripts, I am sure they have some helper functions to do that, like pidofproc.

share|improve this answer
    
Try kill -0 <pid> –  profy Feb 23 '11 at 19:40
add comment

Thank you for the hints @Dennis and @coredump.

I found out some additional information that helped me unravel the mystery.

  1. I wondered why every other daemon recovered fine. It turns out there is code in /etc/rc.d/rc.sysinit to clean up all pidfiles in /var/run and /var/lock at boot.

  2. I had configured my daemon to put its pidfile elsewhere because of trouble with SELinux preventing me from "using potentially mislabeled files".

So I haven't fixed it yet due to the SELinux issues, but the answer I think is "put your pidfile in /var/run or /var/lock and it will work next time"

share|improve this answer
    
... and as it turns out, it was not necessarily SELinux that was causing my problem, but too many hardcoded pidfile locations in the daemon and its init script. –  Nathan Feb 24 '11 at 21:51
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.