Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am running Bacula on a RedHat box. From time to time, the storage daemon bacula-sd stops working and becomes <defunct>.

[root@backup ~]# ps -ef | grep defunct | more
root      4801 29261  0 09:25 pts/5    00:00:00 grep defunct
root      5825     1  0 Oct18 ?        00:00:00 [bacula-sd] <defunct>

My question is, how can I kill this process? Its parent is 1, which is init, as far as I know, and I wouldn't want to kill the init process, would I?

'Normally' killing this process does not work:

[root@backup ~]# kill -0 5825
[root@backup ~]# kill -9 5825

Help is greatly appreciated!

Edit: running

[root@backup ~]# lsof -p 5825

produces the following output:

COMMAND    PID USER   FD   TYPE  DEVICE     SIZE    NODE NAME
bacula-sd 5825 root  cwd    DIR   253,0     4096 3801089 /root
bacula-sd 5825 root  rtd    DIR   253,0     4096       2 /
bacula-sd 5825 root  txt    REG   253,0  2110599  368004 /usr/local/sbin/bacula-sd
bacula-sd 5825 root  mem    REG   253,0    75284  389867 /usr/lib/libz.so.1.2.3
bacula-sd 5825 root  mem    REG   253,0    46680 3604521 /lib/libnss_files-2.5.so
bacula-sd 5825 root  mem    REG   253,0   936908  369115 /usr/lib/libstdc++.so.6.0.8
bacula-sd 5825 root  mem    REG   253,0   125736 3606807 /lib/ld-2.5.so
bacula-sd 5825 root  mem    REG   253,0  1602128 3606885 /lib/libc-2.5.so
bacula-sd 5825 root  mem    REG   253,0   208352 3606892 /lib/libm-2.5.so
bacula-sd 5825 root  mem    REG   253,0   125744 3606887 /lib/libpthread-2.5.so
bacula-sd 5825 root  mem    REG   253,0    25940 3604573 /lib/libacl.so.1.1.0
bacula-sd 5825 root  mem    REG   253,0    15972 3604535 /lib/libattr.so.1.1.0
bacula-sd 5825 root  mem    REG   253,0    46548 3606908 /lib/libgcc_s-4.1.2-20080102.so.1
bacula-sd 5825 root  mem    REG   253,0 56422480  366368 /usr/lib/locale/locale-archive
bacula-sd 5825 root    0r   CHR     1,3             1545 /dev/null
bacula-sd 5825 root    1r   CHR     1,3             1545 /dev/null
bacula-sd 5825 root    2r   CHR     1,3             1545 /dev/null
bacula-sd 5825 root    3u   CHR   9,128             6469 /dev/nst0
bacula-sd 5825 root    4u  IPv4 1023380              TCP backup:bacula-sd (LISTEN)
bacula-sd 5825 root    5u  IPv4 2693268              TCP backup:bacula-sd->backup:53957 (CLOSE_WAIT)
bacula-sd 5825 root    7u  IPv4 3248683              TCP backup:bacula-sd->backup:57629 (CLOSE_WAIT)
bacula-sd 5825 root    8u  IPv4 3250966              TCP backup:bacula-sd->backup:37650 (CLOSE_WAIT)
bacula-sd 5825 root    9u  IPv4 3253908              TCP backup:bacula-sd->backup:37671 (CLOSE_WAIT)
share|improve this question
add comment

7 Answers

up vote 6 down vote accepted

The only way you could remove the zombie/defunct process, would be to kill the parent. Since the parent is init (pid 1), that would also take down your system.

This pretty much leaves you with two options.

  • Manually modify the process table, eg. create a dummy process, link the defunct process as a child of the dummy, then kill them off. Quite dangerous, and you may have to manually clean up other process resources such as semaphores and file handles.
  • Reboot the system.

I'd go with the second.

share|improve this answer
2  
+1. However, there is no rush to do either, as long as more zombie processes aren't appearing, or your zombie process hasn't locked up 4G of your RAM. :) –  Kyle Smith Oct 20 '09 at 14:40
    
"Since the parent is init (pid 1), that would also take down your system" - You cannot kill init as it does not have a signal handler for SIGKILL. See man 2 kill. –  Andrew H Jul 10 '13 at 10:26
    
How do you do the first? –  skerit Feb 12 at 21:01
    
@AndrewH I'm not sure SIGKILL depends on a signal handler in the target process, but it's true that the typical kernel will ignore a SIGKILL to init. However, should you run out of cooler ways to trigger a kernel panic, I think you will find that on most Linux systems a SIGSEGV will do quite nicely. –  Roy Mar 4 at 12:26
add comment

You could try restarting init:

 # telinit u

Otherwise, I wouldn't worry too much. It's not running and it's not taking any resources and it's just there so the kernel can remember it.

share|improve this answer
    
well, i kind of have to worry. it's a production machine running backup (bacula) and voip (asterisk) services. as long as the defunct bacula-sd process is there, bacula cannot seem to access the tape drive ... –  andreas-h Oct 20 '09 at 8:37
    
It shouldn't have any files open. Run lsof -p 5825 and check. –  David Pashley Oct 20 '09 at 8:39
    
Well, there seems to be many things open ... see above. Any ideas what I can do? I've never used lsof ... –  andreas-h Oct 20 '09 at 8:53
1  
Yeah, your zombie has /dev/nst0 open. A system reboot is probably the best bet at this point. –  Kyle Smith Oct 20 '09 at 14:41
1  
Yep, reboot seems to be the prevailing answer. I always feel like I've failed when I have to reboot a server. :( –  David Pashley Oct 20 '09 at 17:22
add comment

You could check if the defunct process has any children still running and kill those.

share|improve this answer
    
i did check -- it doesn't have any children ... –  andreas-h Oct 20 '09 at 8:36
add comment

Seems like you've got an orphaned process. As far as I know the only way to kill these would be to reboot the box. I've had this happen on my ESX servers (which are linux under the hood) from time to time and a host reboot is the fix (from VMware support).

I'm a Windows guy so take that for what its worth.

share|improve this answer
    
unfortunately, rebooting is not a real option. it's a production machine also running voip services, so i cannot reboot it during office hours ... –  andreas-h Oct 20 '09 at 8:38
    
so, you could reboot it after office hours, right? –  warren Oct 20 '09 at 9:54
add comment

Check if there was a kernel panic,

# dmesg |tail

Check if the process is in "D" Unkillable sleep, where it's in kernel mode for some syscall which has not returned yet (either kernel oops, or some other reason) http://www.nabble.com/What-causes-an-unkillable-process--td20645581.html

share|improve this answer
    
annoying formatting –  asdmin Oct 20 '09 at 9:05
    
actually, there hasn't been any kernel panic. process is in 'Z' state -- a zombie... –  andreas-h Oct 20 '09 at 14:33
add comment

Let's keep the panic down, shall we? A "defunct" or "zombie" process is not a process. It is simply an entry in the process table, with a saved exit code. Thus, a zombie holds no resources, takes no CPU cycles, and uses no memory, since it is not a process. Don't get all weird and itchy trying to "kill" zombie processes. Just like their namesakes, they can't be killed, since they're already dead. But unlike the brain-eating kind, they harm absolutely no-one, and won't bite other processes.

Don't let zombie processes eat your brain. Just ignore them.

share|improve this answer
4  
Yes, that is the theory. Unfortunately it's not always true. A defunct process will sometimes hang on to system resources, like andreash has clearly documented. –  Roy Oct 20 '09 at 13:51
    
In his case, per the lsof output, the zombie process is eating the brains of /dev/nst0. He needs those brains to continue backup operations. –  Kyle Smith Oct 20 '09 at 14:42
add comment

If a zombie has init as its parent, then init has stopped working properly. One of the roles of init is to clean up zombies. If it doesn't do it, noone else will. So the only solution is to reboot. If init is broken, then a reboot may fail, so I'd shut down important services, sync the filesystem then hit the power button instead.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.