Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a Java server application running on Mac OS X.

Occasionally this application has become unresponsive, and we have resorted to killing it with kill -9. However, the process doesn't disappear; it still appears to ps, with parentheses around its name and a question mark in the STAT column:

$ ps u -p 776
USER       PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
camadmin   776   0.0  0.0        0      0   ??  ?E    5:12PM   0:00.00 (java)

What's more the port being used by the server is still bound:

$ netstat -na | grep 9902
tcp4       0      0  *.9902      *.*                    LISTEN

even though it's not visible to lsof:

$ sudo lsof -P -i tcp | grep 9902
$

With the port still bound, we can't restart the server application. Short of restarting the machine, what can be done to get the port to release and get this process really killed?

share|improve this question
1  
Could you please check if there is active semaphores with command "ipcs -s | grep camadmin". You can kill those active semaphores with below command ---- for i in $(ipcs -s | grep camadmin | awk '{print $2}'); do ipcrm sem $i; done –  Vaibhav Panmand Jun 11 at 4:42
    
ipcs -s | grep camadmin returns nothing; ipcs -s lists a handful of things with owner _postgres which I believe is not related to the problem Java process. –  Robert Tupelo-Schneck Jun 11 at 13:08
    
it's possible that this process is in "uninterrupted sleep". Execute this command to check stats "ps -o cmd,stat -p <pid>", if stat is D then it's issue. –  Vaibhav Panmand Jun 11 at 14:06
    
STAT is ?E... and question mark is not a documented value, nor have I found information about it by Googling. –  Robert Tupelo-Schneck Jun 11 at 17:17

2 Answers 2

Your process is waiting on something before it can exit.

Maybe a system call. Tryingto access a file which has been unmounted is a common example of this, or an unavailable network share.

Is there a parent or child process that also needs to be killed?

I'm not a MacOS user, but on linux ps wwauxf | less is useful to browse through the processes hierarchy to look for the parent and child processes. strace -p [pid] might tell you something useful about a current system call which has not yet returned.

--

EDIT: Is your process started by launchd? Seems a few people have problems with that (eg How to kill an "exiting" process on OS X (state = E)) and as suggested above, you probably need to kill the parent process, which in this case is launchd. It might be that's not better than a restart.

Presumably launchd keeps a connection to your app in order to restart it if it dies, but that's not all that helpful here.

Does a kill -15 work any better? ie before you get into the state you describe, not as a way out of it.

share|improve this answer
    
ps auxww -o ppid works on Mac. No child processes; parent process is the shell before killing, and 1 after killing. dtruss seems to be the Mac equivalent of strace but after killing the process it doesn't yield anything. kill -15 (the default) didn't stop the process at all. –  Robert Tupelo-Schneck Jun 17 at 3:49
    
The 'f' option to ps is the really useful one for browsing the process hierarchy, though you can work it out from the ppid field. –  mc0e Jun 17 at 17:12
    
It could be that you're still associated with the terminal of the parent shell or it's stdin, stdout or stderr. Did you try killing the shell? Take a look at web.archive.org/web/20120914180018/http://www.steve.org.uk/… for info on how to properly background a daemon process to help avoid this sort of problem –  mc0e Jun 17 at 17:24
    
Daemonizing a Java process is tricky. We typically use java ... & disown which has always been good enough in practice. And we do this a lot. It's only this process on this machine where we see the issue. –  Robert Tupelo-Schneck Jun 19 at 19:59

State E appears to mean that the process is exiting. But your process is basically a zombie that never exits. Everything online I found shows that there is little you can do other than reboot, e.g. How to kill an "exiting" process on OS X (state = E). If you do not want to reboot the OSX server due to other services impacted, you might consider (as workarounds) either:

  • Migrate just this java service to a separate OSX machine that you can reboot whenever needed with minimal impact

  • If you have enough RAM on this server, you could make a VM and run this service from inside the VM. If the process becomes unresponsive and won't be killed, you could just bounce the VM and not affect the host OS. Since it looks like Java, you don't necessarily have to make the guest VM OSX. Maybe try it in an Ubuntu or CentOS VM? You could download either of them as an appliance (OVA/VDI file) and have it up and running in VirtualBox in under 30 mins. Might not even have the zombie issue on a Linux VM. See http://virtualboximages.com/Free.VirtualBox.VDI.Downloads

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.