After kill -9 process won't disappear and port is still bound

Question

We have a Java server application running on Mac OS X.

Occasionally this application has become unresponsive, and we have resorted to killing it with kill -9. However, the process doesn't disappear; it still appears to ps, with parentheses around its name and a question mark in the STAT column:

$ ps u -p 776
USER       PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
camadmin   776   0.0  0.0        0      0   ??  ?E    5:12PM   0:00.00 (java)

What's more the port being used by the server is still bound:

$ netstat -na | grep 9902
tcp4       0      0  *.9902      *.*                    LISTEN

even though it's not visible to lsof:

$ sudo lsof -P -i tcp | grep 9902
$

With the port still bound, we can't restart the server application. Short of restarting the machine, what can be done to get the port to release and get this process really killed?

Answer 1

Your process is waiting on something before it can exit.

Maybe a system call. Tryingto access a file which has been unmounted is a common example of this, or an unavailable network share.

Is there a parent or child process that also needs to be killed?

I'm not a MacOS user, but on linux ps wwauxf | less is useful to browse through the processes hierarchy to look for the parent and child processes. strace -p [pid] might tell you something useful about a current system call which has not yet returned.

--

EDIT: Is your process started by launchd? Seems a few people have problems with that (eg How to kill an "exiting" process on OS X (state = E)) and as suggested above, you probably need to kill the parent process, which in this case is launchd. It might be that's not better than a restart.

Presumably launchd keeps a connection to your app in order to restart it if it dies, but that's not all that helpful here.

Does a kill -15 work any better? ie before you get into the state you describe, not as a way out of it.

Answer 2

State E appears to mean that the process is exiting. But your process is basically a zombie that never exits. Everything online I found shows that there is little you can do other than reboot, e.g. How to kill an "exiting" process on OS X (state = E). If you do not want to reboot the OSX server due to other services impacted, you might consider (as workarounds) either:

• Migrate just this java service to a separate OSX machine that you can reboot whenever needed with minimal impact

• If you have enough RAM on this server, you could make a VM and run this service from inside the VM. If the process becomes unresponsive and won't be killed, you could just bounce the VM and not affect the host OS. Since it looks like Java, you don't necessarily have to make the guest VM OSX. Maybe try it in an Ubuntu or CentOS VM? You could download either of them as an appliance (OVA/VDI file) and have it up and running in VirtualBox in under 30 mins. Might not even have the zombie issue on a Linux VM. See http://virtualboximages.com/Free.VirtualBox.VDI.Downloads