Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a java application running on a CentOS 5.4 server.

Randomly, and seemingly without cause, these processes will die, not through the application exiting, or due to my killing it, but due to something that seems to kill without leaving a trace.

I log all output from the application, as well as sending stderr and stdout to a logfile, and none of these output logs contain anything that would indicate why these processes have died.

My first instinct was the kernel-level OOM killer, but the system is never low on memory, and usually has somewhere between 1GB and 3GB of memory free at any given point in time.

My question is: does anyone know what might be causing it, or does anyone have any ideas where I could start looking?

Thanks.

share|improve this question

2 Answers 2

The processes may be getting some signal that causes them to exit, possibly due to bad hardware (e.g., bad memory can cause segfaults). If you see nothing the syslog/dmesg try enabling core dumps. You can then run gdb on them to get an idea what is happening.

share|improve this answer
    
I've built an identical setup on another of our servers and they're still being killed, so it's not faulty memory, and I'm still not getting any meaningful output. –  Martin Feb 9 '11 at 9:09
    
It seems pressing return submits the comment. Interesting. I was going to ask - when you say "enabling core dumps" do you mean from Java or from the operating system? It seems Java has no warning when it's about to be killed. –  Martin Feb 9 '11 at 9:10

Starting my java app with

strace -tt -o strace.out <java command>

Showed that it was being sent a SIGHUP command which was killing it. Turns out that, although I was setting the process to run in the background (by appending & to the command), the command is still tied to whichever thread created it.

This meant that, for commands executed by apache, they would be killed whenever the worker thread that created them was recycled, and, for commands executed manually, they were killed whenever I logged out.

I remedied this by simply pre-pending nohup to the command, i.e.

nohup java -jar /path/to/my/java.jar arguments &
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.