0

I have a RHEL 6.4 server with 64GB of ram running a java app that loads 60+ war files into tomcat. For some reason, every time it loads every user gets kicked off. This includes a directly connected monitor on the physical server.

The server is not generating any type of crash log but it is not possible to log in. There still appears to be disk activity based on the disk lights.

I'm trying to determine if this application is the cause or merely just a symptom of some underlying hardware/software issue. Anecdotally, I can say it only happens with this app. Even running memtest86 showed no memory errors and a stress test getting the load to 160 with 100% memory usage did not crash.

My two questions are these:

  1. Is there a process to monitor that would terminate any and all connections (remember this includes the VGA port on the physical server)
  2. Are there any tools that would allow me to "wrap" this application and see exactly what point the terminations occur?

EDIT: Did not find out why connections were being severed. However, I moved the physical drives to another server (same hardware) and the issue does not happen on the new box. Leads me to believe there is hardware issue somewhere.

  • Which user is the java process running as? – davey Jan 16 '14 at 19:42
  • The process is running as root – user739866 Jan 16 '14 at 19:55
  • Is it possible to run it as an unprivileged user? That would at least isolate the impact of whatever causes the other users to be logged off. Others will probably advise on the wisdom of running such a thing as root. – davey Jan 16 '14 at 20:02
  • Are you sure the system isn't running out of memory? You could find this logged in /var/log/messages. – sciurus Jan 16 '14 at 20:15
  • There is nothing in the /var/log/messages related to memory. In addition, I have run a shell script running free every five seconds that shows the following: Mem: 32879072 8615180 24263892 0 128760 3731152 -/+ buffers/cache: 4755268 28123804 Swap: 67108856 0 67108856 total used free shared buffers cached Mem: 32879072 8582016 24297056 0 128820 3664440 -/+ buffers/cache: 4788756 28090316 – user739866 Jan 16 '14 at 20:58
2

Try attaching a strace to one of the login shells, and then sending the trace to a file, and throw it in the background. That was it doesn't get caught hopefully.

something like:

PID=''
FILE_OUT=''
strace -D -t -f -v -p ${PID} -o ${FILE_OUT}

This way you can hopefully see what signal/command it's getting that's causing it to shutdown

Now that I think about it, I would throw an strace on one of the logins, but also the java app. You can much up the timestamps to see what happened right before/after in both threads.

  • Just an fyi to anyone interested, it turns out that the vendor application had a bug that was using up all resources. Since the app could only be run as root, this caused everything to become non-responsive. – user739866 Jan 23 '14 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.