1

We've got two Red Hat servers which "hangs" at regular intervals. The systems become unresponsive for 20 - 30 seconds, and then return back to "normal". The systems are not virtualized, and runs on their own dedicated hardware.

When I say that the system becomes unresponsive, I know this because a script runs on the systems which - for every 5 seconds - (1) prints out the current time, (2) executes a wget towards a website on the system, (3) again prints out the end time and (4) sleeps for 5 seconds. We see from the log that the "sleep for 5 seconds" sometimes takes 20 - 30 seconds, and our Nagios monitering confirms that the system can't be reached for the same period of time.

Basically, the systems are clean installs of Red Hat 6.5 running Tomcat6. Furthermore, an NFS share against a GlusterFS storage is mounted. There's a bit of activity on that drive. Furthermore, the Tomcat6 webapps are somewhat memory intensive, requiring in the range of 7 - 15GB RAM.

We've been trying to figure out what's causing this for a couple of weeks now, and we're sort of come to the point where we're out of ideas. The Tomcat applications have been scrutinized, we've tried all sort of options on the NFS mounts, we've experimented with SELinux, and the list goes on. However, the problem persists.

Does anyone have any ideas about what might cause an entire system to hang?

  • 2
    What do your logs say? Are you seeing a high degree of iowait? – EEAA Dec 18 '14 at 14:52
  • 1
    You have two separate physical stand alone servers that are both doing this? Are they doing it at the same time? Is there a common denominator between the two (i.e. SAN, NAS, DNS, LDAP)? – Safado Dec 18 '14 at 15:29
  • @Safado Yes, both servers hang. They do not hang at the same time. They use local discs, but share dns and are connected to the same nas (glusterfs share via nfs). – sbrattla Dec 18 '14 at 17:41
  • @EEAA: are you thinking of any logs in particular? Can't really say I've come across too much in the logs. On the other hand; I dont't know what I'm looking at? – sbrattla Dec 18 '14 at 18:36
0

The system got hang >> There might be a reason; Resource usage like RAM, CPU, HDD etc- + some condition which leads to this.

  1. Collect maximum performance data & time slots of issue occurrence. use Nagios & SAR etc - for this, if time is same then create a shell script to capture the running process & all process resource usage it will help you to identify the hammering process.

Possiblity :- NFS issue - Try to use soft mount option instead of Hardmount.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.