1

We've got two Red Hat servers which "hangs" at regular intervals. The systems become unresponsive for 20 - 30 seconds, and then return back to "normal". The systems are not virtualized, and runs on their own dedicated hardware.

When I say that the system becomes unresponsive, I know this because a script runs on the systems which - for every 5 seconds - (1) prints out the current time, (2) executes a wget towards a website on the system, (3) again prints out the end time and (4) sleeps for 5 seconds. We see from the log that the "sleep for 5 seconds" sometimes takes 20 - 30 seconds, and our Nagios monitering confirms that the system can't be reached for the same period of time.

Basically, the systems are clean installs of Red Hat 6.5 running Tomcat6. Furthermore, an NFS share against a GlusterFS storage is mounted. There's a bit of activity on that drive. Furthermore, the Tomcat6 webapps are somewhat memory intensive, requiring in the range of 7 - 15GB RAM.

We've been trying to figure out what's causing this for a couple of weeks now, and we're sort of come to the point where we're out of ideas. The Tomcat applications have been scrutinized, we've tried all sort of options on the NFS mounts, we've experimented with SELinux, and the list goes on. However, the problem persists.

Does anyone have any ideas about what might cause an entire system to hang?

4
  • 2
    What do your logs say? Are you seeing a high degree of iowait?
    – EEAA
    Dec 18, 2014 at 14:52
  • 1
    You have two separate physical stand alone servers that are both doing this? Are they doing it at the same time? Is there a common denominator between the two (i.e. SAN, NAS, DNS, LDAP)?
    – Safado
    Dec 18, 2014 at 15:29
  • @Safado Yes, both servers hang. They do not hang at the same time. They use local discs, but share dns and are connected to the same nas (glusterfs share via nfs).
    – sbrattla
    Dec 18, 2014 at 17:41
  • @EEAA: are you thinking of any logs in particular? Can't really say I've come across too much in the logs. On the other hand; I dont't know what I'm looking at?
    – sbrattla
    Dec 18, 2014 at 18:36

1 Answer 1

0

The system got hang >> There might be a reason; Resource usage like RAM, CPU, HDD etc- + some condition which leads to this.

  1. Collect maximum performance data & time slots of issue occurrence. use Nagios & SAR etc - for this, if time is same then create a shell script to capture the running process & all process resource usage it will help you to identify the hammering process.

Possiblity :- NFS issue - Try to use soft mount option instead of Hardmount.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .