We've got two Red Hat servers which "hangs" at regular intervals. The systems become unresponsive for 20 - 30 seconds, and then return back to "normal". The systems are not virtualized, and runs on their own dedicated hardware.
When I say that the system becomes unresponsive, I know this because a script runs on the systems which - for every 5 seconds - (1) prints out the current time, (2) executes a wget towards a website on the system, (3) again prints out the end time and (4) sleeps for 5 seconds. We see from the log that the "sleep for 5 seconds" sometimes takes 20 - 30 seconds, and our Nagios monitering confirms that the system can't be reached for the same period of time.
Basically, the systems are clean installs of Red Hat 6.5 running Tomcat6. Furthermore, an NFS share against a GlusterFS storage is mounted. There's a bit of activity on that drive. Furthermore, the Tomcat6 webapps are somewhat memory intensive, requiring in the range of 7 - 15GB RAM.
We've been trying to figure out what's causing this for a couple of weeks now, and we're sort of come to the point where we're out of ideas. The Tomcat applications have been scrutinized, we've tried all sort of options on the NFS mounts, we've experimented with SELinux, and the list goes on. However, the problem persists.
Does anyone have any ideas about what might cause an entire system to hang?