So we have Ubuntu 10.04 running - all 64 bit. Kernel 2.6.36 and 2.6.38-15 are having the same issues.
Problem is random VPS servers are hanging. We have about 30 instances with this particular host. It freezes with nothing in the logs - neither in dmesg, kernel.log, syslog or anywhere. Logging is configured correctly.
However in the console there are multiple "task hung for 120 second" errors during this crash. The system apparently cannot write during this crash.
We have two types of servers, Java webapp servers and MySQL servers.
Both types are having errors ""task flush-2**/java/kjournald blocked for more than 120 seconds." Even on the server with no Java app installed it has "mysqld blocked for more than 120 seconds." Every few days.
The memory usage is normal. Almost no swap being used. But randomly every few days, when a load spike happens, between 8-10 load avg - a random server/vps instance just freezes with the error. Load tests to 100 avg (800% on 8 core) have been done, I/O stress tests have been done. IO wait times are normal during the crash. We can't seem to replicate this issue with stress tests.
The MySQL system has 8 cores (2 cpus) - the Java app servers have a 2 core system - All Intel Xeons. They are also on different Xen parent servers apparently. The ISP is saying there are no known hardware problems and the other "guests" are doing fine. They are also stumped. Is there anything I can do to figure out what the problem is? There is no core dump, or anything the system can write to when this crash happens.
I have tried changing the I/O scheduler with some progress. The default was CFQ and Deadline on some servers, I have changed the scheduler to noop with some level of success but the servers still crash. fstab is "/dev/xvda3 / ext3 errors=remount-ro 0 1"
Servers are all patched, firewalled and there is nothing strange in any of the logs. Fsck has been done. Xen version 3.4 according to dmesg. We are using XYMON for monitoring and just before the crash TOP, memory etc seems all normal.
Also another strange oddity I have found is the clock set coincides with system load.
In a graph the cpu load average lines up with higher clock offset - ntpudate needs to be run every 5 minutes for this reason. is this normal for load? or could this be a hardware issue?
Is there anyway I can figure out what is causing this?