Today one of our CentOS 6.5 KVM nodes from OVH crashed after 46 days of uptime and we were unable to find the reason why, we had this issue before with the OVH kernel on another server but this one is running a normal kernel:
Linux 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Our monitoring tools showed the following:
Looks like a very high load even though no VM's were doing anything strange.
I took a look at the /var/log/messages/
file at first it showed nothing during the downtime:
Jun 16 11:15:11 server dhcpd: DHCPINFORM from 5.XXX.XX.104 via viifbr0
Jun 16 11:15:11 server dhcpd: DHCPACK to 5.XXX.XX.104 (02:XX:00:XX:XX:d3) via viifbr0
-----Downtime no logs-----
Jun 16 12:24:01 server kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jun 16 12:24:01 server rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1493" x-info="http://www.rsyslog.com"] start
Jun 16 12:24:01 server kernel: Initializing cgroup subsys cpuset
Jun 16 12:24:01 server kernel: Initializing cgroup subsys cpu
I checked dmesg
as well, although I'm not sure what the output means since most of it looks normal:
http://pastebin.com/raw.php?i=YyRCYZdn
I'm unsure what this could be, are there any other logs I could check?