First of all the back story -
All of a sudden (literally overnight) an instance starts throwing CPU utilization alerts. This is a rather lowly VM (1 vCPU, 2GB RAM) but all it does is very low NFS serving and Cacti polling and serving for a handful of systems. This VM is hosted at an IaaS provider on vSphere 4.x and sits upon enterprise kit (HP/NetApp SAN, etc.).
The last time I changed anything on this system was nearly 4 weeks ago. Looking over the metrics one of the provider's agents/processes used by McAfee (cma) consumed WAY more RAM than usual until a cron job I have restarted the service the weekend prior (the cron job is there because I'm convinced this agent has a memory leak). Anyway, the problem is that I cannot run Cacti (httpd/mysql/php cron job that runs poller.php) on this system anymore - the load will go up over 10 and iowait is really high (~ 90%). I've tried the following:
- run Cacti with the McAfee service stopped
- systematically updated php*, httpd/mod_ssl, mysql-server, after each trying to run Cacti
- yum update to all latest packages, it's now RHEL 5.8 (x86_64)
The yum update (all) put the system over a load of 6 and took hours.
I asked the hosting provider if there was anything wrong with the storage layer but they said there wasn't. But this just doesn't compute. This got me wondering if maybe there could be a problem with partition misalignment since I've read that it can cause the kind of symptoms I seem to be experiencing. Now the provider would have created these VMFS partitions in the vSphere/vCenter client which I understand ensures that there is alignment. But can it get out of alignment over time? If so, is there any way from a VM/Guest that you can detect this? The mbrscan (NetApp) utility looks like it detects but that has to be run from the host's ESX console.
Edit: sfdisk output with uS added:
[root@nfs1 ~]# sfdisk -luS /dev/sda Disk /dev/sda: 13054 cylinders, 255 heads, 63 sectors/track Units = sectors of 512 bytes, counting from 0 Device Boot Start End #sectors Id System /dev/sda1 * 63 208844 208782 83 Linux /dev/sda2 208845 164055779 163846935 83 Linux /dev/sda3 164055780 209712509 45656730 8e Linux LVM /dev/sda4 0 - 0 0 Empty
A reboot of this instance completely solved the performance problems. Further analysis by the Hosting Provider did indicate that there is some misalignment but in their opinion it would not result in the symptoms experienced. They say for example that misalignment in Windows VMs is greater. At this point we're going to wait and see if it happens again and if so change the sector offset.