How to avoid VMware stunning a client during imaging with Veeam

Question

Recently our MySQL server has been "going away" (ie. the client connection drops out). After weeks of trying different things (like adjusting packet size), we've discovered that it's our Veeam imaging backups which use the VMWare API to snapshot and copy the vmdks etc.

We are using ESXi 5 with a Centos 6.4 guest, running (pretty much) only MySQL 5.1.69-log.

The change which seemed to initiate this problem was increasing the physical disk size to 300GB, from about 100, and resizing the guest filesystem to use most of the new capacity. Ever since the disk was increased, we've been getting these problems during backups - presumably due to the increase time it takes to perform snapshot related functions.

The new disks are 2x300GB Gen8 15k SAS in RAID1. The old disks would have been similar only smaller. The target of the Veeam process is a ReadyNAS over a 1Gb dedicated ethernet (i.e. separated from general office traffic).

The host is an HP DL380P tower:

==server spec (BASE CHASSIS)==
SERIES DL380P GEN8
PROCESSOR TYPE Intel Xeon E5-2609 v2 (2.5GHz/4-core/10MB/6.4GT-s QPI/80W)
NUMBER OF PROCESSORS 2 
MEMORY 80GB
INTERNAL DRIVE BAYS 8 SFF HDD Bays
COMPATIBLE HDD SFF SAS/SATA
HARD DISK CONTROLLER SMART ARRAY P420I/ZERO MEMORY CONTROLLER (RAID 0/1/1+0)

My "IT guy" has made a few tweaks to the Veeam config including skipping empty blocks (the majority of the new disk is empty), but this didn't seem to help at all.

Veeam haven't been much help either, saying "reboot the target" or "we just use VMWare APIs".

I believe the "stun" means the virtual machine simply freezes for a time (around 30s) then continues normally.

VMWare.log example:

Line 7411: 2016-06-08T17:11:44.910Z| vcpu-0| I120: Checkpoint_Unstun: vm stopped for 21068381 us
Line 7556: 2016-06-08T17:22:24.608Z| vcpu-0| I120: Checkpoint_Unstun: vm stopped for 19819322 us
Line 7700: 2016-06-08T17:22:30.140Z| vcpu-0| I120: Checkpoint_Unstun: vm stopped for 1130044 us
Line 7929: 2016-06-08T17:23:08.616Z| vcpu-0| I120: Checkpoint_Unstun: vm stopped for 30197618 us

So my problem has two likely solutions:

1. Is there a way to prevent or reduce the "stunning" of a VMWare guest during imaging.

2. Is there a way to reduce the impact of the stun onto MySQL or the virtual network or Centos.

Answer 1

This is an HP ProLiant server running with a Smart Array RAID controller without a Flash-backed cache module.

As a result, you have no write cache (or read cache), and operations like snapshots of virtual machines will suffer. You've experienced the effect of this. The current configuration is unsuitable for most workloads, especially virtualization.

Your best option is to simply buy a cache module and battery/FBWC; HP parts 631681-B21, 631679-B21, or 631069-B21.

This will accelerate performance and eliminate the problem you're seeing.

Also see:

FBWC and Zero Memory (ZM) RAID Controller on HP DL360p

BBWC: in theory a good idea but has one ever saved your data?

What is the memory module on a RAID card needed for?

Answer 2

Answering my own question from research. (I will only accept my own answer if one of these approaches actually works and it's before someone else's suggestion.)

This (older) article WHAT ARE THE DANGERS OF SNAPSHOTS AND HOW TO AVOID? mentions a few possible causes and three preventative measures. Interestingly it mentions how the issue similarly affects MS SQL Server and other server products.

If you do not want to stun / pause the virtual machine you can set snapshot.maxIterations to 20 (or higher). This means vSphere will do more tries (iterations) to commit the snapshot files. More information in this KB article.

It then goes on to describe the risks and downsides of this approach.

Secondly it suggests:

Alternatively you can set snapshot.maxConsolidateTime to 60 seconds. This means you can accept a pause of the virtual machine for 60 seconds to do a synchronous consolidate. This is often a better option than wait for the snapshot file grow so big the virtual machine will require to be stunned for a much longer time.

But I do not know the different between "stun" and "pause".

And lastly:

ESXi 4.1 has a update which added parameter snapshot.asyncConsolidate.forceSync = “FALSE” which needs to be added to the VMX file. This setting disables synchronous consolidate and the virtual machine will never be stunned. More info in this KB.

It doesn't describe the potential drawbacks with these solutions, but I'd presume there are some, else they'd be default.

I haven't yet checked if these parameters or solutions are still relevant in v5.

UPDATE: Veeam have recommended we make the above mentioned changes as listed in this KB which is relevant to v4 and v5 of ESXi.When removing a snapshot virtual machines become unresponsive for over 30 minutes (2039754)

UPDATE2: We are making these configuration changes tonight and rebooting the host, as it's cheaper and quicker than waiting for the cache. We will then monitor for a few days to see if this alone resolves it for us.