0

I have a Dell server running VMware ESX, with 12TB local SSD drives, 1TB memory, Xeon Gold processor, and a single debian VM.

On that VM, when I do simultaneous writes on disk, or even just execute the following command:

dd if=/dev/urandom of=/local/ssd/drive/path/largefile bs=1M count=1024

I have a critical disk latency alert on VSphere for that VM.

The dd command finished successfully after 10 minutes.

Why does vSphere trigger a critical alert that is not critical?

How is it possible to overwhelm high-end SSD drives, with a single dd command?

EDIT:

  • The critical alert is triggered if latency exceeds 75ms latency in a period of five minutes.

  • In practice, the disk latency seems to be around 200-250 ms for that vm:

image

EDIT 2:

  • Provisioning: thick lazy zeroed (no eager unfortunately)

EDIT 3:

I tried to define an IOPS limit on that disk, at the VM level (as you can see on the graph bellow).

I tried, 1000 IOPS, then 800, 600, 400, 200, 100. The critical disk latency alert is triggered even with 100 IOPS.

What is strange (as you can see on the graph), is that decreasing the limit (1000 IOPS to 100 IOPS) tend to increase the disk latency reported by vSphere. With 100 IOPS limit, the latency is 16,000 ms.

image

EDIT 4:

On the sofware side, I try to reduce the max simultaneous files writes from 24 to 4. The latency go from 200ms to 100ms, but the write bandwith go from 100MB/sec to 50MB/sec.

EDIT 5:

The switch from thick lazy zeroes provisionning, to thick eager zeroes, has not changed anything regarding the latency, always at 200ms

EDIT 6:

I think, I will have to further investigate the software side. Indeed, when running fio tests, it seems fine, with 24 threads write file sequencially (write) or randomly (randwrite)

Base command :

fio --filename=/data/testfile --size=10GB --direct=1 --rw=write --bs=4k --ioengine=sync --iodepth=64 --runtime=120 --numjobs=24 --time_based --group_reporting --name=throughput-test-job --eta-newline=1

Tests :

  • direct=1, bs=4k, rw=write => 0ms latency
  • direct=1, bs=512k, rw=write => 60ms latency
  • direct=0, bs=4k, rw=write => 55ms latency
  • direct=0, bs=4k, rw=randwrite => 20ms latency
3
  • How the virtual disk is provisioned to VM: thin, lazy zeroed, eager zeroed? Both thin and lazy generate additional load to underlaying storage, that reduces performance and increases latency. Commented Jun 11 at 10:16
  • The provisionning is indeed "thick lazy zeroed". As migrate to "eager" is a bit difficult afterward, I tried to mitigate that aspect, by fulling the VM disk with DD unix, then dropping the large file created. But may be, ESX is reclaming space automatically, so "eager" in that case will be more efficient than "lazy" in any case. I must definitely give the eager approach a try, as I'm still taken aback by the fact that even with minimal I/O operations on the virtual machine, the critical alert threshold can be reached swiftly
    – Klun
    Commented Jun 11 at 11:21
  • Just to follow up, the switch from lazy to eager zeroed, has no impact on the latency in that case. The latency is still about 200ms when production workload is here (actively writing files from multiples threads (24) )
    – Klun
    Commented Jun 12 at 12:37

1 Answer 1

3

Notice that it's latency alert. It means, the system issues a whole lot of I/O commands, which are get queued.

The latency grows, since if a new command is issued it's appended to the end of the queue and needs to wait for its turn to be executed, which happens only after all the commands before are executed — so this new command needs to wait for much longer than usual. This waiting time is called latency of the system. Imagine a large store or supermarket where customers get queued on checkout; when there are many customers all the cashiers will be busy, so the latency, the time any customer spends in the queue, grows.

This can be important, can be not. It depends on what this system does. For the online RDBMS, which handles some high load in real time, this will be unacceptable, because all the processing will slow down considerably, reducing the performance of the database down to crawl. Databases are very sensitive for the storage latency. For the interactive system, like desktop, this is of mild importance; starting of a new program or loading new document will be noticeably longer, but other load patterns are typically not affected. For the file server this can be safely tolerated, because there'll be much longer network and other protocol delays, so the additional delay incurred by the overloaded storage would be not as noticeable.

VMWare does't know what kind of a workload it runs. So, it stays on the safe side, alerting you that the storage is overloaded. To take actions in response or to not take is your decision. It also allows you to set resource limits on the VM instance for it to not be able to overload it that much.

11
  • Thank you. I updated my topic with the actual latency graph (about 200ms) and the alert (> 75ms in 5 minutes). I wondering how it is possible to reach such high latency, with high-end hardware, and not exceptionnal workload at all.. ? Can it be realted to thin instead of thick provisinning penalty ? On the other hands, I can set an IOPS limit on vSphere for that VM, but how to set that number in order to keep the latency between 75ms ? Is there any formula, or it is just empirical ?
    – Klun
    Commented Jun 11 at 9:26
  • 200 ms is very large. Even 75 is quite big. Normally, one tries to have latency in the ~10ms range. Yes, this can be related to thin, however, what has much more effect is the kind of a storage. You didn't specified the concrete controller and configuration type. For instance, writing on RAID6 tends to have very high latency (=very low performance) since it requires to read and write all the devices in the group; writing e.g. RAID10 is much, much faster since only a few devices are written and there is no need to read anything. Commented Jun 11 at 9:35
  • If all the information about all the components would be available, the formula in principle could have been derived. However, SSDs have sophisticated controllers and internal operations you don't know anything about and can't control (garbage collection and so on), which makes them unpredictable, so no, you have to experiment. Use fio. HDDs were much more predictable in this aspect. Commented Jun 11 at 9:42
  • Thank you. I will investigate both fio and the concrete controller and configuration type.
    – Klun
    Commented Jun 11 at 11:37
  • In the interval, I have updated my initial post (edit 3), to sum up my tests regarding IOPS limit on the VM. Even with 100 IOPS limit, the VM still complains (latency alert triggered), and the reported latency go beyond 16000ms in that case instead of 200ms (without any limit) .. So I understand that I will have to reduce software pressure on disk, because limiting the IOPS VM based, do not prevent the VM based disk latency critical alert to be triggered, even if the underlying storage is proctected by the VM limit.. Isn't it ?
    – Klun
    Commented Jun 11 at 11:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .