1

I have a decent CentOS 6.5 dedicated host (CentOS 6.5/E3-1230 3.2Ghz Quad Core + HT/16GB/Software Raid 1 SATA II/WD2503ABYX/ext4) with the default CentOS kernel and "elevator=deadline" in grub.

I/O write operations cause a huge spike in CPU usage. Reads are working fine. For example,

dd if=/dev/zero of=test bs=1048576 count=2048

causes the host's CPU utilization to shoot above 3 or 4. Under normal operation it remains under 0.40, but when there is some more intense I/O operation everything grinds to a halt.

mpstat 1 during these dd tests shows io wait at 20-25%.

Here's the disk layout:

Disk /dev/sda: 251.1 GB, 251059544064 bytes
255 heads, 63 sectors/track, 30522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c6673

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          26      204800   fd  Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sda2              26         548     4194304   fd  Linux raid autodetect
Partition 2 does not end on cylinder boundary.
/dev/sda3             548       30523   240775168   fd  Linux raid autodetect

Disk /dev/sdb: 251.1 GB, 251059544064 bytes
255 heads, 63 sectors/track, 30522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00095c99

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          26      204800   fd  Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sdb2              26         548     4194304   fd  Linux raid autodetect
Partition 2 does not end on cylinder boundary.
/dev/sdb3             548       30523   240775168   fd  Linux raid autodetect

Disk /dev/md2: 246.6 GB, 246552588288 bytes
2 heads, 4 sectors/track, 60193503 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/md1: 4293 MB, 4293910528 bytes
2 heads, 4 sectors/track, 1048318 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_main-LogVol00: 246.5 GB, 246549577728 bytes
255 heads, 63 sectors/track, 29974 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/md0: 209 MB, 209702912 bytes
2 heads, 4 sectors/track, 51197 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

The problem (high CPU usage) started happening sometime late december last year, which leads me to believe it is software related (the disk susbsystem has been checked by the folks at the DC).

Which tests should I run next to try to isolate the problem?

PS: I'm not looking for performance maximization tips. The server is underutilized. I'm just looking to reduce the CPU load during disk writes.

UPDATE: question reworked to better describe the problem.

UPDATE: FOUND A SOLUTION I finally discovered what the problem was when I came across this post.

root> modprobe vhost_net root> echo vhost_net > /etc/modules

For some reason, the virtio interfaces were not loading the driver before. All is good now.

1
  • 3
    If these are spinning SATA II disks, then the VM's numbers are right and the host's "1.8GB/s" is wrong by an order of magnitude or so. You should more thoroughly describe your disk subsystem. – Michael Hampton Mar 25 '14 at 21:53
4

On CentOS, the dirty_ratio is set to 20%.

What this means is that writing a file out doing

dd if=/dev/zero of=test bs=1048576 count=2048

Actually writes the data into memory as writeback (up to 3.2GB) and does not actually write it out to disk.

Its slower (but not a realistic performance benchmark) on the VM because you've probably assigned a much lower memory assignment to the VM itself (lets say 2G) and this leads to dirty_writeback only providing ~400MB of writeback before it will force the contents to disk.

If you run that command, then run sync, you'll notice sync takes a long time to return.

You need to run your command doing the following instead to get a better idea of your actual throughput.

dd if=/dev/zero of=test oflag=sync bs=1048576 count=2048
8
  • That sounds like exactly what's happening and why he has an unrealistic "1.8GB/s" number. – Michael Hampton Mar 25 '14 at 21:59
  • Thanks for that info. Yes, now I get 19.3MB/s and 17.1MB/s, which is completely reasonable. The problem is the CPU usage shooting up due to the I/O subsystem being improperly configured. What's the recommended setting and where should I apply it, host/guest/both? – Gaia Mar 25 '14 at 22:14
  • Please update your question with the IO statistics. You can run mpstat 5 for the period you run your test and provide the average. – Matthew Ife Mar 25 '14 at 22:20
  • I will update it and accept this as the answer. The question lost its main point since its based on an improperly executed test. – Gaia Mar 25 '14 at 22:22
  • Whichever one was giving you the CPU bottleneck. Or both if that was the case. I'm willing to bet its mostly I/O wait though.. – Matthew Ife Mar 25 '14 at 22:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.