Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

For a long period of time, i'm experiencing performance issues on CentOS servers that are used for KVM virtualization.

I'm running software RAID-10 with 4 disks (Seagate ST2000DM001, 2TB, 7200rpm SATA-600) These KVM VM's are running off their own LV.

Once there are a few VM's installed, the performance keeps dropping with every VM that's added.

Right now, it takes about 3 hours to install Debian 7 minimal. On an empty box with no VM's, this will take just a few minutes max.

The weird thing is that even with 37 VM's, there is pretty much no disk I/O. But every I/O that needs to be done at that moment, brings the "dm" device regarding that specific LV volume to 100%.

For some reason it looks like the "dm-xx" devices are not capable of using all the disks.

Down below is an example of what I mean. The sd[a/b/c/d] are usually about 40 to 60% loaded during this problem, but one or more dm devices are having high load without any serious I/O.

Notice that the %util for specific dm-0's that are in use by VM's.

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb              10.00     0.00    5.00   36.00   256.00   424.00    33.17     0.86   20.73  11.02  45.20
sdd               0.00     1.00    0.00   48.00     0.00   389.00    16.21     0.85   17.71  10.90  52.30
sda               0.00     0.00    2.00   39.00    32.00   161.00     9.41     0.77   18.63  11.44  46.90
sdc               0.00     1.00    0.00   50.00     0.00   764.00    30.56     1.59   47.08  13.90  69.50
md1               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md2               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md3               0.00     0.00   18.00   68.00   288.00   813.00    25.60     0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    6.00     0.00    40.00    13.33     0.31   51.50  51.17  30.70
dm-1              0.00     0.00    0.00    7.00     0.00    72.00    20.57     0.74   49.43  99.14  69.40
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-3              0.00     0.00   16.00    0.00   256.00     0.00    32.00     0.96   59.69   8.38  13.40
dm-4              0.00     0.00    0.00    8.00     0.00    68.00    17.00     0.28   34.62  31.25  25.00
dm-5              0.00     0.00    0.00    4.00     0.00   156.00    78.00     0.05   12.00  12.00   4.80
dm-6              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-7              0.00     0.00    0.00    4.00     0.00    20.00    10.00     0.08   19.00   6.75   2.70
dm-8              0.00     0.00    0.00    3.00     0.00     4.00     2.67     0.54  258.33 154.00  46.20
dm-9              0.00     0.00    0.00    5.00     0.00   148.00    59.20     0.10   19.40  19.40   9.70
dm-10             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-11             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-12             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-13             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-14             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-15             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-16             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-17             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-18             0.00     0.00    2.00    0.00    32.00     0.00    32.00     0.03   12.50   6.50   1.30
dm-19             0.00     0.00    0.00    2.00     0.00    12.00    12.00     0.03   13.00   6.50   1.30
dm-20             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-21             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-22             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-23             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-24             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-25             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-26             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-27             0.00     0.00    0.00    1.00     0.00     0.00     0.00     0.11  294.00 110.00  11.00
dm-28             0.00     0.00    0.00    3.00     0.00    16.00    10.67     0.10   50.33  25.00   7.50
dm-29             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-30             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-31             0.00     0.00    0.00    4.00     0.00    12.00     6.00     0.56  140.25 140.25  56.10
dm-32             0.00     0.00    0.00    6.00     0.00    32.00    10.67     0.76  125.83 125.83  75.50
dm-33             0.00     0.00    0.00    4.00     0.00    12.00     6.00     0.21   53.00  53.00  21.20
dm-34             0.00     0.00    0.00    3.00     0.00     4.00     2.67     0.40  172.33 133.33  40.00
dm-35             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-36             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-37             0.00     0.00    0.00    9.00     0.00   217.00    48.22     1.09  135.33 111.00  99.90
=================================
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb               0.00     0.00    0.00    7.00     0.00    72.00    20.57     3.84  430.14 126.00  88.20
sdd              10.00     0.00    4.00   13.00   240.00   304.00    64.00     2.07  126.59  33.00  56.10
sda               0.00     1.00    1.00   11.00    16.00   320.00    56.00     0.25   20.42   8.83  10.60
sdc               0.00     0.00    2.00    7.00    32.00    96.00    28.44     0.13   14.89  13.78  12.40
md1               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md2               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md3               0.00     0.00   18.00   17.00   288.00   416.00    40.23     0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-3              0.00     0.00   13.00    0.00   208.00     0.00    32.00     3.66  281.85  21.69  28.20
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-5              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-7              0.00     0.00    0.00    5.00     0.00    92.00    36.80     1.84  179.40 189.80  94.90
dm-8              0.00     0.00    0.00    1.00     0.00     4.00     8.00     1.00  655.00 1000.00 100.00
dm-9              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-10             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-11             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-12             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-13             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-14             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-15             0.00     0.00    0.00    1.00     0.00     0.00     0.00     1.44 1512.00 1000.00 100.00
dm-16             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-17             0.00     0.00    0.00    3.00     0.00    16.00    10.67     0.56    6.00 187.67  56.30
dm-18             0.00     0.00    5.00    0.00    80.00     0.00    32.00     0.37   73.40  71.40  35.70
dm-19             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-20             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-21             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-22             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-23             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-24             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-25             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-26             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-27             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.44    0.00   0.00  43.80
dm-28             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-29             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-30             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-31             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-32             0.00     0.00    0.00    1.00     0.00     4.00     8.00     0.00    1.00   1.00   0.10
dm-33             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-34             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-35             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-36             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-37             0.00     0.00    0.00    7.00     0.00   300.00    85.71     2.00  101.00 142.86 100.00
=================================
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb               1.00     1.00    1.00   47.00    48.00   676.00    30.17     1.47   30.62  11.65  55.90
sdd               1.00     0.00    9.00   88.00   176.00   395.00    11.77     0.83    8.59   5.12  49.70
sda               0.00     0.00    0.00   50.00     0.00   541.50    21.66     0.83   16.54  11.22  56.10
sdc               1.00     0.00    6.00   87.00   112.00   533.50    13.88     0.87    9.42   4.37  40.60
md1               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md2               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md3               0.00     0.00   21.00  116.00   336.00   991.00    19.37     0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-3              0.00     0.00    4.00    0.00    64.00     0.00    32.00     0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-5              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-7              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-8              0.00     0.00    0.00   82.00     0.00   848.00    20.68     1.46   17.93   6.29  51.60
dm-9              0.00     0.00    0.00    3.00     0.00    16.00    10.67     0.03   11.33  11.33   3.40
dm-10             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-11             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-12             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-13             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-14             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-15             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-16             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-17             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-18             0.00     0.00   17.00    0.00   272.00     0.00    32.00     0.49   28.59  10.35  17.60
dm-19             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-20             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-21             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-22             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-23             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-24             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-25             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-26             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-27             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-28             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-29             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-30             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-31             0.00     0.00    0.00    5.00     0.00    16.00     6.40     0.13   26.00  26.00  13.00
dm-32             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-33             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-34             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-35             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-36             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-37             0.00     0.00    0.00   27.00     0.00   111.00     8.22     1.15   42.41  37.00  99.90

Another example :

Importing an SQL file of 2MB takes 15+ minutes when doing this on a random VM. The I/O for that LV goes too 100% until it's finished. It will only have a couple writes/sec, and just a few KB/s I/O.

Even with these loads, write speeds with dd are around 80MB/s to 160MB/s. But reads are horrible, and most of the time not exceeding 10MB/s.

This is how my RAID setups looks like. This box is running on the "o2" layout, but doesn't shows any difference with the default "n2" layout.

# cat /proc/mdstat
Personalities : [raid10] [raid1]
md3 : active raid10 sdb5[4] sdc5[2] sdd5[3] sda5[0]
3872954368 blocks super 1.2 512K chunks 2 offset-copies [4/4] [UUUU]

md0 : active raid1 sdb1[4] sdc1[2] sdd1[3] sda1[0]
255988 blocks super 1.0 [4/4] [UUUU]

md2 : active raid10 sdb2[1] sdc2[2] sda2[0] sdd2[3]
31454208 blocks super 1.1 512K chunks 2 near-copies [4/4] [UUUU]
bitmap: 1/1 pages [4KB], 65536KB chunk

md1 : active raid10 sdb3[4] sda3[0] sdc3[2] sdd3[3]
2096128 blocks super 1.1 512K chunks 2 near-copies [4/4] [UUUU]

This is how the individual disks (sda for example) and LV's are partitioned by each VM. Should be okay for 4K disks.

Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000648b7

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      514047      256000   fd  Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sda2          514048    31971327    15728640   fd  Linux raid autodetect
/dev/sda3        31971328    34068479     1048576   fd  Linux raid autodetect
/dev/sda4        34068480  3907029167  1936480344    5  Extended
/dev/sda5        34070528  3907028991  1936479232   fd  Linux raid autodetect

Disk /dev/mapper/raid10storage-kvm396_img: 75.2 GB, 75161927680 bytes
255 heads, 63 sectors/track, 9137 cylinders, total 146800640 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 2097152 bytes
Disk identifier: 0x000a613c

                          Device Boot      Start         End      Blocks   Id  System
/dev/mapper/raid10storage-kvm396_imgp1   *        2048   145752063    72875008   83  Linux
/dev/mapper/raid10storage-kvm396_imgp2       145752064   146800639      524288   82  Linux swap / Solaris

I'm using VirtIO on all VM's and the cache is set to "none". Tried different cache settings, but that resulted in even worser performance.

I have a few other servers running for smapper setups (less VM's), but with the same type disks in RAID-1 (2 disks) without any problem.

This really seems to be related to the RAID-10 setups.

I currently have 4 of these RAID10 boxes in use, and all with different CPU and motherboard configurations. But all of these do have the same horrible performance issue.

Does anyone has an idea or theory that could explain why this setups performs so horrible slow?

It's really now workable like this, and it's quite frustrating that I still haven't found the cause after 1 year... :(


EDIT - server specs and CPU usage:

All these boxes have different CPU's, but here a few of configurations:

  • Xeon E3-1240v2, 32GB DDR3 ECC, SuperMicro X9SCM-F
  • Intel Core i7 2600, 32GB DDR3, Intel DQ67OW
  • Intel Core i7 3820, 32GB DDR3, Intel DX79TO

And here an output from top, with having one LV @ 100% util and a few others bouncing between 30% and 60%.

Load average are quite low at this moment, but it's quite normal to see them between 10.00 to 30.00 depending on how much LV's have 100% I/O load.

The CPU's are are not really loaded, and there is always at least 30%idle.

top - 02:26:21 up 8 days, 11:34,  2 users,  load average: 3.48, 3.99, 3.83
Tasks: 244 total,   2 running, 242 sleeping,   0 stopped,   0 zombie
Cpu(s): 15.3%us,  9.7%sy,  0.0%ni, 62.8%id, 11.4%wa,  0.0%hi,  0.8%si,  0.0%st
Mem:  33001640k total, 23715816k used,  9285824k free,    10740k buffers
Swap:  2096124k total,        0k used,  2096124k free,  1159788k cached

The %wa load is at some moments even 50%, but that usually only happens with 5 or more LV's touching the 100%util continuously.

Even with 5 or more LV's running @ 100%, there is no single SATA disk that's really having 90%+ load. They are all having utilization in the 40% to 60% area.

-------------------------------

EDIT 2, 18-feb-2014;

Today I decided to build another VPS node to do more testing. This system uses the same setup (KVM + SolusVM control panel) as all the other ones. This test setup uses the following hardware:

Intel Core i7 2600
32GB memory
4x WD Black 1.5TB (WD1502FAEX), 7200rpm, 64MB cache, SATA600
2x Samsung 830 Pro 128GB SSD's
So basically a completely different storage layer.

OS: CentOS 6.5 + all recent updates. Stock kernel kernel, but tried the latest mainline (3.13) kernel too which didn't change the performance.

Created a RAID-10 array (n2 layout, 512K chunks, stock config with bitmaps) on the WD Black drives. Created a RAID-0 array on the SSD drives which will be used by flashcache. This setup is configured the way that the all LVM I/O goes through the flashcache (as write-through cache). Both the SATA and SSD drives are having the old school 512 sectors, so no extended format. I chose these disks, just to see if there was difference between these and extended format ones which I use most of the time. But I've partitioned them as 4K drives.

I started with some testing on the host node. I created a blank LV volume. While doing some read benchmarks with DD, I had these read results on the host:

  • Uncached ~390MB/s read
  • SSD Cached ~950MB/s read (second run, when data is cached by flashcache)

Well, these results look pretty good!

Now the same tests, but on a VM with 4GB memory, VirtIO and 'cache=none'. I booted an Ubuntu live DVD, and did the same tests.... this with horrible results:

  • Uncached ~90MB/s read
  • SSD Cached ~90MB/s read (second run, when data is cached by flashcache)

During the test, the "dm-x" device (LV of the test VM) was loaded all the way up to 95%+, while the individual SATA disks disks didn't go above 35%. Same for the SSD's. No single devices exceeded the 50%util at any moment.

Here is an output of the highest load I've ever mentioned to get on this setup. This was during the uncached test (which explains the writes on the SSDs):

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  361.00 2547.00  1444.00 10188.00     8.00     0.62    0.21   0.03   8.90
sdc             992.00     1.00  246.00    7.00  4932.00    14.50    39.10     0.87    3.43   2.16  54.60
sdb               0.00     0.00  303.00 1754.00  1212.00  7016.00     8.00     0.41    0.20   0.03   5.80
sdd             825.00     1.00  173.00    7.00  3992.00    14.50    44.52     0.60    3.34   2.56  46.00
sde             866.00    25.00  229.00   11.00  4404.00   130.00    37.78     0.85    3.50   2.23  53.40
sdf             771.00    25.00  163.00   11.00  3736.00   130.00    44.44     0.65    3.72   2.98  51.80
md2               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md1               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md3               0.00     0.00  663.00 4301.00  2652.00 17204.00     8.00     0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md4               0.00     0.00 4266.00   36.00 17064.00   140.50     8.00     0.00    0.00   0.00   0.00
dm-0              0.00     0.00 4929.00   36.00 19716.00   140.50     8.00    14.56    2.92   0.20  99.60
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00 4929.00   36.00 19716.00   140.50     8.00    14.56    2.92   0.20  99.60

(dm-0 is flashcache, dm-2 the guest)

To make sure that the flashcache layer doesn't destroys all performance, I've done the same tests again with flashcache (dm-0) disabled and had pretty much the same horrible 90-100MB/s performance inside the guest.

So for some reason, all performance is gone inside guest. I got these results on a Windows and Linux guest. All using VirtIO and cache set to none. When using IDE the performances drops even further.

Actually a single disk or RAID1 setup with the same flashcache setup performs much better then all MDADM RAID-10 setups @ KVM I used so far. MDADM RAID-10 performs great on the host, but inside the guest it's horrible.

Right now i'm building a new raid-10 array, but with the offset layout. But I wouldn't expect better performance, as the RAID drive itself isn't fully loaded - it's just the individual "dm-x" LV volumes that are seeing 100% util.

I still can't believe why those RAID-10 + LVM + KVM setups are horrible slow :( Quite frustrating.

share|improve this question
    
Speaking as a SQL person, 7200 rpm sounds awfully slow to me. Maybe I'm spoiled by supporting SQL, but it sounds to me like your disks can't support your IO. –  Katherine Villyard Feb 8 at 2:42
    
SQl was just an example of how bad the I/O was. The VMs are doing a lot more. But if the dings couldn't take all the read/writes... why is the read I/O so slow compared to the write. The disks are also having +/- 50% load each, so that tells me that they are now fully loaded. They go only to 100% when I do some benchmarking on the host itself. With RAID-1 setups there are no performance issues. –  user208595 Feb 8 at 14:23
    
I know you're only using that as an example, but... SQL generally does random reads and sequential writes, split by file type, with the writes being more demanding than the reads. I generally demand 15K RPM in RAID 10 and refuse anything else as too slow. I'm assuming your virtual machines are also random reads, so it's certainly possible that 7200 rpm is too slow to support the number of random reads the number of VMs generates. You might want to talk to a storage guru (which isn't me, alas). You might also want to look for hardware problems in the storage system (aside from slow disks). –  Katherine Villyard Feb 8 at 14:41
    
I've done some new tests on a test box. Poor results inside the guest. But for more details see the "edit 2 - 18-feb-2014" section inside the question :-) –  user208595 Feb 18 at 22:12
    
I'm guessing you are using LVM partitions as the harddisks for your VMs. Did you disable caching and use the virtio driver in the VMs kvm config as described in linux-kvm.org/page/Tuning_KVM#Storage ? –  ZaphodB Feb 21 at 23:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.