Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a server running several KVM virtualized guests that's reporting a high %wa in top. The server hardware is as follows:

  • 4x2TB in hardware RAID10
  • 64GB RAM
  • Dual E5-2620

top output

% top - 03:56:51 up 12:24,  1 user,  load average: 17.13, 14.02, 12.88
Tasks: 582 total,   2 running, 563 sleeping,  17 stopped,   0 zombie
Cpu(s):  1.3%us,  3.4%sy,  0.0%ni, 64.9%id, 30.4%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  65881560k total, 25619484k used, 40262076k free,   781820k buffers
Swap:  4194296k total,   240788k used,  3953508k free,  7971412k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  3203 qemu      20   0 1930m 1.0g 5064 S 31.8  1.7 272:16.38 qemu-kvm
 64232 qemu      20   0 1732m 1.0g 4500 S 12.9  1.7  90:42.31 qemu-kvm
 90795 qemu      20   0 2203m 1.0g 4508 S 12.9  1.6   1:45.63 qemu-kvm
 32095 qemu      20   0 1602m 1.0g 4500 S 12.6  1.7 100:41.43 qemu-kvm
 89081 qemu      20   0 2133m 1.0g 4512 S  9.9  1.6   4:47.92 qemu-kvm
 71839 qemu      20   0 1592m 1.0g 4500 S  9.6  1.6  14:44.13 qemu-kvm
 66958 qemu      20   0 1592m 1.0g 4500 S  9.3  1.6 108:36.42 qemu-kvm
 69070 qemu      20   0 1592m 1.0g 4496 S  9.3  1.6  16:14.73 qemu-kvm
 23726 qemu      20   0 1518m 1.0g 4488 R  7.9  1.6  82:14.87 qemu-kvm
  3222 root      20   0     0    0    0 S  1.3  0.0   7:30.04 kvm-pit-wq
 80204 qemu      20   0 4490m 2.7g 4504 S  1.3  4.3  11:43.57 qemu-kvm
 91488 root      20   0 15428 1664  952 R  0.7  0.0   0:02.88 top
   347 root      39  19     0    0    0 S  0.3  0.0   1:28.38 kipmi0
  5534 qemu      20   0 3861m 2.9g 4528 S  0.3  4.7  10:08.12 qemu-kvm
  5554 root      20   0     0    0    0 S  0.3  0.0   0:08.51 vhost-5534
 23746 root      20   0     0    0    0 S  0.3  0.0   0:04.74 vhost-23726
 64252 root      20   0     0    0    0 S  0.3  0.0   0:07.05 vhost-64232

We've been having this problem since we've received this server, please help me to figure it out.

Screenshot from iostat output: Screenshot from iostat output

share|improve this question
    
Just two comments: a) for a RAID10 you need 2n HDDs, w/ n>=2, so whatever you're running, it's not a RAID10 w/ only two disks. b) I/O wait can occur w/ network devices as well as with local storage. –  tink Apr 13 '13 at 6:19
    
for the type of cpu and memory on your machine i would look into having a 6 or 8 drive raid 10. to get a lot of perfomance if your gonna use raid 8 drives may wanna look into a raid 100 –  WojonsTech Apr 13 '13 at 8:21
    
That's not good to provide extra IO performance just because you have a lot of CPU and RAM, it's about application requirements, not ruthless 'cool server' competetion. Idea of RAID 100 is to spread data across multiple controllers using software raid0, so I doubt it's a common thing to use. –  DukeLion Apr 17 '13 at 7:10
add comment

2 Answers

2x 4TB drives is a lot of space, but not much performance at all.

For example, a quick hunt around shows Western Digital make an enterprise 4TB disk in Nearline-SAS form, with 7,200 RPM spindle speed , and Wikipedia gives an estimate for that kind of drive handling 75 to 150 input/ouput operations per second (IOPS), depending how you estimate.

In a RAID-1, that gives you up to 300 requests per second serviced by the disks, with any more coming from the disk cache, controller cache, OS and service caches.

What do the virtual machines do? Unless you mean 2x 4TB pools, each made up of lots of smaller disks, I suspect you just don't have the storage performance to support a dozen virtual machines.

share|improve this answer
    
Thanks for your reply, I have another server with almost the same configuration but from different provider and all is going well, I think that there is something wrong with one of the hard-drives, they are Toshiba DT01ACA200, could you please let me know how to detect that this problem is from the HDD ? –  George Youssry Apr 13 '13 at 1:55
    
Sorry, I don't know specifically how you can check that, apart from looking for error messages in the server log files such as /var/log/messages, or using a S.M.A.R.T. utility to query the drive status. Incidentally, those are consumer/desktop class drives, benchmarking 80 read IOPS and 150 write IOPS. –  TessellatingHeckler Apr 13 '13 at 2:31
add comment

run iostat -xk 1 to see how utilised are your HDDs.

If you have low number of read/write requests with high utilisation % - your HDD might be broken.

Or there could be some heavy IO process overloading it.

UPD: You need to check status of your raid array, it might be rebuilding or malfunctioning

share|improve this answer
    
Ohh sorry we have 4x2 as it is RAID10, attached a screenshot from iostat -xk 1 command i.imgur.com/7g5mfem.jpg –  George Youssry Apr 13 '13 at 7:02
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.