Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have CPU I/O wait steady around 50%, but when I run iostat 1 it shows little to no disk activity.

What causes wait without iops?

NOTE: There no NFS or FUSE filesystems here, but it is using Xen virtualization.

enter image description here

share|improve this question
    
What distro? What version? –  ZaMoose Mar 7 '12 at 23:55
2  
Also: is this a the Xen hyper visor machine or a VM with the iowaits? –  ZaMoose Mar 7 '12 at 23:56
    
Does iotop show you anything? –  Janne Pikkarainen Mar 8 '12 at 7:59
add comment

10 Answers 10

NFS can do this, and it wouldn't surprise me if other network filesystems (and even FUSE-based devices) had similar effects.

share|improve this answer
    
Thanks, but in this case there's no NFS and no FUSE. I'll add that to the question too. –  Jason Cohen Mar 7 '12 at 23:51
add comment

Is there any chance other VMs on the server are thrashing the disk?

I know with virtualisation that you can get some strange results if the host node is overloaded.

share|improve this answer
    
True but that should be in steal% instead of io% right? Or can it cross over there too? –  Jason Cohen Mar 8 '12 at 0:10
2  
Steal happens when there's less CPU capacity available than requested by the VMs. If the physical disk is overloaded, your processes are going to spend a lot of time in iowait waiting for their turn at the disk even if they're not hitting the disk much. –  lbft Mar 8 '12 at 0:21
    
Yeah, this. See another question with the same answer at serverfault.com/a/209031/57468 –  mattdm Mar 8 '12 at 0:47
add comment

If this is the Amazon EC2 Xen environment using instance-based storage, ask Amazon to check the health of the host containing this image.

If this is a Xen environment that you can gain access to the hypervisor, then check the IOwait from without for the disk image (file, network, LVM-slice, whatever) being used for the xvda and xvdb devices. You'll also want to check the I/O system, in general, for the hypervisor since other disk devices might be monopolizing the system's resources.

iostat -txk 5

is usually a good starting diagnostic tool. It takes 5-second summaries of I/O for ALL devices available to it, and thus is useful both with-in and wither-out the VM image.

share|improve this answer
add comment

Check your available file descriptors / inodes. When you hit the limit, they swap and mimic iowait

Edit

I saw you are using xen, have a look at your current interrupts, you might find blkif is higher than normal.

Bit late now, but get munin installed and it will really help future debugging.

share|improve this answer
add comment
sudo sysctl vm.block_dump=1

Then check dmesg to see what is performing block read / writes or dirtying inodes.

Also check nofile limit in limits.conf, a process could be requesting more files than it is permitted to open.

share|improve this answer
add comment

WARNING: HDPARM IS DANGEROUS, ALWAYS READ ABOUT THE COMMAND YOU ARE GOING TO USE!

If no other virtual machines are stressing the hard disk(s), do

hdparm -f

on the underlying physical disk(s). Possibly the disk cache don't work accurately. This will flush the data stored in the cache, and you can constantly monitoring the I/O, whether it is about to rise again after the flush. If yes, it will be a cache problem.

share|improve this answer
add comment

With load average, I've seen blocked networking operations (i.e. long calls to an external DB server) increase. I don't know for sure but I'm guessing network IO can cause CPU wait to go up? Can anyone confirm?

share|improve this answer
1  
In most modern machines, no. Most, if not all recent systems have DMA-capable NICs to prevent precisely this sort of situation. –  ZaMoose Mar 7 '12 at 23:54
add comment

Could be loopback devices, that are themselves mounted over the network.

share|improve this answer
add comment

On my machines NFS is the biggest IO-WAIT "producer". I have a SSD in my laptop which is fast as hell, so "real IO" is not the problem. Nevertheless I have sometimes lots of IO wait due to my mounted nfs shares.

SCP sometimes also seems to lead to IO Wait but to a far lesser extend.

share|improve this answer
add comment

This can be anything. It just means that something is waiting for end of I/O operation. You can figure out what process it is via ps, then attach gdb to it and check out backtrace to determine which call is hang (usually this is some network-related stuff or suddenly disconnected disk). For fd info, check out /proc.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.