Can anyone explain precisely what IOWait is?

Question

As much as I have read about iowait, it is still mystery to me.

I know it's the time spent by the CPU waiting for a IO operations to complete, but what kind of IO operations precisely? What I am also not sure, is why it so important? Can't the CPU just do something else while the IO operation completes, and then get back to processing data?

Also what are the right tools to diagnose what process(es) did exactly wait for IO.

And what are the ways to minimize IO wait time?

Thanks!

Answer 1

I know it's the time spent by the CPU waiting for a IO operations to complete, but what kind of IO operations precisely? What I am also not sure, is why it so important? Can't the CPU just do something else while the IO operation completes, and then get back to processing data?

Yes, the operating system will schedule other processes to run while one is blocked on IO. However inside that process, unless it's using asynchronous IO, it will not progress until whatever IO operation is complete.

Also what are the right tools to diagnose what process(es) did exactly wait for IO.

Some tools you might find useful

• iostat, to monitor the service times of your disks
• iotop (if your kernel supports it), to monitor the breakdown of IO requests per process
• strace, to look at the actual operations issued by a process

And what are the ways to minimize IO wait time?

• ensure you have free physical memory so the OS can cache disk blocks in memory
• keep your filesystem below 80% to avoid excessive fragmentation
• tune your filesystem
• use a battery backed array controller
• choose good buffer sizes when performing io operations

Answer 2

iowait

iowait is time that the processor/processors are waiting (i.e. is in an idle state and does nothing), during which there in fact was outstanding disk I/O requests.

This usually means that the block devices (i.e. physical disks, not memory) is too slow, or simply saturated.

You should hence note that if you see a high load average on your system, and on inspection notice that most of this is actually due to I/O wait, it does not necessarily mean that your system is in trouble - and this occurs when your machine simply has nothing to do, other than than I/O-bound processes (i.e. processes that do more I/O than anything else (non-I/O-bound system calls)). That should also be apparent from the fact that anything you do on the system is still very responsive.

tools

• sar (from the sysstat package, available on most *nix machines)
• iostat
• sarface (a front-end to sar)

Answer 3

For Solaris, I use DTrace to look at what the processes are doing if I need to see what I/O operations are running. For Linux, there's a similar program called systemtap which provides a similar level of exposure to the kernel and process calls.

One example I used when learning DTrace was to compare a cp command to a dd command. You can see that dd does a lot more reads for the write, while cp does not, mostly because of the buffer size dd uses by default (if I'm remembering correctly).

Answer 4

What kind of IO operations will depend on your applications and setup.

It is important as in some cases the CPU can't get the data or instructions that it needs to continue. In some cases it can continue, but it will depend on what apps are running as to what it can do. If you have a single threaded application which does lots of disk access then you will need to wait.

To minimise the IO time, buy more and faster memory, get faster disks, defrag the disks you have.

If it is an in house application which is the bottleneck see if it can be optimised to read in bigger blocks or to do IO asynchronously.