11

On Windows, whenever I want to validate / confirm that there might be IO-related issues on a volume that a database or other low-latency app lives on, I check disk latency.

If I see the Windows Average Disk sec / Transfer counter > 18-20ms consistently, then my canary in a coal mine just died and I need to investigate further. Drop-dead simple.

I'm looking at Linux now, and don't see a similar latency-based metric. The quick research that I've done indicates that I might not even WANT to...I see lots of references to I/O Wait being the way most people track this.

Is there a ballpark rule of thumb that you use in regards to this? For example is ANY i/o wait I see bad for a database's volume? Is there a simple iostat command that gives me a better look at overall disk health than just eyeballing TOP?

Thanks much!

  • 4
    You can look up ioping – ewwhite Apr 4 '17 at 0:28
  • Thanks, @ewwhite. I guess I'm just wondering if I need to change my focus entirely and instead monitor this in a different way, you know? – Russell Christopher Apr 4 '17 at 0:30
  • 2
    Enable sysstat collection on your systems. Then you can examine the iowait CPU percentage, which is very useful for diagnosing IO-related slowness. – EEAA Apr 4 '17 at 0:36
  • 2
    @RussellChristopher You can see example sar output here. Pay attention to the %iowait column. – EEAA Apr 4 '17 at 1:12
  • @Matt while it is VERY similar, the focus is slightly different. That QA is more focused on performing tests in a simulated environment, where as this Q seems to be more about monitoring current performance in production environment. – BeowulfNode42 Apr 4 '17 at 7:54
12

Personally I use the command iostat -xk 10 and look at the await column.

  • -x Display extended statistics.
  • -k Display statistics in kilobytes per second. Or use m for megabytes/s.
  • 10 display interval in seconds

This is a virtually identical metric to the windows Average Disk sec / Transfer and is listed in ms instead of seconds. So similar rules of thumb could be applied, though this will depend on all sorts of things. I typically find that users start grumbling at 15ms and 20ms is very bad.

Press ctrl+c to quit, or specify the number of iterations to view with the count parameter. Note that the first iteration result is skewed heavily due to the small time sample used in the first iteration.

From the man iostat page

await The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.

Edit: await is the main metric I use to watch a disk under production loads to see if its throughput and iops is able to keep up with demand.

The %iowait stat is more about the balance between cpu and disk usage. %iostat will remain lower than expected if both cpu and disk activity are high. On the other side, starting at fairly low disk usage levels, %iostat can be relatively high if the cpu is idle. This being said await needs to be taken with a grain of salt as well. If there is a lot of sequential read/write happening it will skew the figure to a lower value, and your 18~20ms rule of thumb will not be useful under these conditions because most chunks being written will be the sequential data and will be serviced by the disk very quickly, while the other random io will be waiting, due to the Native-Command-Queuing (NCQ) system built in to the disk to optimise throughput by letting the disk choose the sequence that requests are serviced.

  • Thanks @beowulfNode42. Is this the primary metric you use in terms of eyeing "bad disk"? New Relic, seems to focus on i/o wait and disk utilization (read and write) percentage...This makes me wonder if I'm chasing the wrong metric, or if THEY are simply reporting less useful info.... – Russell Christopher Apr 4 '17 at 13:07
  • @RussellChristopher the other stats provide the required context in which to interpret the await info. eg, is there lots of iops (r/ and w/s), lots of MB/s, is the avg request size (avgrq-sz) large or small, and what the avg queue size is (avgqu-sz). Yes, along with the cpu related metrics %iowait, %user, %system etc to see if the disk is slowing the cpu or vice-versa. – BeowulfNode42 Apr 7 '17 at 0:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.