16

Sometimes my servers will show a high load average in the "top" program (e.g. load is ~10 on a 4-core machine), but the actual CPU usage isn't particularly high.

I assume the issue is that there are many I/O-intensive jobs running. Is there any easy way to identify these jobs that are causing the load, if their "%CPU" values in top aren't that high?

1
  • Could you paste the summary area that you see when you run the top command?
    – mfriedman
    Aug 19, 2009 at 15:14

3 Answers 3

7

iostat can report statistics like that. Usually included in your distro in the package sysstat.

dstat might also be worth a look, it's a modern replacement.

0
9

To find what's causing high load you can check few things.

  • vmstat -w will show you ovierwiem (processes, swap, mem, cpu, io, system)
  • pmstat -P ALL will provide you statistics (with %iowait) per cpu core
  • iostat -x look for high %util or long await or big average queue size
    • dig deeper with iotop
  • ps -ax look for state D which is uninterruptible sleep (usually IO), run it one more time check if they are still in D state
    • dig deeper with strace
    • check files and connections of those processes with lsof and netstat
  • sar/sysstat - with that tool you can explore not just "now" stats but also check what was happening yestarday at midnight
    • sar -b - overall io activities
    • sar -d - individual block device io activities
1
  • I found ps -ax | grep " D " particularly useful, since it clearly shows whats "hanging".
    – gargii
    Jun 3, 2022 at 21:02
4

If you have IO accounting in your kernel, then you can use iotop to give information like that. Also, monitoring tools like collectd can record and report on the data.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .