12

Sometimes my servers will show a high load average in the "top" program (e.g. load is ~10 on a 4-core machine), but the actual CPU usage isn't particularly high.

I assume the issue is that there are many I/O-intensive jobs running. Is there any easy way to identify these jobs that are causing the load, if their "%CPU" values in top aren't that high?

  • Could you paste the summary area that you see when you run the top command? – mfriedman Aug 19 '09 at 15:14
7

iostat can report statistics like that. Usually included in your distro in the package sysstat.

dstat might also be worth a look, it's a modern replacement.

  • +1 because iostat was going to be my answer. – Ernie Aug 19 '09 at 15:50
4

If you have IO accounting in your kernel, then you can use iotop to give information like that. Also, monitoring tools like collectd can record and report on the data.

4

To find what's causing high load you can check few things.

  • vmstat -w will show you ovierwiem (processes, swap, mem, cpu, io, system)
  • pmstat -P ALL will provide you statistics (with %iowait) per cpu core
  • iostat -x look for high %util or long await or big average queue size
    • dig deeper with iotop
  • ps -ax look for state D which is uninterruptible sleep (usually IO), run it one more time check if they are still in D state
    • dig deeper with strace
    • check files and connections of those processes with lsof and netstat
  • sar/sysstat - with that tool you can explore not just "now" stats but also check what was happening yestarday at midnight
    • sar -b - overall io activities
    • sar -d - individual block device io activities

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.