How can I tell what processes are causing high loads if they are not high-cpu usage?

Question

Sometimes my servers will show a high load average in the "top" program (e.g. load is ~10 on a 4-core machine), but the actual CPU usage isn't particularly high.

I assume the issue is that there are many I/O-intensive jobs running. Is there any easy way to identify these jobs that are causing the load, if their "%CPU" values in top aren't that high?

Answer 1

iostat can report statistics like that. Usually included in your distro in the package sysstat.

dstat might also be worth a look, it's a modern replacement.

Answer 2

If you have IO accounting in your kernel, then you can use iotop to give information like that. Also, monitoring tools like collectd can record and report on the data.

Answer 3

Load, on Linux, is calculated with a non-trivial formula somewhere deep in the kernel's guts. But you can simply think of it as (somewhat) directly proportional with the number of processes waiting to run. From my experience there are two situations - either you have a billion concurrent processes and the run queue waiting time is naturally large or some proccesses got blocked somehow. In the first case it should be pretty obvious where the problem is. Note that in certain contexts high load doesn't really mean the system is trashed.
Now, in the second case you problem might be with processes stucked in uninterruptible sleep that typically block the run queue. They have the 'D' flag in ps -ef. Use strace, lsof, fuser to figure out what is happening.

For long-term profiling you can use Collectd and Process+CPU plugins.