Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sometimes my servers will show a high load average in the "top" program (e.g. load is ~10 on a 4-core machine), but the actual CPU usage isn't particularly high.

I assume the issue is that there are many I/O-intensive jobs running. Is there any easy way to identify these jobs that are causing the load, if their "%CPU" values in top aren't that high?

share|improve this question
    
Could you paste the summary area that you see when you run the top command? –  mfriedman Aug 19 '09 at 15:14

3 Answers 3

up vote 3 down vote accepted

iostat can report statistics like that. Usually included in your distro in the package sysstat.

dstat might also be worth a look, it's a modern replacement.

share|improve this answer
    
+1 because iostat was going to be my answer. –  Ernie Aug 19 '09 at 15:50

If you have IO accounting in your kernel, then you can use iotop to give information like that. Also, monitoring tools like collectd can record and report on the data.

share|improve this answer

Load, on Linux, is calculated with a non-trivial formula somewhere deep in the kernel's guts. But you can simply think of it as (somewhat) directly proportional with the number of processes waiting to run. From my experience there are two situations - either you have a billion concurrent processes and the run queue waiting time is naturally large or some proccesses got blocked somehow. In the first case it should be pretty obvious where the problem is. Note that in certain contexts high load doesn't really mean the system is trashed.
Now, in the second case you problem might be with processes stucked in uninterruptible sleep that typically block the run queue. They have the 'D' flag in ps -ef. Use strace, lsof, fuser to figure out what is happening.

For long-term profiling you can use Collectd and Process+CPU plugins.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.