Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We have a server with unusual high load and cpu util, but we can't figure out why. When we run top all the procs seem to be very low cpu.

Is there a better way to look for what is causing this?

share|improve this question
I should note I am on Ubuntu 10.04 – Ben May 22 '11 at 4:55
up vote 4 down vote accepted

I think this bug is your case. From what I see from the output, you have enough memory (note the cached 14 GB or so), no I/O issues, but you have xen-related processes running. This make me think it is a bug.

share|improve this answer
It turned out to be a bug in Ubuntu 10.04 and the hardware that AWS was using – Ben Jun 3 '11 at 7:14

Load is a measure of the workload a system has had on a 1, 5 and 15 minute basis.

The most common misconception is that Load Average is purely connected to the CPU usage of a system.
Load does however incorporate additional measurements such as CPU waiting for I/O which I think is your issue.

Based on the image I'm guessing you ran out of memory and started swapping data to disk.

A simple free -m will tell you how much RAM and swap is used.
The interesting column is the free column besides -/+ buffers/cache.
If it's close to zero you've run out of RAM and should act accordingly.

share|improve this answer
+1 for swap. You could use iotop or vmstat to try and see if it is I/O. – Axel Knauf May 22 '11 at 9:32
-/+ buffers/cache: 2933 14586 – Ben May 22 '11 at 15:06

Are you inside a virtual server? The high load may come from processes in other virtual servers, which you can't see from inside yours.

share|improve this answer
yes ... I am on EC2 – Ben May 22 '11 at 16:25
I believe that top reports CPU load for that virtual server only, regardless of who else uses the machine. If the physical machine is slow that will have an impact on your virtual machine, but you will not be able to tell just with top in your virtual machine. – asoundmove May 22 '11 at 20:06
@Ben, which instance-type are you on? – Mark May 23 '11 at 5:46
@asoundmove I guess it depends on the implementation, I observed (on Linux-VServer) top and uptime reporting high CPU usage while all the processes would stay below 1%. Turned out a faulty blog in a different vserver had its comments spammed, bloating the database with a few million rows causing MySQL to wreak havoc on the systems performance. – HSchumann May 24 '11 at 13:55

Noticed that the load average is quite high (68, wow). Is it possible that there are a lot of processes which takes up a little bit of CPU, thus add up consuming all CPU time? Maybe, those processes just start and finish very quickly thus top cannot capture the existence of them, you may try to see if atop can see that or not.

share|improve this answer



IO was it for me most of the time.

share|improve this answer

Try using:

top -o cpu

The -o flag will force top to order the processes by CPU usage in descending order.

share|improve this answer
when i ran top -o cpu I got "top: unknown argument 'o'" – Ben May 22 '11 at 4:54
Ok, try running top and hitting o while it is running. It should ask you for a primary key. Type cpu and hit enter. – Bandit May 22 '11 at 6:10
even when sorted by cpu there is nothing at the top of the list above 1% and there are only a handful (2-3) at any given time. The rest are 0% – Ben May 22 '11 at 6:40

It could be locked files on nfs or any other thing that locks a file that another process needs access to

could also be missed configured service with too many threads active

share|improve this answer

Looks like CPU usage is coming from a thread. top seems to not take this into account. I recently saw this on a mysql server. there are running INSERT statements but I was unable to get the new rows with SELECT because some thread of mysqld was updating the table index. top shows 100% user load on one core but every process including mysqld was an 0.0% CPU. hours later the same SELECT provided the expected result set.

See also

Getting a per thread cpu stats

'htop' process and threads cpu usage?

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.