Quite a late reply but the accepted answer still has incorrect statements, is missing parts of the point, and suggest statistics lies while there is no reason here not to trust the ones reported by the OS.
Here is an in-depth explanation of the statistics observed.
The load average reported by
uptime and other commands is a floating average of 1, 5, and 15 minutes of the average number of threads waiting for a CPU (run queue) plus the average number of threads actually running on a CPU.
The idea is to smooth the display of the run queue size and running processes count which is often very irregular.
The run queue size is the first column of vmstat output (
r). Any non zero value here means that your system would have run faster should it had more CPUs available.
vmstat first data line shows the average since last boot. An average of 3 threads were waiting on your machine before you launched
vmstat. This value is generally meaningless being biased by long inactivity periods like week-ends and other non working hours:
r b w swap free re mf pi po fr de sr rm s0 s2 -- in sy cs us sy id
3 0 0 8747008 5562704 865 1866 188 63 63 0 0 -0 9 40 0 762 8588 1495 26 8 66
All other samples show an empty run queue except the second last one which shows a whopping average number of 102 threads:
102 1 0 7717952 4979088 0 1 0 0 0 0 0 0 112 4 0 900 3464 7683 15 9 76
The CPU is nevertheless 76% idle during this 10 seconds sample which is what puzzles you.
To understand the apparent discrepancy, you need to understand 102 is the average value for this sample. One way to get it is to assume the run queue was holding 1020 threads during one second, then was empty during the remaining 9 seconds. Any other combination leading to that 102 number is also conceivable, like 204 threads during 5 seconds and none during the other 5, and so on.
vmstat last column, we know your system was 76% idle during this period. A plausible value accommodating the average run queue and the idle CPU would be 408 threads competing during 2.4 seconds (100% busy CPUs) and no thread active during 7.6 seconds leading (0% busy CPU).
Now we know there was definitely a CPU contention. Should you have had more than 408 CPUs available instead of 2 and assuming all thread would have been able to run full speed in parallel, you would have reduced these 2.5 seconds to around 6 ms. This would have had a dramatic effect on interactive application but not that much on a batch job as the remaining time wouldn't have benefit from the extra CPUs anyway.
If your application is interactive, your system is seriously overloaded, if not, it is between slightly overloaded and just "regular".
There is a tradeoff to consider, 6 ms is likely "too good" for a response time and 408 CPU too expensive. Assuming 60 ms is a more reasonable goal, around 40 CPUs might do the job and of course if 2.5 s is fine, your system is behaving correctly.
Generally, a best practice is to assume there is a contention when the overall average run queue size exceeds the number of CPUs, here ~37 vs 2. Figuring out whether it is a problem or not cannot be done without analyzing what applications and threads are affected and how it impacts the platform operation.