Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I understand load averages on Linux, but am a bit mystified by the load averages on a legacy Solaris 10 machine my app runs on. The load averages seem impossibly high. Here's the output.

[netcool1 (root)/]$ uptime
 11:49am  up 580 day(s), 10:51,  3 users,  load average: 35.50, 38.54, 39.03
[netcool1 (root)/]$ uname -a
SunOS netcool1 5.10 Generic_139555-08 sun4u sparc SUNW,Sun-Fire-V245
[netcool1 (root)/]$ psrinfo -v
Status of virtual processor 0 as of: 01/11/2012 11:52:52
  on-line since 06/10/2010 01:58:29.
  The sparcv9 processor operates at 1504 MHz,
        and has a sparcv9 floating point processor.
Status of virtual processor 1 as of: 01/11/2012 11:52:52
  on-line since 06/10/2010 01:58:27.
  The sparcv9 processor operates at 1504 MHz,
        and has a sparcv9 floating point processor.
[netcool1 (root)/]$ 

I don't see how you can have a load average of 35 on a two-processor system. That seems incredibly high to me. When I view the processes with top, the system is about 60-70% idle. Could someone help explain this?

vmstat 10 6

kthr      memory            page            disk          faults      cpu
r b w   swap  free  re  mf pi po fr de sr rm s0 s2 --   in   sy   cs us sy id
3 0 0 8747008 5562704 865 1866 188 63 63 0 0 -0 9 40 0 762 8588 1495 26  8 66
0 0 0 7715256 5068016 73 23 5 17 17  0  0  0 110 66 0 1135 3888 9855 59 12 30
0 0 0 7717936 5069128 0  5  0  6  6  0  0  0 100 4  0 1071 3273 4191 62  6 32
0 0 0 7717952 5027912 0 11649 0 5 5  0  0  0 115 21 0 1017 26370 3260 32 15 53
102 1 0 7717952 4979088 0 1 0  0  0  0  0  0 112 4  0  900 3464 7683 15  9 76
0 0 0 7717952 4978936 0  1  0  0  0  0  0  0 105 4  0  886 3379 8698 19  9 72
share|improve this question
    
Each UltraSPARC IIIi processor is four-way. The V245 has two, so it can run eight threads at the same time (i.e. load=8 means no thread needed to wait for processor time). –  kubanczyk Jan 11 '12 at 18:03
    
So a 36 means there are 4.5 threads waiting at any given time? –  coding_hero Jan 11 '12 at 19:04
    
Statistically, yes; practically usually not. See my answer. –  kubanczyk Jan 11 '12 at 19:46
1  
The UltraSPARC IIIi is a single core, single thread CPU. Load=36 means there are about 34 threads waiting in average. –  jlliagre Jan 12 '12 at 1:00
    
I was wrong, UltraSPARC IIIi is single-thread indeed. –  kubanczyk Jan 12 '12 at 8:16

3 Answers 3

up vote 0 down vote accepted

The "load" is normally an average of the first column of vmstat (column r, the run queue). The first load is averaged over 1 minute, second over 5 minutes, and the last over 15 minutes. As you can see, in your system vmstat at one point reported no less than 102 threads woken up to use the processor (probably some massively multi-threaded application).

But no worries, as certainly this burst of workload has been handled, and run queue went back to zero on the next probe and continuing. The V245 has two processors, each single-core and single-thread, so it can run two threads at the same time (i.e. r=2 means no thread needed to wait for processor time).

Statistically this could translate to an average of 35, but as you can see this value says very very little about actual system usage. Adage says "there are three kinds of lies: lies, damned lies, and statistics", and I think this serves well as a conclusion.

share|improve this answer
    
That actually makes a lot of sense, considering what this server does. I have a multithreaded app that connects to network devices, another that waits for and interprets snmp traffic, and a third that handles user connections. –  coding_hero Jan 11 '12 at 20:13
1  
Your definition of the load average is incorrect, not only processes waiting in the run queue but also processes effectively running are taken into account in the calculation. You should also remove the incorrect statement about the UltraSPARC IIIi decribed as a four-way processor. It is actually a single core, single thread CPU so the V245 can only run two threads in parallel, not eight. Also, you write r=8 means no thread need to wait. This is incorrect. As soon as r != 0, a thread is waiting for a CPU. –  jlliagre Jan 12 '12 at 0:39
    
It seems to me that traditionally run queue (vmstat r column) includes both running and runnable, isn't it? The confusion stems from the name "queue" which suggests "runnable but not actually running". You are right about IIIi and I am correcting that right away - it supports four-way systems (4 processors, not 4 threads). –  kubanczyk Jan 12 '12 at 8:15
    
The r queue is only containing runnable processes waiting. Running processes aren't in queue(s), they are on CPUs. Your statement "r=2 means no thread needed to wait for processor time" is still incorrect. –  jlliagre Jan 12 '12 at 13:04

On older-solaris, the load-average is the average number of runnable and running threads. In other words, it is the number of threads running on the CPUs, plus the number of threads in the run queue, waiting for CPUs, averaged over time.

So... a CPU that completed processing 10 threads for the last second... and had 5 more waiting to be processed would show 15.

In contrast...

Linux load averages are calculated as "overload" of a CPU... i.e. during the last period of time, how many threads were waiting for CPU time over how many were completed. (as a percentage)

So... a CPU that completed processing 10 threads for the last second... and had 5 more waiting to be processed would show 0.5

In Solaris 10... they changed the formula a bit... and I'm not 100% sure what it entails, but it should be pretty-close.

share|improve this answer
1  
In Solaris 10... the "exact" method for calculating load is explained as "The calculation is made by summing high-resolution user time, system time, and thread wait time, then processing this total to generate averages with exponential decay." –  TheCompWiz Jan 11 '12 at 17:32
    
Hehe, yah, I read that too. Clear as mud. –  coding_hero Jan 11 '12 at 17:47
    
So, is a 35 bad on Solaris 10? Does that mean that each processor is handling 17.5 threads each? How do I interpret this? –  coding_hero Jan 11 '12 at 17:56
    
Wikipedia does not confirm that Linux only counts "overload". Linux counts running+runnable as any Unix (Solaris). The only difference is that Linux also counts uninterruptibly sleeping threads, but this does not pertain to this question. en.wikipedia.org/wiki/Load_%28computing%29 –  kubanczyk Jan 11 '12 at 19:57

Load averages >>1 and high idle percentage are usually a sign of heavy disk i/o. This might be useful to find out why.

share|improve this answer
    
Next time read the question. –  TheCompWiz Jan 11 '12 at 17:20
    
Why so rude? coding_hero asks how his system can have load avg of 35 with idle% of 60-70, i answer "probably heavy io", and point to a related question, what's wrong with that? –  wallenborn Jan 11 '12 at 17:31
    
He was asking the difference between linux load averages vs solaris load averages. (the definitions and how they compare) Not what might cause high-load. –  TheCompWiz Jan 11 '12 at 17:35
    
While not directly what I asked, it is still useful info. Adding the vmstat output above. –  coding_hero Jan 11 '12 at 17:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.