Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a new workstation with an unusual high load average. The machine specs are:

  • Xeon CPUs
  • 256 GB of RAM
  • 4 x 512 GB SSD connected to a LSI2108 RAID controller (in raid 5)

I have installed CentOS 6.5 64 bit from the live cd, configured the partitions (using LVM), the network, the users/groups, and installed some software like development tools and MATLAB. The workstation load average, after few minutes from the boot, has a value ranging from 0.5 to 0.9. But it is not doing anything. Thus I cannot understand why the load average is so high.

Could you help me to diagnose this issue?

Here are some information:

uname -a

Linux delta.grid 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

top

top - 15:44:53 up 22 min,  1 user,  load average: 0.84, 0.69, 0.39
Tasks: 729 total,   1 running, 728 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  264422908k total,  1986152k used, 262436756k free,    25640k buffers
Swap: 134217720k total,        0k used, 134217720k free,   147784k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                        
 4094 root      20   0 15564 1764  944 R  0.3  0.0   0:01.44 top                                                            
    1 root      20   0 19364 1536 1228 S  0.0  0.0   0:01.26 init                                                           
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.01 kthreadd                                                       
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.27 migration/0                                                    
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0                                                    
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                                    
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/0                                                     
    7 root      RT   0     0    0    0 S  0.0  0.0   0:00.14 migration/1                                                    
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1                                                    
    9 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/1                                                    
   10 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/1                                                     
   11 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/2                                                    
   12 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/2                                                    
   13 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/2                                                    
   14 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/2                                                     
   15 root      RT   0     0    0    0 S  0.0  0.0   0:00.03 migration/3                                                    
   16 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/3                                                    
   17 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/3                                                    
   18 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/3                                                     
   19 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/4                                                    
   20 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/4                                                    
   21 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/4                                                    
   22 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/4                                                     
   23 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/5                                                    
   24 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/5                                                    
   25 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/5                                                    
   26 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/5                                                     
   27 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/6                                                    
   28 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/6                                                    
   29 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/6                                                    
   30 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/6                                                     
   31 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/7                                                    
   32 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/7 

free -m

             total       used       free     shared    buffers     cached
Mem:        258225       1939     256286          0         25        144
-/+ buffers/cache:       1769     256455
Swap:       131071          0     131071

iotop

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                        
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    4 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    5 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
    8 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
    9 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]
   10 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/1]
   11 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/2]
   12 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/2]
   13 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/2]
   14 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/2]
   15 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/3]
   16 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/3]
   17 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/3]
   18 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/3]
   19 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/4]
   20 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/4]
   21 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/4]
   22 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/4]
   23 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/5]
   24 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/5]
   25 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/5]
   26 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/5]
   27 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/6]
   28 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/6]
   29 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/6]
   30 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/6]
   31 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/7]
   32 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/7]
   33 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/7]
   34 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/7]
   35 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/8]
   36 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/8]
   37 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/8]
   38 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/8]
   39 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/9]
   40 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/9]
   41 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/9]
   42 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/9]
   43 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/10]
   44 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/10]
   45 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/10]
   46 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/10]
   47 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/11]
   48 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/11]
   49 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/11]
   50 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/11]
   51 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/12]
   52 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/12]
   53 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/12]

vmstat 5 5

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 262437312  25728 147808    0    0     3     0    4    5  0  0 100  0  0    
 0  0      0 262437296  25728 147808    0    0     0     0   58  101  0  0 100  0  0    
 1  0      0 262437312  25728 147808    0    0     0     0   64  117  0  0 100  0  0    
 2  0      0 262437312  25728 147808    0    0     0     0   48   95  0  0 100  0  0    
 1  0      0 262437200  25728 147808    0    0     0     0   58  117  0  0 100  0  0

I have not enough space to copy/paste the output of 'ps axf', but: - there are no processes in D state (disk sleep - uninterruptible) - all the processes are in S state except the monitoring tool executed (like top or iotop)

Thanks in advance for your help!

share|improve this question
    
Is the CD tray empty ? –  Emmanuel Dec 13 '13 at 15:49
    
Yes it is. No cd-rom inside the tray. –  marco skv Dec 13 '13 at 15:59
    
What about if you wait for 15 -30 minutes after the boot? Paste top and vmstat after that. –  Danila Ladner Dec 13 '13 at 16:41
    
It does not change. It starts the unusual load average 5/10 minutes after the boot, and it continues forever. For example, after an "idle weekend" the current load average values are: 1.03 0.94 0.73. –  marco skv Dec 16 '13 at 8:03
    
Try to monitor /proc/interrupts. Also You can try disabling/disconnecting network interfaces (I've seen it as a source of load in the past). –  kestasx Dec 31 '14 at 23:24

1 Answer 1

Without having more details about your server, I am just going to add something I have run into on a handful of servers with Xeon procs. There is a known ACPI bug that was "fixed" but not entirely fixed that affects CentOS 6 servers and some hardware.

You might try adding the following to your kernel boot commands to see if it helps (requires reboot):

grubby --update-kernel=ALL --args="acpi_pad.disable=1"

If that does not help, be sure to remove it:

grubby --update-kernel=ALL --remove-args="acpi_pad.disable"

I assume you have also looked through dmesg for any devices that could not be initialized. Some devices are supposed to be blacklisted by /etc/modprobe.d/blacklist.conf via the "hwdata" RPM.

You may also want to enable thread view in "top". There are additional tools that help debugging kernel threads that are waiting, but that is a much bigger topic.

share|improve this answer
    
Hi Aaron, and thanks for the suggestion. Unfortunately I have changed my job and I cannot access anymore that workstation. But I forwarded the suggestion to my former colleague. If he manages to fix the problem, I will let you know ;) –  marco skv Jan 6 at 20:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.