I have following problem with debian 7 (LAMP stack):
top - 12:16:08 up 16:36, 4 users, load average: 27.97, 27.66, 27.42
Tasks: 240 total, 2 running, 238 sleeping, 0 stopped, 0 zombie
%Cpu(s): 8.5 us, 3.3 sy, 86.8 ni, 1.0 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem: 32980448 total, 32278828 used, 701620 free, 264368 buffers
KiB Swap: 11882492 total, 0 used, 11882492 free, 29265456 cached
So load average ~24 is always, even if all services (http,postfix,bind) are down (and also network is completely blocked with iptables). It near 100% nice load and ~27 la, but running processes are only 3-5. Web site works, but there is now idle at all.
Any ideas what can it be? How cat I determine which processes are running "niced"?
BTW server has 12 cores and total is is 24 threads... Is it possible that it is bug and each core is "hanging"
UPDATE Process list:
top - 12:29:46 up 16:49, 3 users, load average: 27.55, 27.30, 27.35
Tasks: 234 total, 4 running, 229 sleeping, 0 stopped, 1 zombie
%Cpu(s): 10.1 us, 4.2 sy, 84.5 ni, 0.7 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem: 32980448 total, 32363708 used, 616740 free, 266392 buffers
KiB Swap: 11882492 total, 0 used, 11882492 free, 29357760 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17760 mysql 20 0 2278m 972m 8544 S 71 3.0 330:59.07 mysqld
31139 www-data 20 0 81896 4524 2076 S 4 0.0 4:27.55 nginx
31140 www-data 20 0 81776 3256 1064 R 3 0.0 4:23.99 nginx
52110 web12543 20 0 240m 14m 3328 R 3 0.0 0:00.09 apache2
33669 root 20 0 236m 7484 580 S 1 0.0 0:01.18 apache2
40600 root 20 0 236m 7484 580 S 1 0.0 0:00.76 apache2
43386 root 20 0 236m 7484 580 S 1 0.0 0:00.57 apache2
43645 root 20 0 236m 7480 576 S 1 0.0 0:00.54 apache2
44197 root 20 0 236m 7484 580 S 1 0.0 0:00.49 apache2
46111 root 20 0 236m 7480 576 S 1 0.0 0:00.38 apache2
47041 root 20 0 236m 7480 576 S 1 0.0 0:00.33 apache2
47644 root 20 0 236m 7480 576 S 1 0.0 0:00.28 apache2
48683 root 20 0 0 0 0 Z 1 0.0 0:00.23 apache2
48765 root 20 0 236m 7480 576 S 1 0.0 0:00.23 apache2
49749 root 20 0 236m 7480 576 S 1 0.0 0:00.14 apache2
49833 root 20 0 236m 7480 576 S 1 0.0 0:00.14 apache2
49931 root 20 0 236m 7480 576 S 1 0.0 0:00.13 apache2
49935 root 20 0 236m 7480 576 R 1 0.0 0:00.12 apache2
50691 root 20 0 236m 7480 576 S 1 0.0 0:00.09 apache2
50692 root 20 0 236m 7480 576 S 1 0.0 0:00.08 apache2
50786 root 20 0 236m 7480 576 S 1 0.0 0:00.08 apache2
50787 root 20 0 236m 7484 580 S 1 0.0 0:00.07 apache2
51487 root 20 0 236m 7480 576 S 1 0.0 0:00.04 apache2
51588 root 20 0 25320 1584 1080 R 1 0.0 0:00.05 top
31138 www-data 20 0 81824 3436 1064 S 0 0.0 4:18.99 nginx
45688 root 20 0 236m 7480 576 S 0 0.0 0:00.42 apache2
46535 root 20 0 236m 7480 576 S 0 0.0 0:00.35 apache2
46869 root 20 0 236m 7480 576 S 0 0.0 0:00.34 apache2
49933 root 20 0 236m 7480 576 S 0 0.0 0:00.12 apache2
49934 root 20 0 236m 7480 576 S 0 0.0 0:00.12 apache2
51311 root 20 0 236m 7480 576 S 0 0.0 0:00.03 apache2
52085 web12543 20 0 237m 11m 3272 S 0 0.0 0:00.01 apache2
62551 root 20 0 25492 1864 1192 S 0 0.0 0:07.21 top
1 root 20 0 12708 844 704 S 0 0.0 0:04.20 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:00.92 ksoftirqd/0
5 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:0
6 root rt 0 0 0 0 S 0 0.0 0:00.38 migration/0
7 root rt 0 0 0 0 S 0 0.0 0:00.30 watchdog/0
8 root rt 0 0 0 0 S 0 0.0 0:00.20 migration/1
9 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/1:0
10 root 20 0 0 0 0 S 0 0.0 0:00.26 ksoftirqd/1
12 root rt 0 0 0 0 S 0 0.0 0:00.21 watchdog/1
13 root rt 0 0 0 0 S 0 0.0 0:00.19 migration/2
14 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/2:0
15 root 20 0 0 0 0 S 0 0.0 0:00.24 ksoftirqd/2
16 root rt 0 0 0 0 S 0 0.0 0:00.16 watchdog/2
17 root rt 0 0 0 0 S 0 0.0 0:00.14 migration/3
18 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/3:0
19 root 20 0 0 0 0 S 0 0.0 0:00.18 ksoftirqd/3
20 root rt 0 0 0 0 S 0 0.0 0:00.13 watchdog/3
21 root rt 0 0 0 0 S 0 0.0 0:00.14 migration/4
23 root 20 0 0 0 0 S 0 0.0 0:00.22 ksoftirqd/4
24 root rt 0 0 0 0 S 0 0.0 0:00.13 watchdog/4
25 root rt 0 0 0 0 S 0 0.0 0:00.14 migration/5
26 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/5:0
27 root 20 0 0 0 0 S 0 0.0 0:00.25 ksoftirqd/5
28 root rt 0 0 0 0 S 0 0.0 0:00.12 watchdog/5
29 root rt 0 0 0 0 S 0 0.0 0:00.32 migration/6
31 root 20 0 0 0 0 S 0 0.0 0:00.79 ksoftirqd/6
32 root rt 0 0 0 0 S 0 0.0 0:00.15 watchdog/6
33 root rt 0 0 0 0 S 0 0.0 0:00.27 migration/7
34 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/7:0
35 root 20 0 0 0 0 S 0 0.0 0:00.28 ksoftirqd/7
36 root rt 0 0 0 0 S 0 0.0 0:00.15 watchdog/7
37 root rt 0 0 0 0 S 0 0.0 0:00.19 migration/8
39 root 20 0 0 0 0 S 0 0.0 0:00.25 ksoftirqd/8
40 root rt 0 0 0 0 S 0 0.0 0:00.16 watchdog/8
41 root rt 0 0 0 0 S 0 0.0 0:00.18 migration/9
43 root 20 0 0 0 0 S 0 0.0 0:00.28 ksoftirqd/9
44 root rt 0 0 0 0 S 0 0.0 0:00.16 watchdog/9
45 root rt 0 0 0 0 S 0 0.0 0:00.19 migration/10
46 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/10:0
47 root 20 0 0 0 0 S 0 0.0 0:00.21 ksoftirqd/10
48 root rt 0 0 0 0 S 0 0.0 0:00.16 watchdog/10
49 root rt 0 0 0 0 S 0 0.0 0:00.16 migration/11
50 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/11:0
51 root 20 0 0 0 0 S 0 0.0 0:00.18 ksoftirqd/11
52 root rt 0 0 0 0 S 0 0.0 0:00.16 watchdog/11
53 root rt 0 0 0 0 S 0 0.0 0:00.44 migration/12
54 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/12:0
55 root 20 0 0 0 0 S 0 0.0 0:00.10 ksoftirqd/12
56 root rt 0 0 0 0 S 0 0.0 0:00.12 watchdog/12
57 root rt 0 0 0 0 S 0 0.0 0:00.35 migration/13
59 root 20 0 0 0 0 S 0 0.0 0:00.08 ksoftirqd/13
60 root rt 0 0 0 0 S 0 0.0 0:00.12 watchdog/13
61 root rt 0 0 0 0 S 0 0.0 0:00.28 migration/14
62 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/14:0
63 root 20 0 0 0 0 S 0 0.0 0:00.10 ksoftirqd/14
64 root rt 0 0 0 0 S 0 0.0 0:00.11 watchdog/14
65 root rt 0 0 0 0 S 0 0.0 0:00.24 migration/15
66 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/15:0
67 root 20 0 0 0 0 S 0 0.0 0:00.05 ksoftirqd/15
68 root rt 0 0 0 0 S 0 0.0 0:00.11 watchdog/15
69 root rt 0 0 0 0 S 0 0.0 0:00.23 migration/16
70 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/16:0
71 root 20 0 0 0 0 S 0 0.0 0:00.07 ksoftirqd/16
72 root rt 0 0 0 0 S 0 0.0 0:00.11 watchdog/16
73 root rt 0 0 0 0 S 0 0.0 0:00.21 migration/17
74 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/17:0
75 root 20 0 0 0 0 S 0 0.0 0:00.11 ksoftirqd/17
76 root 20 0 0 0 0 S 0 0.0 0:04.63 kworker/8:1
79 root 20 0 0 0 0 S 0 0.0 0:04.42 kworker/5:1
80 root 20 0 0 0 0 S 0 0.0 0:04.36 kworker/4:1
81 root 20 0 0 0 0 S 0 0.0 0:04.20 kworker/3:1
82 root 20 0 0 0 0 S 0 0.0 0:04.33 kworker/2:1
83 root rt 0 0 0 0 S 0 0.0 0:00.12 watchdog/17
84 root rt 0 0 0 0 S 0 0.0 0:00.53 migration/18
85 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/18:0
86 root 20 0 0 0 0 S 0 0.0 0:00.06 ksoftirqd/18
87 root 20 0 0 0 0 S 0 0.0 0:06.01 kworker/1:1
88 root rt 0 0 0 0 S 0 0.0 0:00.14 watchdog/18
89 root rt 0 0 0 0 S 0 0.0 0:00.41 migration/19
91 root 20 0 0 0 0 S 0 0.0 0:00.08 ksoftirqd/19
92 root rt 0 0 0 0 S 0 0.0 0:00.17 watchdog/19
93 root rt 0 0 0 0 S 0 0.0 0:00.34 migration/20
94 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/20:0
95 root 20 0 0 0 0 S 0 0.0 0:00.06 ksoftirqd/20
96 root rt 0 0 0 0 S 0 0.0 0:00.14 watchdog/20
97 root rt 0 0 0 0 S 0 0.0 0:00.30 migration/21
98 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/21:0
99 root 20 0 0 0 0 S 0 0.0 0:00.12 ksoftirqd/21
100 root rt 0 0 0 0 S 0 0.0 0:00.14 watchdog/21
101 root rt 0 0 0 0 S 0 0.0 0:00.28 migration/22
102 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/22:0
103 root 20 0 0 0 0 S 0 0.0 0:00.06 ksoftirqd/22
104 root rt 0 0 0 0 S 0 0.0 0:00.14 watchdog/22
105 root rt 0 0 0 0 S 0 0.0 0:00.26 migration/23
106 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/23:0
107 root 20 0 0 0 0 S 0 0.0 0:00.07 ksoftirqd/23
108 root rt 0 0 0 0 S 0 0.0 0:00.14 watchdog/23
109 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
110 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
111 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
112 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
113 root 20 0 0 0 0 S 0 0.0 0:00.14 sync_supers
114 root 20 0 0 0 0 S 0 0.0 0:00.00 bdi-default
115 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd
116 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd
117 root 20 0 0 0 0 S 0 0.0 0:00.06 khungtaskd
118 root 20 0 0 0 0 S 0 0.0 0:06.12 kswapd0
119 root 20 0 0 0 0 S 0 0.0 0:04.55 kswapd1
120 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
121 root 39 19 0 0 0 S 0 0.0 0:00.00 khugepaged
122 root 20 0 0 0 0 S 0 0.0 0:00.00 fsnotify_mark
123 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto
126 root 20 0 0 0 0 S 0 0.0 0:04.37 kworker/15:1
129 root 20 0 0 0 0 S 0 0.0 0:04.41 kworker/13:1
145 root 20 0 0 0 0 S 0 0.0 0:04.76 kworker/9:1
190 root 20 0 0 0 0 S 0 0.0 0:04.60 kworker/12:1
191 root 20 0 0 0 0 S 0 0.0 0:04.40 kworker/17:1
192 root 20 0 0 0 0 S 0 0.0 0:04.19 kworker/16:1
193 root 20 0 0 0 0 S 0 0.0 0:04.84 kworker/11:1
194 root 20 0 0 0 0 S 0 0.0 0:04.80 kworker/10:1
376 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff
377 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
461 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
463 root 20 0 0 0 0 S 0 0.0 0:00.02 hpsa
467 root 20 0 0 0 0 S 0 0.0 0:00.97 kworker/u:1
476 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
477 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2
480 root 20 0 0 0 0 S 0 0.0 0:04.88 kworker/23:1
481 root 20 0 0 0 0 S 0 0.0 0:04.76 kworker/19:1
484 root 20 0 0 0 0 S 0 0.0 0:04.81 kworker/18:1
486 root 20 0 0 0 0 S 0 0.0 0:04.75 kworker/20:1
489 root 20 0 0 0 0 S 0 0.0 0:04.75 kworker/22:1
497 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/19:2
507 root 20 0 0 0 0 S 0 0.0 0:04.85 kworker/21:1
508 root 0 -20 0 0 0 S 0 0.0 0:00.00 kdmflush
517 root 0 -20 0 0 0 S 0 0.0 0:00.00 kdmflush
524 root 20 0 0 0 0 S 0 0.0 0:04.36 kworker/14:2
533 root 20 0 0 0 0 S 0 0.0 0:29.16 jbd2/dm-0-8
534 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
675 root 20 0 23772 1692 816 S 0 0.0 0:00.16 udevd
1008 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/4:2
1075 root 0 -20 0 0 0 S 0 0.0 0:00.00 edac-poller
1080 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/8:2
1081 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/9:2
1099 root 0 -20 0 0 0 S 0 0.0 0:00.00 kpsmoused
2291 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/13:2
2637 root 20 0 0 0 0 S 0 0.0 0:44.75 flush-254:0
2730 root 20 0 118m 2348 1152 S 0 0.0 0:05.10 rsyslogd
2782 daemon 20 0 18736 368 208 S 0 0.0 0:00.00 atd
2851 root 20 0 8240 684 528 S 0 0.0 0:00.01 acpid
2878 messageb 20 0 31864 488 220 S 0 0.0 0:00.00 dbus-daemon
2923 root 20 0 51992 1308 688 S 0 0.0 0:00.00 sshd
3557 root 20 0 22468 1112 832 S 0 0.0 0:00.67 cron
3627 root 20 0 23584 1004 676 S 0 0.0 0:00.27 dovecot
3782 dovecot 20 0 15244 1068 884 S 0 0.0 0:00.10 anvil
3783 root 20 0 15240 1248 904 S 0 0.0 0:00.06 log
3830 root 20 0 53648 9912 2120 S 0 0.0 0:04.89 munin-node
4399 root 20 0 39248 2628 1860 S 0 0.0 0:00.22 ihttpd
4450 root 20 0 54220 1568 1164 S 0 0.0 0:00.02 login
4451 root 20 0 20384 992 816 S 0 0.0 0:00.00 getty
4452 root 20 0 20384 988 816 S 0 0.0 0:00.00 getty
4453 root 20 0 20384 988 816 S 0 0.0 0:00.00 getty
4454 root 20 0 20384 992 816 S 0 0.0 0:00.00 getty
4455 root 20 0 20384 988 816 S 0 0.0 0:00.00 getty
5721 web12542 20 0 237m 9860 2368 S 0 0.0 0:00.00 apache2
5724 mysql 20 0 2011m 649m 256 S 0 2.0 0:00.00 mysqld
17427 root 20 0 8304 776 628 S 0 0.0 0:00.02 mysqld_safe
23825 postfix 20 0 42008 2476 1904 S 0 0.0 0:00.01 pickup
28501 root 20 0 0 0 0 S 0 0.0 0:04.18 kworker/0:0
28689 root 20 0 23348 1080 552 S 0 0.0 0:00.00 udevd
31059 root 20 0 39944 2520 1944 S 0 0.0 0:00.75 master
31073 postfix 20 0 42592 3188 2032 S 0 0.0 0:00.94 qmgr
31093 root 20 0 236m 12m 5952 S 0 0.0 0:14.22 apache2
31137 root 20 0 81476 1836 336 S 0 0.0 0:00.00 nginx
31141 www-data 20 0 81900 3512 1064 S 0 0.0 3:47.78 nginx
31878 root 20 0 0 0 0 S 0 0.0 0:05.02 kworker/0:1
32472 postfix 20 0 42008 2364 1824 S 0 0.0 0:00.00 anvil
35437 postfix 20 0 45116 3864 2508 S 0 0.0 0:00.49 tlsmgr
35880 root 20 0 206m 12m 8804 S 0 0.0 0:09.27 ispmgr
37067 postfix 20 0 46828 4368 3316 S 0 0.0 0:00.01 smtp
44298 postfix 20 0 66532 5152 3496 S 0 0.0 0:00.01 smtpd
48002 postfix 20 0 42004 2368 1828 S 0 0.0 0:00.00 error
48004 postfix 20 0 42040 2380 1840 S 0 0.0 0:00.00 bounce
48972 root 20 0 236m 7480 576 S 0 0.0 0:00.12 apache2
48975 root 20 0 74876 5152 2940 S 0 0.0 0:00.82 sshd
49288 root 20 0 0 0 0 S 0 0.0 0:04.01 kworker/7:1
49471 root 20 0 21636 2384 1724 S 0 0.0 0:00.31 bash
50882 web12542 20 0 242m 16m 3140 S 0 0.1 0:06.45 apache2
51111 bind 20 0 332m 55m 2352 S 0 0.2 0:04.50 named
52083 web12543 20 0 236m 9228 2004 S 0 0.0 0:00.00 apache2
52114 root 20 0 0 0 0 S 0 0.0 0:00.15 kworker/6:2
54542 root 20 0 236m 7480 576 S 0 0.0 0:00.10 apache2
55209 postfix 20 0 42020 2388 1844 S 0 0.0 0:00.02 trivial-rewrite
56098 postfix 20 0 42340 3180 2352 S 0 0.0 0:00.03 cleanup
56171 web12542 20 0 240m 14m 3300 S 0 0.0 0:00.08 apache2
56214 root 20 0 23348 1124 584 S 0 0.0 0:00.00 udevd
56218 root 20 0 21560 2248 1648 S 0 0.0 0:00.01 bash
56282 root 20 0 73352 3732 2936 S 0 0.0 0:00.48 sshd
61268 root 20 0 21544 2280 1696 S 0 0.0 0:00.04 bash
63955 root 20 0 0 0 0 S 0 0.0 0:01.36 kworker/6:0
UPDATE2 Now after 2 days load average shows 0.00 - always!!!
top - 10:57:06 up 2 days, 15:17, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 377 total, 10 running, 366 sleeping, 0 stopped, 1 zombie
Cpu(s): 29.1%us, 11.8%sy, 57.9%ni, 0.2%id, 0.0%wa, 0.0%hi, 0.9%si, 0.0%st
Mem: 32980448k total, 31662152k used, 1318296k free, 293472k buffers
Swap: 11882492k total, 744k used, 11881748k free, 25885360k cached
Debian bugs?!
UPDATE3
root@rd1:~# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
40 0 2076 1233156 294124 25595008 0 0 26 35 0 6 94 5 1 0
29 0 2076 1240300 294124 25595200 0 0 0 428 22645 61952 86 14 0 0
28 0 2076 1233792 294124 25595456 0 0 36 364 12324 23591 92 8 0 0
34 0 2076 1251060 294124 25595608 0 0 0 556 13593 33874 91 8 1 0
39 0 2076 1223912 294124 25595752 0 0 0 2332 38084 89618 81 19 0 0
35 0 2076 1205256 294124 25595960 0 0 0 556 25791 62903 81 18 0 0
27 0 2076 1238748 294124 25596768 0 0 0 344 20367 46819 84 16 1 0
35 0 2076 1211688 294124 25597256 0 0 32 484 13068 28677 87 12 1 0
35 0 2076 1216440 294124 25597800 0 0 32 132 26783 75160 81 19 0 0
38 0 2076 1218932 294124 25598188 0 0 64 6952 19746 47230 83 16 0 0
30 0 2076 1244800 294124 25598728 0 0 0 4 16268 35489 87 13 0 0
33 0 2076 1229928 294124 25599048 0 0 184 404 10958 21926 88 11 1 0
30 0 2076 1237536 294124 25599436 0 0 32 592 12408 29603 89 11 0 0
30 1 2076 1234664 294124 25599788 0 0 12 416 13072 31855 88 12 0 0
32 0 2076 1247136 294124 25599924 0 0 12 1928 12717 23503 88 11 1 0
31 0 2076 1236484 294124 25600312 0 0 32 2880 12893 33544 88 12 0 0
toporps. Some of the kernel threads are niced. If you are using an encrypted device, the kernel thread responseble for that whould be my suspect.vmstat 1show anything odd happening? I suspected I/O wait, buttopdoesn't show that.