Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Recently I notice that one ksoftirqd is always in top of my processlist and takes 90% - 100% cpu.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                            
   23 root      20   0     0    0    0 R 99.4  0.0  26:52.86 ksoftirqd/4   

the only thing I did recently that could maybe effect this is changing bonding mode from 4 to 6 BONDING_OPTS="mode=6 miimon=100 xmit_hash_policy=layer3+4"

top - 00:10:19 up 43 days,  4:44,  3 users,  load average: 24.80, 25.00, 25.82
Tasks: 274 total,   3 running, 271 sleeping,   0 stopped,   0 zombie
Cpu0  : 45.0%us, 10.6%sy,  0.0%ni, 32.8%id,  8.6%wa,  0.0%hi,  3.0%si,  0.0%st
Cpu1  : 49.7%us,  7.3%sy,  0.0%ni, 34.8%id,  7.0%wa,  0.0%hi,  1.3%si,  0.0%st
Cpu2  : 15.7%us, 11.7%sy,  0.0%ni, 63.3%id,  9.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  : 24.5%us,  8.3%sy,  0.0%ni, 57.0%id,  9.9%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu4  :  0.0%us,  0.3%sy,  0.0%ni,  0.3%id,  0.0%wa,  0.0%hi, 99.3%si,  0.0%st
Cpu5  : 67.0%us,  2.3%sy,  0.0%ni, 23.3%id,  4.7%wa,  0.0%hi,  2.7%si,  0.0%st
Cpu6  : 22.5%us,  9.3%sy,  0.0%ni, 61.3%id,  7.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  : 18.6%us,  5.6%sy,  0.0%ni, 65.8%id,  8.3%wa,  0.0%hi,  1.7%si,  0.0%st
Cpu8  : 14.0%us,  6.3%sy,  0.0%ni, 74.1%id,  5.3%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu9  : 36.9%us,  3.7%sy,  0.0%ni, 52.2%id,  6.3%wa,  0.0%hi,  1.0%si,  0.0%st
Cpu10 : 24.3%us,  4.0%sy,  0.0%ni, 67.4%id,  4.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu11 :  9.9%us,  3.0%sy,  0.0%ni, 81.8%id,  5.0%wa,  0.0%hi,  0.3%si,  0.0%st

/proc/interrupts

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      
   0:      12629          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      timer
   3:          1          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge    
   4:          1          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge    
   8:          1          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      rtc0
   9:          1          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
  16:         50         49          0          0          0          0          0          0          0          8          0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb6, uhci_hcd:usb7, uhci_hcd:usb8
  17:     170593 1177899633    3598058          0          0          0      29976          0          0   18503988     869534 1411851689   IO-APIC-fasteoi   firewire_ohci
  19:         43          0          0          0          0          0          0          0          0          0          0         26   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb3, uhci_hcd:usb4, uhci_hcd:usb5
  24:        303          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   nouveau
  34:        232          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   hda_intel
  64:          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      aerdrv, PCIe PME
  65:          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      aerdrv, PCIe PME
  66:          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      aerdrv, PCIe PME
  67:          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME, pciehp
  68:          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME, pciehp
  69:          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME, pciehp
  70:      34030 4280526451          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      mpt2sas0
  71:      76960          0          0          0          0  923932819       4271          0          0     921266          0          0   PCI-MSI-edge      ahci
  72:         11          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0
  73:         23  283689091      40793          0          0          0          0   24878170          0          0          0          0   PCI-MSI-edge      eth0-TxRx-0
  74:          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1
  75:         24         90          0      62770          0          5          0          0          0          0          0       3080   PCI-MSI-edge      eth1-TxRx-0
  76:         10          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth2
  77:      33507  960116410     215542    2547933    1355702          0          0 2089517302          0  518452948          0          0   PCI-MSI-edge      eth2-TxRx-0
  78:         10          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth3
  79:      26601          0    1375802     100390          0 2997094009          0  519230266          0          0          0          0   PCI-MSI-edge      eth3-TxRx-0
  80:         10          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth4
  81:      57006 2072370423          0          0    3016087          0    1467036  894382393          0    1981873          0  461286073   PCI-MSI-edge      eth4-TxRx-0
  82:         20          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth5
  83:      65029    2585159          0 2983606743          0  539781220          0          0          0          0          0          0   PCI-MSI-edge      eth5-TxRx-0
  84:        224          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      hda_intel
  85:          5          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  86:          5          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  87:          5          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  88:          5          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  89:          5          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  90:          5          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  91:          5          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  92:          5          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
 NMI:    3044892    3637346    2383233    3406372    1945624    3128005    1566617    2883393    1255060    1490727    1027297    1253099   Non-maskable interrupts
 LOC: 3122098247 3645001548 2852862808 3486174040 2535550145 3339840362 2200997394 3190919345 1882387653 2243213764 1574909643 1812565825   Local timer interrupts
 SPU:          0          0          0          0          0          0          0          0          0          0          0          0   Spurious interrupts
 PMI:    3044892    3637346    2383233    3406372    1945624    3128005    1566617    2883393    1255060    1490727    1027297    1253099   Performance monitoring interrupts
 IWI:          0          0          0          0          0          0          0          0          0          0          0          0   IRQ work interrupts
 RES: 1347304296 1510540778 1340165603 1734712948 1122541792 1575456517  917773463 1444666972  744552436 1130056245  594321402  834946777   Rescheduling interrupts
 CAL:  900949954    7559543   29508252   10667247   44252665   15085555   70190607   20396890  108238228   71698016  154634835  139264543   Function call interrupts
 TLB:   11114353   18939938   12986103   20272180   12868058   22459167   11531081   18828399    9834505    9577271    8513033    8272658   TLB shootdowns
 TRM:          0          0          0          0          0          0          0          0          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0          0          0          0          0          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0          0          0          0          0          0          0          0          0   Machine check exceptions
 MCP:      12456      12456      12456      12456      12456      12456      12456      12456      12456      12456      12456      12456   Machine check polls
 ERR:          0
 MIS:          0
share|improve this question
    
is this a physical or virtual machine? I guess physical since you use bonding? Does changing back BONDING_OPTS make ksoftirqd behave normally? Does dmesg show any useful info? –  Petter H Oct 12 '13 at 21:52
    
physical. it's a production so I need to wait until I can make a restart. nothing special in dmesg –  dvrecmfo Oct 12 '13 at 21:55
2  
Given the bonding driver should be associated with soft interrupts, your change against the bonding hash sounds pretty suspect. Revert the change and see if the problems goes away. –  Joshua Miller Oct 12 '13 at 23:34
    
Yes that was exactly the case, reverting back to 4 fixed the problem. –  dvrecmfo Oct 13 '13 at 0:55

1 Answer 1

You may try using ftrace's function profiler for cpu4 to see which kernel function is consuming most time/hits. But that's only if it's alrady enabled in your kernel (should be in Centos) and you are fearless enough to dig there. Then you may guess what it's doing by function name.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.