Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

My web server is crashing several times a week. I see errors in syslog but I can't find a solution. I tried changing kernel to older or newer versions but no difference. I moved everything to a new machine, but problem is still there. This has to be software related. Any idea ?

Apr 30 18:11:16 kronos kernel: [751015.535784] ------------[ cut here ]------------
Apr 30 18:11:16 kronos kernel: [751015.535804] WARNING: CPU: 1 PID: 29 at /build/buildd/linux-3.13.0/net/sched/sch_generic.c:264 dev_watchdog+0x276/0x280()
Apr 30 18:11:16 kronos kernel: [751015.535810] NETDEV WATCHDOG: eth0 (igb): transmit queue 6 timed out
Apr 30 18:11:16 kronos kernel: [751015.535814] Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink bluetooth binfmt_misc xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables ast ttm drm_kms_helper coretemp drm kvm_intel kvm gpio_ich syscopyarea sysfillrect sysimgblt crct10dif_pclmul crc32_pclmul i2c_ismt ghash_clmulni_intel shpchp joydev aesni_intel lpc_ich aes_x86_64 lrw gf128mul glue_helper mac_hid ablk_helper ipmi_si cryptd lp parport btrfs libcrc32c raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 igb raid0 hid_generic i2c_algo_bit dca multipath usbhid ptp ahci hid linear libahci pps_core
Apr 30 18:11:16 kronos kernel: [751015.535928] CPU: 1 PID: 29 Comm: ksoftirqd/1 Not tainted 3.13.0-24-generic #46-Ubuntu
Apr 30 18:11:16 kronos kernel: [751015.535932] Hardware name: Supermicro A1SAi/A1SAi, BIOS 1.0b 11/06/2013
Apr 30 18:11:16 kronos kernel: [751015.535936]  0000000000000009 ffff880469655c80 ffffffff81715a64 ffff880469655cc8
Apr 30 18:11:16 kronos kernel: [751015.535948]  ffff880469655cb8 ffffffff810676bd 0000000000000006 ffff880464258000
Apr 30 18:11:16 kronos kernel: [751015.535957]  ffff880469002940 0000000000000008 0000000000000001 ffff880469655d18
Apr 30 18:11:16 kronos kernel: [751015.535966] Call Trace:
Apr 30 18:11:16 kronos kernel: [751015.535980]  [<ffffffff81715a64>] dump_stack+0x45/0x56
Apr 30 18:11:16 kronos kernel: [751015.535990]  [<ffffffff810676bd>] warn_slowpath_common+0x7d/0xa0
Apr 30 18:11:16 kronos kernel: [751015.535998]  [<ffffffff8106772c>] warn_slowpath_fmt+0x4c/0x50
Apr 30 18:11:16 kronos kernel: [751015.536007]  [<ffffffff810a2d34>] ? update_curr+0xe4/0x180
Apr 30 18:11:16 kronos kernel: [751015.536020]  [<ffffffff8163ae76>] dev_watchdog+0x276/0x280
Apr 30 18:11:16 kronos kernel: [751015.536029]  [<ffffffff8163ac00>] ? dev_graft_qdisc+0x80/0x80
Apr 30 18:11:16 kronos kernel: [751015.536037]  [<ffffffff81074226>] call_timer_fn+0x36/0x100
Apr 30 18:11:16 kronos kernel: [751015.536046]  [<ffffffff8163ac00>] ? dev_graft_qdisc+0x80/0x80
Apr 30 18:11:16 kronos kernel: [751015.536054]  [<ffffffff810751bf>] run_timer_softirq+0x1ef/0x2f0
Apr 30 18:11:16 kronos kernel: [751015.536063]  [<ffffffff8106caec>] __do_softirq+0xec/0x2c0
Apr 30 18:11:16 kronos kernel: [751015.536072]  [<ffffffff8106ccf0>] run_ksoftirqd+0x30/0x50
Apr 30 18:11:16 kronos kernel: [751015.536080]  [<ffffffff8109263f>] smpboot_thread_fn+0xff/0x1b0
Apr 30 18:11:16 kronos kernel: [751015.536088]  [<ffffffff81092540>] ? SyS_setgroups+0x1a0/0x1a0
Apr 30 18:11:16 kronos kernel: [751015.536097]  [<ffffffff8108b312>] kthread+0xd2/0xf0
Apr 30 18:11:16 kronos kernel: [751015.536106]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0
Apr 30 18:11:16 kronos kernel: [751015.536115]  [<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
Apr 30 18:11:16 kronos kernel: [751015.536124]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0
Apr 30 18:11:16 kronos kernel: [751015.536129] ---[ end trace c24a078afb41bdb5 ]---
Apr 30 18:11:16 kronos kernel: [751015.536167] igb 0000:00:14.0 eth0: Reset adapter
Apr 30 18:12:09 kronos kernel: [751068.767039] INFO: rcu_sched detected stalls on CPUs/tasks: { 4} (detected by 0, t=15002 jiffies, g=9072572, c=9072571, q=0)
Apr 30 18:12:09 kronos kernel: [751068.767263] sending NMI to all CPUs:
Apr 30 18:12:09 kronos kernel: [751068.767275] NMI backtrace for cpu 0
Apr 30 18:12:09 kronos kernel: [751068.767283] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.13.0-24-generic #46-Ubuntu
Apr 30 18:12:09 kronos kernel: [751068.767287] Hardware name: Supermicro A1SAi/A1SAi, BIOS 1.0b 11/06/2013
Apr 30 18:12:09 kronos kernel: [751068.767293] task: ffffffff81c15480 ti: ffffffff81c00000 task.ti: ffffffff81c00000
Apr 30 18:12:09 kronos kernel: [751068.767298] RIP: 0010:[<ffffffff8136c597>]  [<ffffffff8136c597>] __bitmap_empty+0x17/0x80
Apr 30 18:12:09 kronos kernel: [751068.767312] RSP: 0018:ffff88047fc03df0  EFLAGS: 00000006
Apr 30 18:12:09 kronos kernel: [751068.767316] RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000004
Apr 30 18:12:09 kronos kernel: [751068.767321] RDX: 0000000000000c00 RSI: 0000000000000100 RDI: ffffffff81d121a0
Apr 30 18:12:09 kronos kernel: [751068.767326] RBP: ffff88047fc03df0 R08: 0000000000000082 R09: 00000000000003d3
Apr 30 18:12:09 kronos kernel: [751068.767330] R10: 0000000000000000 R11: ffff88047fc03b2e R12: ffffffff81c4e140
Apr 30 18:12:09 kronos kernel: [751068.767335] R13: ffffffff81d12620 R14: ffffffff81c4e140 R15: 0000000000000000
Apr 30 18:12:09 kronos kernel: [751068.767341] FS:  0000000000000000(0000) GS:ffff88047fc00000(0000) knlGS:0000000000000000
Apr 30 18:12:09 kronos kernel: [751068.767345] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 30 18:12:09 kronos kernel: [751068.767350] CR2: 00000000013b3000 CR3: 0000000001c0e000 CR4: 00000000001007f0
Apr 30 18:12:09 kronos kernel: [751068.767354] Stack:
Apr 30 18:12:09 kronos kernel: [751068.767358]  ffff88047fc03e08 ffffffff81044c61 ffff88047fc0e800 ffff88047fc03e60
Apr 30 18:12:09 kronos kernel: [751068.767369]  ffffffff810cab21 ffffffff81c4e140 ffffffff00000004 0000000000000000
Apr 30 18:12:09 kronos kernel: [751068.767379]  0000000000000001 ffffffff81c15480 0000000000000000 0000000000000000
Apr 30 18:12:09 kronos kernel: [751068.767388] Call Trace:
Apr 30 18:12:09 kronos kernel: [751068.767392]  <IRQ> 
Apr 30 18:12:09 kronos kernel: [751068.767397]  [<ffffffff81044c61>] arch_trigger_all_cpu_backtrace+0x81/0xb0
Apr 30 18:12:09 kronos kernel: [751068.767414]  [<ffffffff810cab21>] rcu_check_callbacks+0x631/0x650
Apr 30 18:12:09 kronos kernel: [751068.767425]  [<ffffffff81076217>] update_process_times+0x47/0x70
Apr 30 18:12:09 kronos kernel: [751068.767434]  [<ffffffff810d5be5>] tick_sched_handle.isra.17+0x25/0x60
Apr 30 18:12:09 kronos kernel: [751068.767442]  [<ffffffff810d5c61>] tick_sched_timer+0x41/0x60
Apr 30 18:12:09 kronos kernel: [751068.767451]  [<ffffffff8108e537>] __run_hrtimer+0x77/0x1d0
Apr 30 18:12:09 kronos kernel: [751068.767460]  [<ffffffff810d5c20>] ? tick_sched_handle.isra.17+0x60/0x60
Apr 30 18:12:09 kronos kernel: [751068.767468]  [<ffffffff8108ed3f>] hrtimer_interrupt+0xef/0x230
Apr 30 18:12:09 kronos kernel: [751068.767479]  [<ffffffff81043087>] local_apic_timer_interrupt+0x37/0x60
Apr 30 18:12:09 kronos kernel: [751068.767489]  [<ffffffff817287ff>] smp_apic_timer_interrupt+0x3f/0x60
Apr 30 18:12:09 kronos kernel: [751068.767498]  [<ffffffff8172719d>] apic_timer_interrupt+0x6d/0x80
Apr 30 18:12:09 kronos kernel: [751068.767502]  <EOI> 
Apr 30 18:12:09 kronos kernel: [751068.767506]  [<ffffffff815c9582>] ? cpuidle_enter_state+0x52/0xc0
Apr 30 18:12:09 kronos kernel: [751068.767523]  [<ffffffff815c96a9>] cpuidle_idle_call+0xb9/0x1f0
Apr 30 18:12:09 kronos kernel: [751068.767533]  [<ffffffff8101ceae>] arch_cpu_idle+0xe/0x30
Apr 30 18:12:09 kronos kernel: [751068.767542]  [<ffffffff810beb85>] cpu_startup_entry+0xc5/0x290
Apr 30 18:12:09 kronos kernel: [751068.767552]  [<ffffffff81703f37>] rest_init+0x77/0x80
Apr 30 18:12:09 kronos kernel: [751068.767562]  [<ffffffff81d34f70>] start_kernel+0x438/0x443
Apr 30 18:12:09 kronos kernel: [751068.767571]  [<ffffffff81d34941>] ? repair_env_string+0x5c/0x5c
Apr 30 18:12:09 kronos kernel: [751068.767580]  [<ffffffff81d34120>] ? early_idt_handlers+0x120/0x120
Apr 30 18:12:09 kronos kernel: [751068.767589]  [<ffffffff81d345ee>] x86_64_start_reservations+0x2a/0x2c
Apr 30 18:12:09 kronos kernel: [751068.767598]  [<ffffffff81d34733>] x86_64_start_kernel+0x143/0x152
Apr 30 18:12:09 kronos kernel: [751068.767602] Code: 40 c7 04 24 10 00 00 00 48 89 44 24 10 e8 42 ff ff ff c9 c3 8d 4e 3f 85 f6 55 0f 49 ce 48 89 e5 c1 f9 06 85 c9 7e 61 48 83 3f 00 <75> 57 48 8d 47 08 31 d2 eb 12 0f 1f 80 00 00 00 00 48 83 c0 08 
Apr 30 18:12:09 kronos kernel: [751068.767682] NMI backtrace for cpu 1
Apr 30 18:12:09 kronos kernel: [751068.767692] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W    3.13.0-24-generic #46-Ubuntu
Apr 30 18:12:09 kronos kernel: [751068.767697] Hardware name: Supermicro A1SAi/A1SAi, BIOS 1.0b 11/06/2013
Apr 30 18:12:09 kronos kernel: [751068.767703] task: ffff8804696217f0 ti: ffff88046962a000 task.ti: ffff88046962a000
Apr 30 18:12:09 kronos kernel: [751068.767707] RIP: 0010:[<ffffffff813dfd78>]  [<ffffffff813dfd78>] intel_idle+0xd8/0x140
Apr 30 18:12:09 kronos kernel: [751068.767719] RSP: 0018:ffff88046962be28  EFLAGS: 00000046
Apr 30 18:12:09 kronos kernel: [751068.767723] RAX: 0000000000000051 RBX: 0000000000000040 RCX: 0000000000000001
Apr 30 18:12:09 kronos kernel: [751068.767728] RDX: 0000000000000000 RSI: ffffffff81c93220 RDI: 0000000000000001
Apr 30 18:12:09 kronos kernel: [751068.767733] RBP: ffff88046962be50 R08: ffff88047fc512cc R09: 0000000000002018
Apr 30 18:12:09 kronos kernel: [751068.767737] R10: 00000000ffffffff R11: 000000000000f80b R12: 0000000000000002
Apr 30 18:12:09 kronos kernel: [751068.767742] R13: 0000000000000051 R14: 0000000000000006 R15: ffffffff81c932e8
Apr 30 18:12:09 kronos kernel: [751068.767747] FS:  0000000000000000(0000) GS:ffff88047fc40000(0000) knlGS:0000000000000000
Apr 30 18:12:09 kronos kernel: [751068.767752] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 30 18:12:09 kronos kernel: [751068.767757] CR2: 00007f71205d2220 CR3: 0000000001c0e000 CR4: 00000000001007e0
Apr 30 18:12:09 kronos kernel: [751068.767761] Stack:
Apr 30 18:12:09 kronos kernel: [751068.767764]  000000016962be50 ffff88047fc5b000 ffffffff81c93220 0002aa899e46daf8
Apr 30 18:12:09 kronos kernel: [751068.767775]  0000000000000002 ffff88046962be88 ffffffff815c9570 ffff88046962bf38
Apr 30 18:12:09 kronos kernel: [751068.767784]  ffff88047fc5b000 0000000000000002 0000000000000001 ffffffff81c93220
Apr 30 18:12:09 kronos kernel: [751068.767794] Call Trace:
Apr 30 18:12:09 kronos kernel: [751068.767805]  [<ffffffff815c9570>] cpuidle_enter_state+0x40/0xc0
Apr 30 18:12:09 kronos kernel: [751068.767815]  [<ffffffff815c96a9>] cpuidle_idle_call+0xb9/0x1f0
Apr 30 18:12:09 kronos kernel: [751068.767824]  [<ffffffff8101ceae>] arch_cpu_idle+0xe/0x30
Apr 30 18:12:09 kronos kernel: [751068.767832]  [<ffffffff810beb85>] cpu_startup_entry+0xc5/0x290
Apr 30 18:12:09 kronos kernel: [751068.767841]  [<ffffffff81040fc8>] start_secondary+0x218/0x2c0
Apr 30 18:12:09 kronos kernel: [751068.767845] Code: 48 89 d1 48 2d c8 1f 00 00 0f 01 c8 0f ae f0 65 48 8b 04 25 f0 c7 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d 9a 34 8b 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 12 39 cf 
Apr 30 18:12:09 kronos kernel: [751068.767926] NMI backtrace for cpu 2
Apr 30 18:12:09 kronos kernel: [751068.767940] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W    3.13.0-24-generic #46-Ubuntu
Apr 30 18:12:09 kronos kernel: [751068.767945] Hardware name: Supermicro A1SAi/A1SAi, BIOS 1.0b 11/06/2013
Apr 30 18:12:09 kronos kernel: [751068.767951] task: ffff880469622fe0 ti: ffff88046962c000 task.ti: ffff88046962c000
Apr 30 18:12:09 kronos kernel: [751068.767969] RIP: 0010:[<ffffffff813dfd78>]  [<ffffffff813dfd78>] intel_idle+0xd8/0x140
Apr 30 18:12:09 kronos kernel: [751068.768005] RSP: 0018:ffff88046962de28  EFLAGS: 00000046
Apr 30 18:12:09 kronos kernel: [751068.768023] RAX: 0000000000000051 RBX: 0000000000000040 RCX: 0000000000000001
Apr 30 18:12:09 kronos kernel: [751068.768041] RDX: 0000000000000000 RSI: ffffffff81c93220 RDI: 0000000001c0e000
Apr 30 18:12:09 kronos kernel: [751068.768057] RBP: ffff88046962de50 R08: ffff88047fc912d0 R09: 0000000000000018
Apr 30 18:12:09 kronos kernel: [751068.768075] R10: 000000000003a0c3 R11: 0000000000078cd7 R12: 0000000000000002
Apr 30 18:12:09 kronos kernel: [751068.768091] R13: 0000000000000051 R14: 0000000000000006 R15: ffffffff81c932e8
Apr 30 18:12:09 kronos kernel: [751068.768109] FS:  0000000000000000(0000) GS:ffff88047fc80000(0000) knlGS:0000000000000000
Apr 30 18:12:09 kronos kernel: [751068.768128] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 30 18:12:09 kronos kernel: [751068.768146] CR2: 00007ffa85dce090 CR3: 0000000001c0e000 CR4: 00000000001007e0
Apr 30 18:12:09 kronos kernel: [751068.768163] Stack:
Apr 30 18:12:09 kronos kernel: [751068.768179]  000000026962de50 ffff88047fc9b000 ffffffff81c93220 0002aa89978add53
Apr 30 18:12:09 kronos kernel: [751068.768242]  0000000000000002 ffff88046962de88 ffffffff815c9570 ffff88046962df38
Apr 30 18:12:09 kronos kernel: [751068.768268]  ffff88047fc9b000 0000000000000002 0000000000000002 ffffffff81c93220
Apr 30 18:12:09 kronos kernel: [751068.768277] Call Trace:
Apr 30 18:12:09 kronos kernel: [751068.768292]  [<ffffffff815c9570>] cpuidle_enter_state+0x40/0xc0
Apr 30 18:12:09 kronos kernel: [751068.768301]  [<ffffffff815c96a9>] cpuidle_idle_call+0xb9/0x1f0
Apr 30 18:12:09 kronos kernel: [751068.768312]  [<ffffffff8101ceae>] arch_cpu_idle+0xe/0x30
Apr 30 18:12:09 kronos kernel: [751068.768321]  [<ffffffff810beb85>] cpu_startup_entry+0xc5/0x290
Apr 30 18:12:09 kronos kernel: [751068.768331]  [<ffffffff81040fc8>] start_secondary+0x218/0x2c0
Apr 30 18:12:09 kronos kernel: [751068.768335] Code: 48 89 d1 48 2d c8 1f 00 00 0f 01 c8 0f ae f0 65 48 8b 04 25 f0 c7 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d 9a 34 8b 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 12 39 cf 
Apr 30 18:12:09 kronos kernel: [751068.768415] NMI backtrace for cpu 3
Apr 30 18:12:09 kronos kernel: [751068.768422] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 1.058 msecs
Apr 30 18:12:09 kronos kernel: [751068.768435] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G        W    3.13.0-24-generic #46-Ubuntu
Apr 30 18:12:09 kronos kernel: [751068.768439] Hardware name: Supermicro A1SAi/A1SAi, BIOS 1.0b 11/06/2013
Apr 30 18:12:09 kronos kernel: [751068.768444] task: ffff8804696247d0 ti: ffff88046962e000 task.ti: ffff88046962e000
Apr 30 18:12:09 kronos kernel: [751068.768449] RIP: 0010:[<ffffffff813dfd78>]  [<ffffffff813dfd78>] intel_idle+0xd8/0x140
Apr 30 18:12:09 kronos kernel: [751068.768460] RSP: 0018:ffff88046962fe28  EFLAGS: 00000046
Apr 30 18:12:09 kronos kernel: [751068.768464] RAX: 0000000000000051 RBX: 0000000000000040 RCX: 0000000000000001
Apr 30 18:12:09 kronos kernel: [751068.768469] RDX: 0000000000000000 RSI: ffffffff81c93220 RDI: 0000000000000003
Apr 30 18:12:09 kronos kernel: [751068.768473] RBP: ffff88046962fe50 R08: ffff88047fcd12d0 R09: 0000000000000018
Apr 30 18:12:09 kronos kernel: [751068.768478] R10: 000000000001865d R11: 0000000000079f7d R12: 0000000000000002
Apr 30 18:12:09 kronos kernel: [751068.768482] R13: 0000000000000051 R14: 0000000000000006 R15: ffffffff81c932e8
Apr 30 18:12:09 kronos kernel: [751068.768488] FS:  0000000000000000(0000) GS:ffff88047fcc0000(0000) knlGS:0000000000000000
Apr 30 18:12:09 kronos kernel: [751068.768493] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 30 18:12:09 kronos kernel: [751068.768497] CR2: 00007fff867d9718 CR3: 0000000001c0e000 CR4: 00000000001007e0
Apr 30 18:12:09 kronos kernel: [751068.768501] Stack:
Apr 30 18:12:09 kronos kernel: [751068.768505]  000000036962fe50 ffff88047fcdb000 ffffffff81c93220 0002aa899a7643cc
Apr 30 18:12:09 kronos kernel: [751068.768515]  0000000000000002 ffff88046962fe88 ffffffff815c9570 ffff88046962ff38
Apr 30 18:12:09 kronos kernel: [751068.768525]  ffff88047fcdb000 0000000000000002 0000000000000003 ffffffff81c93220
Apr 30 18:12:09 kronos kernel: [751068.768534] Call Trace:
Apr 30 18:12:09 kronos kernel: [751068.768546]  [<ffffffff815c9570>] cpuidle_enter_state+0x40/0xc0
Apr 30 18:12:09 kronos kernel: [751068.768555]  [<ffffffff815c96a9>] cpuidle_idle_call+0xb9/0x1f0
Apr 30 18:12:09 kronos kernel: [751068.768564]  [<ffffffff8101ceae>] arch_cpu_idle+0xe/0x30
Apr 30 18:12:09 kronos kernel: [751068.768572]  [<ffffffff810beb85>] cpu_startup_entry+0xc5/0x290
Apr 30 18:12:09 kronos kernel: [751068.768591]  [<ffffffff81040fc8>] start_secondary+0x218/0x2c0
Apr 30 18:12:09 kronos kernel: [751068.768608] Code: 48 89 d1 48 2d c8 1f 00 00 0f 01 c8 0f ae f0 65 48 8b 04 25 f0 c7 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d 9a 34 8b 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 12 39 cf 
Apr 30 18:12:09 kronos kernel: [751068.768758] NMI backtrace for cpu 6
Apr 30 18:12:09 kronos kernel: [751068.768765] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 1.399 msecs
Apr 30 18:12:09 kronos kernel: [751068.768779] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G        W    3.13.0-24-generic #46-Ubuntu
Apr 30 18:12:09 kronos kernel: [751068.768784] Hardware name: Supermicro A1SAi/A1SAi, BIOS 1.0b 11/06/2013
Apr 30 18:12:09 kronos kernel: [751068.768790] task: ffff8804696397f0 ti: ffff880469634000 task.ti: ffff880469634000
Apr 30 18:12:09 kronos kernel: [751068.768794] RIP: 0010:[<ffffffff813dfd78>]  [<ffffffff813dfd78>] intel_idle+0xd8/0x140
Apr 30 18:12:09 kronos kernel: [751068.768821] RSP: 0018:ffff880469635e28  EFLAGS: 00000046
Apr 30 18:12:09 kronos kernel: [751068.768837] RAX: 0000000000000051 RBX: 0000000000000040 RCX: 0000000000000001
Apr 30 18:12:09 kronos kernel: [751068.768842] RDX: 0000000000000000 RSI: ffffffff81c93220 RDI: 0000000001c0e000
Apr 30 18:12:09 kronos kernel: [751068.768846] RBP: ffff880469635e50 R08: ffff88047fd912d4 R09: 0000000000000018
Apr 30 18:12:09 kronos kernel: [751068.768851] R10: 00000000000590e8 R11: 00000000000cd1e5 R12: 0000000000000002
Apr 30 18:12:09 kronos kernel: [751068.768855] R13: 0000000000000051 R14: 0000000000000006 R15: ffffffff81c932e8
Apr 30 18:12:09 kronos kernel: [751068.768861] FS:  0000000000000000(0000) GS:ffff88047fd80000(0000) knlGS:0000000000000000
Apr 30 18:12:09 kronos kernel: [751068.768866] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 30 18:12:09 kronos kernel: [751068.768871] CR2: 00007fcae77e6fe0 CR3: 0000000001c0e000 CR4: 00000000001007e0
Apr 30 18:12:09 kronos kernel: [751068.768889] Stack:
Apr 30 18:12:09 kronos kernel: [751068.768903]  0000000669635e50 ffff88047fd9b000 ffffffff81c93220 0002aa898db88408
Apr 30 18:12:09 kronos kernel: [751068.768915]  0000000000000002 ffff880469635e88 ffffffff815c9570 ffff880469635f38
Apr 30 18:12:09 kronos kernel: [751068.768925]  ffff88047fd9b000 0000000000000002 0000000000000006 ffffffff81c93220
Apr 30 18:12:09 kronos kernel: [751068.768934] Call Trace:
Apr 30 18:12:09 kronos kernel: [751068.768959]  [<ffffffff815c9570>] cpuidle_enter_state+0x40/0xc0
Apr 30 18:12:09 kronos kernel: [751068.768972]  [<ffffffff815c96a9>] cpuidle_idle_call+0xb9/0x1f0
Apr 30 18:12:09 kronos kernel: [751068.768983]  [<ffffffff8101ceae>] arch_cpu_idle+0xe/0x30
Apr 30 18:12:09 kronos kernel: [751068.768999]  [<ffffffff810beb85>] cpu_startup_entry+0xc5/0x290
Apr 30 18:12:09 kronos kernel: [751068.769009]  [<ffffffff81040fc8>] start_secondary+0x218/0x2c0
Apr 30 18:12:09 kronos kernel: [751068.769021] Code: 48 89 d1 48 2d c8 1f 00 00 0f 01 c8 0f ae f0 65 48 8b 04 25 f0 c7 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d 9a 34 8b 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 12 39 cf 
Apr 30 18:12:09 kronos kernel: [751068.769102] NMI backtrace for cpu 7
Apr 30 18:12:09 kronos kernel: [751068.769109] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 1.736 msecs
Apr 30 18:12:09 kronos kernel: [751068.769122] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G        W    3.13.0-24-generic #46-Ubuntu
Apr 30 18:12:09 kronos kernel: [751068.769127] Hardware name: Supermicro A1SAi/A1SAi, BIOS 1.0b 11/06/2013
Apr 30 18:12:09 kronos kernel: [751068.769132] task: ffff88046963afe0 ti: ffff880469636000 task.ti: ffff880469636000
Apr 30 18:12:09 kronos kernel: [751068.769137] RIP: 0010:[<ffffffff813dfd78>]  [<ffffffff813dfd78>] intel_idle+0xd8/0x140
Apr 30 18:12:09 kronos kernel: [751068.769156] RSP: 0018:ffff880469637e28  EFLAGS: 00000046
Apr 30 18:12:09 kronos kernel: [751068.769173] RAX: 0000000000000051 RBX: 0000000000000040 RCX: 0000000000000001
Apr 30 18:12:09 kronos kernel: [751068.769183] RDX: 0000000000000000 RSI: ffffffff81c93220 RDI: 0000000000000007
Apr 30 18:12:09 kronos kernel: [751068.769188] RBP: ffff880469637e50 R08: ffff88047fdd12c8 R09: 0000000000000018
Apr 30 18:12:09 kronos kernel: [751068.769192] R10: 0000000000165604 R11: 00000000001dd88c R12: 0000000000000002
Apr 30 18:12:09 kronos kernel: [751068.769197] R13: 0000000000000051 R14: 0000000000000006 R15: ffffffff81c932e8
Apr 30 18:12:09 kronos kernel: [751068.769202] FS:  0000000000000000(0000) GS:ffff88047fdc0000(0000) knlGS:0000000000000000
Apr 30 18:12:09 kronos kernel: [751068.769207] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 30 18:12:09 kronos kernel: [751068.769212] CR2: 00007f40d680f000 CR3: 0000000001c0e000 CR4: 00000000001007e0
Apr 30 18:12:09 kronos kernel: [751068.769215] Stack:
Apr 30 18:12:09 kronos kernel: [751068.769219]  0000000769637e50 ffff88047fddb000 ffffffff81c93220 0002aa893339d20a
Apr 30 18:12:09 kronos kernel: [751068.769230]  0000000000000002 ffff880469637e88 ffffffff815c9570 ffff880469637f38
Apr 30 18:12:09 kronos kernel: [751068.769239]  ffff88047fddb000 0000000000000002 0000000000000007 ffffffff81c93220
Apr 30 18:12:09 kronos kernel: [751068.769276] Call Trace:
Apr 30 18:12:09 kronos kernel: [751068.769291]  [<ffffffff815c9570>] cpuidle_enter_state+0x40/0xc0
Apr 30 18:12:09 kronos kernel: [751068.769300]  [<ffffffff815c96a9>] cpuidle_idle_call+0xb9/0x1f0
Apr 30 18:12:09 kronos kernel: [751068.769309]  [<ffffffff8101ceae>] arch_cpu_idle+0xe/0x30
Apr 30 18:12:09 kronos kernel: [751068.769318]  [<ffffffff810beb85>] cpu_startup_entry+0xc5/0x290
Apr 30 18:12:09 kronos kernel: [751068.769327]  [<ffffffff81040fc8>] start_secondary+0x218/0x2c0
Apr 30 18:12:09 kronos kernel: [751068.769344] Code: 48 89 d1 48 2d c8 1f 00 00 0f 01 c8 0f ae f0 65 48 8b 04 25 f0 c7 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d 9a 34 8b 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 12 39 cf 
Apr 30 18:12:09 kronos kernel: [751068.769426] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 2.055 msecs
Apr 30 18:15:09 kronos kernel: [751248.933567] INFO: rcu_sched detected stalls on CPUs/tasks: { 4} (detected by 0, t=60007 jiffies, g=9072572, c=9072571, q=0)
Apr 30 18:15:09 kronos kernel: [751248.933674] sending NMI to all CPUs:
Apr 30 18:15:09 kronos kernel: [751248.933681] NMI backtrace for cpu 0
Apr 30 18:15:09 kronos kernel: [751248.933685] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.13.0-24-generic #46-Ubuntu
share|improve this question
    
Does plus10.blogspot.co.uk/2013/01/… help ? –  Iain May 1 '14 at 7:25
    
I tried another similar solution : "nosmp". I'm waiting a few days to see if it solves the issue. I'll try pcie_aspm=off however because nosmp causes the server to use only one core so that's not optimal. Thanks ! –  Antares May 1 '14 at 21:04

1 Answer 1

up vote 0 down vote accepted

Problem solved by disabling SMP (added "nosmp" parameter in Grub conf). The server is running fine now.

share|improve this answer
    
That's going to decrease the server performance. But that is of course better than crashing. –  kasperd May 20 '14 at 14:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.