Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Reloading Apache2 caused Ubuntu 11.04 to Hang/Crash.

 Load average was 32

CPU Cores:8

Model: Intel xeon E5620

RAM:16 GB

Kernel :2.6.38-13-generic

Apr 6 09:05:11 406852 kernel: [3715446.033031] general protection fault: 0000 [#1] SMP 
Apr 6 09:05:11 406852 kernel: [3715446.054726] last sysfs file:   /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
Apr 6 09:05:11 406852 kernel: [3715446.097404] CPU 5 
Apr 6 09:05:11 406852 kernel: [3715446.097869] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6t_LOG xt_tcpudp ipt_REDIRECT xt_conntrack iptable_mangle nf_conntrack_ftp ipt_REJECT ipt_LOG xt_limit xt_multiport xt_state ip6table_filter ip6_tables iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables vesafb snd_hda_intel snd_hda_codec psmouse ioatdma snd_hwdep i7core_edac ghes edac_core lp hed dca joydev snd_pcm serio_raw parport snd_timer snd soundcore snd_page_alloc usbhid hid e1000e
Apr 6 09:05:11 406852 kernel: [3715446.279465] 
Apr 6 09:05:11 406852 kernel: [3715446.303429] Pid: 19118, comm: apache2 Not tainted 2.6.38-13-generic #56-Ubuntu Supermicro X8DTL/X8DTL
Apr 6 09:05:11 406852 kernel: [3715446.355544] RIP: 0010:[<ffffffff81054cfa>]  [<ffffffff81054cfa>] task_rq_lock+0x4a/0xa0
Apr 6 09:05:11 406852 kernel: [3715446.411635] RSP: 0018:ffff88060b853da8  EFLAGS: 00010082
Apr 6 09:05:11 406852 kernel: [3715446.440241] RAX: 010021b86505c7ff RBX: 0000000000013d00 RCX: 00000001162d8937
Apr 6 09:05:11 406852 kernel: [3715446.497492] RDX: 0000000000000282 RSI: ffff88060b853df0 RDI: 00007fdac0088280
Apr 6 09:05:11 406852 kernel: [3715446.559362] RBP: ffff88060b853dc8 R08: 0000000000000040 R09: 001fc00000000000
Apr 6 09:05:11 406852 kernel: [3715446.625144] R10: 0000000000000000 R11: dead000000100100 R12: 00007fdac0088280
Apr 6 09:05:11 406852 kernel: [3715446.695569] R13: ffff88060b853df0 R14: 0000000000013d00 R15: 0000000000000005
Apr 6 09:05:11 406852 kernel: [3715446.770654] FS:  00007fdac0023760(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000
Apr 6 09:05:11 406852 kernel: [3715446.849786] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 6 09:05:11 406852 kernel: [3715446.889882] CR2: 00007fdac187ca80 CR3: 000000058cda1000 CR4: 00000000000006e0
Apr 6 09:05:11 406852 kernel: [3715446.968627] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 6 09:05:11 406852 kernel: [3715447.049676] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr 6 09:05:11 406852 kernel: [3715447.130842] Process apache2 (pid: 19118, threadinfo ffff88060b852000, task ffff88058c11c4a0)
 Apr 6 09:05:11 406852 kernel: [3715447.212160] Stack:
Apr 6 09:05:11 406852 kernel: [3715447.251311]  00007fdac0088280 ffff880be1ca5ec8 000000000000000f 0000000000000000
Apr 6 09:05:11 406852 kernel: [3715447.331017]  ffff88060b853e28 ffffffff8105f2e1 0000000000000000 0000000081a4c270
Apr 6 09:05:11 406852 kernel: [3715447.412179]  ffff88060b853e38 0000000000000282 0000000000000021 ffff880b92505ec8
Apr 6 09:05:11 406852 kernel: [3715447.493302] Call Trace:
 Apr 6 09:05:11 406852 kernel: [3715447.533014]  [<ffffffff8105f2e1>] try_to_wake_up+0x31/0x3e0
Apr 6 09:05:11 406852 kernel: [3715447.573262]  [<ffffffff8105f6c5>] wake_up_process+0x15/0x20
Apr 6 09:05:11 406852 kernel: [3715447.612669]  [<ffffffff8126b7c7>] wake_up_sem_queue_do+0x37/0x60
Apr 6 09:05:11 406852 kernel: [3715447.651327]  [<ffffffff8126c236>] freeary+0x1c6/0x200
Apr 6 09:05:11 406852 kernel: [3715447.689083]  [<ffffffff8126c32b>] semctl_down.clone.5+0xbb/0x110
Apr 6 09:05:11 406852 kernel: [3715447.726360]  [<ffffffff8107b6ae>] ? sys_kill+0x7e/0x90
Apr 6 09:05:11 406852 kernel: [3715447.762833]  [<ffffffff811663f5>] ? fput+0x25/0x30
Apr 6 09:05:11 406852 kernel: [3715447.798362]  [<ffffffff8126d05e>] sys_semctl+0x7e/0xd0
Apr 6 09:05:11 406852 kernel: [3715447.833126]  [<ffffffff8100c002>] system_call_fastpath+0x16/0x1b
Apr 6 09:05:11 406852 kernel: [3715447.867350] Code: 00 48 c7 c3 00 3d 01 00 49 89 fc 49 89 f5 9c 58 0f 1f 44 00 00 48 89 c2 fa 66 0f 1f 44 00 00 49 89 55 00 49 8b 44 24 08 49 89 de <8b> 40 18 4c 03 34 c5 80 c8 aa 81 4c 89 f7 e8 53 4e 57 00 49 8b 
 Apr 6 09:05:11 406852 kernel: [3715447.970388] RIP  [<ffffffff81054cfa>] task_rq_lock+0x4a/0xa0
 Apr 6 09:05:11 406852 kernel: [3715448.004042]  RSP <ffff88060b853da8>
 Apr 6 09:05:11 406852 kernel: [3715448.083219] ---[ end trace 244a1ec2d6f912fa ]---

I understood these things from above logs,may not be correct:

  • Cause Related to Apache as it was showing Apache process id.
  • Happened on CPU 5.
  • Kernel is not Tainted
  • Since General Protection Fault,hard reset was required to bring it back.

Can any one please help me in Analyzing this stack trace further more with reason.

share|improve this question
    
Did you put an entry in .htaccess before you restarted? i was also facing the same issue. After editing .htaccess and do a restart the server get crash.Cpu load get increase too. What we did was we moved to a stable version of ubuntu. –  ZVIK Apr 8 '13 at 6:37
    
yes exactly editing .htaccess and reloading crashed server,but apache restart didn't cause any crash.Do you have any idea about debugging stack trace. –  krypto Apr 8 '13 at 8:07
add comment

1 Answer 1

up vote 1 down vote accepted

Looks like a race in Linux kernel between semtimedop() timeout and semctl(2) with IPC_RMID: https://bugzilla.kernel.org/show_bug.cgi?id=27142

Ubuntu Kernel git says that fix was merged in Ubuntu-2.6.38-13.56-7-ge635bd0 on 5 Mar 2012.

Upgrading your kernel to (at least) 2.6.38-14-generic (or moving to LTS release on your production servers) should fix your problem.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.