2

Our server is having load average of 44,before we lost ssh connection to that.

Load average increased all of a sudden and server response time increased and finally ssh connection got terminated.Finally hard rebooted machine and after that everything came normal.

on system logs i can see following

 Feb 10 20:34:11 406852 kernel: [3715446.033031] general protection fault: 0000 [#1] SMP 
Feb 10 20:34:11 406852 kernel: [3715446.054726] last sysfs file: sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
  Feb 10 20:34:11 406852 kernel: [3715446.097404] CPU 5 
Feb 10 20:34:11 406852 kernel: [3715446.097869] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6t_LOG xt_tcpudp ipt_REDIRECT xt_conntrack iptable_mangle nf_conntrack_ftp ipt_REJECT ipt_LOG xt_limit xt_multiport xt_state ip6table_filter ip6_tables iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables vesafb snd_hda_intel snd_hda_codec psmouse ioatdma snd_hwdep i7core_edac ghes edac_core lp hed dca joydev snd_pcm serio_raw parport snd_timer snd soundcore snd_page_alloc usbhid hid e1000e
 Feb 10 20:34:11 406852 kernel: [3715446.279465] 
Feb 10 20:34:11 406852 kernel: [3715446.303429] Pid: 19118, comm: apache2 Not tainted 2.6.38-13-generic #56-Ubuntu Supermicro X8DTL/X8DTL
Feb 10 20:34:11 406852 kernel: [3715446.355544] RIP: 0010:[<ffffffff81054cfa>]  [<ffffffff81054cfa>] task_rq_lock+0x4a/0xa0
Feb 10 20:34:11 406852 kernel: [3715446.411635] RSP: 0018:ffff88060b853da8  EFLAGS: 00010082
Feb 10 20:34:11 406852 kernel: [3715446.440241] RAX: 010021b86505c7ff RBX: 0000000000013d00 RCX: 00000001162d8937
Feb 10 20:34:11 406852 kernel: [3715446.497492] RDX: 0000000000000282 RSI: ffff88060b853df0 RDI: 00007fdac0088280
Feb 10 20:34:11 406852 kernel: [3715446.559362] RBP: ffff88060b853dc8 R08: 0000000000000040 R09: 001fc00000000000
Feb 10 20:34:11 406852 kernel: [3715446.625144] R10: 0000000000000000 R11: dead000000100100 R12: 00007fdac0088280
Feb 10 20:34:11 406852 kernel: [3715446.695569] R13: ffff88060b853df0 R14: 0000000000013d00 R15: 0000000000000005
Feb 10 20:34:11 406852 kernel: [3715446.770654] FS:  00007fdac0023760(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000
Feb 10 20:34:11 406852 kernel: [3715446.849786] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 Feb 10 20:34:11 406852 kernel: [3715446.889882] CR2: 00007fdac187ca80 CR3: 000000058cda1000 CR4: 00000000000006e0
 Feb 10 20:34:11 406852 kernel: [3715446.968627] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 Feb 10 20:34:11 406852 kernel: [3715447.049676] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
 Feb 10 20:34:11 406852 kernel: [3715447.130842] Process apache2 (pid: 19118, threadinfo ffff88060b852000, task ffff88058c11c4a0)
 Feb 10 20:34:11 406852 kernel: [3715447.212160] Stack:
 Feb 10 20:34:11 406852 kernel: [3715447.251311]  00007fdac0088280 ffff880be1ca5ec8 000000000000000f 0000000000000000
  Feb 10 20:34:11 406852 kernel: [3715447.331017]  ffff88060b853e28 ffffffff8105f2e1 0000000000000000 0000000081a4c270
  Feb 10 20:34:11 406852 kernel: [3715447.412179]  ffff88060b853e38 0000000000000282 0000000000000021 ffff880b92505ec8
  Feb 10 20:34:11 406852 kernel: [3715447.493302] Call Trace:
 Feb 10 20:34:11 406852 kernel: [3715447.533014]  [<ffffffff8105f2e1>] try_to_wake_up+0x31/0x3e0
Feb 10 20:34:11 406852 kernel: [3715447.573262]  [<ffffffff8105f6c5>] wake_up_process+0x15/0x20
Feb 10 20:34:11 406852 kernel: [3715447.612669]  [<ffffffff8126b7c7>] wake_up_sem_queue_do+0x37/0x60
Feb 10 20:34:11 406852 kernel: [3715447.651327]  [<ffffffff8126c236>] freeary+0x1c6/0x200
Feb 10 20:34:11 406852 kernel: [3715447.689083]  [<ffffffff8126c32b>] semctl_down.clone.5+0xbb/0x110
Feb 10 20:34:11 406852 kernel: [3715447.726360]  [<ffffffff8107b6ae>] ? sys_kill+0x7e/0x90
Feb 10 20:34:11 406852 kernel: [3715447.762833]  [<ffffffff811663f5>] ? fput+0x25/0x30
Feb 10 20:34:11 406852 kernel: [3715447.798362]  [<ffffffff8126d05e>] sys_semctl+0x7e/0xd0
Feb 10 20:34:11 406852 kernel: [3715447.833126]  [<ffffffff8100c002>] system_call_fastpath+0x16/0x1b
Feb 10 20:34:11 406852 kernel: [3715447.867350] Code: 00 48 c7 c3 00 3d 01 00 49 89 fc 49 89 f5 9c 58 0f 1f 44 00 00 48 89 c2 fa 66 0f 1f 44 00 00 49 89 55 00 49 8b 44 24 08 49 89 de <8b> 40 18 4c 03 34 c5 80 c8 aa 81 4c 89 f7 e8 53 4e 57 00 49 8b 
 Feb 10 20:34:11 406852 kernel: [3715447.970388] RIP  [<ffffffff81054cfa>] task_rq_lock+0x4a/0xa0
 Feb 10 20:34:11 406852 kernel: [3715448.004042]  RSP <ffff88060b853da8>
 Feb 10 20:34:11 406852 kernel: [3715448.083219] ---[ end trace 244a1ec2d6f912fa ]---

As per wiki http://en.wikipedia.org/wiki/General_protection_fault CPU become non responsive and will respond only to a hard reset. if this is the case then isn't it obvious that scheduler queue processes and thus load average increases?

If CPU became non responsive then how could ssh and top command work?

4

All I can offer an answer to is how some stuff continues to work if a CPU locks up. Your output above makes it clear that CPU number 5 has locked up, which almost certainly means you've got at least eight cores. Seven of those haven't locked up, so the system will continue to limp along, at least for a while; but if there was anything vital that's now stuck on CPU5, then jobs on other cores will deadlock when they hit a need for a resource that's stuck on #5.

My limited experience is that these deadlocked jobs often stay on the run queue, and contribute therefore to the load count, while they wait endlessly for a resource that can never come. Usually, enough of the system eventually ends up in that state that it becomes unresponsive and has to be rebooted.

As for why this is happening, I don't know, though I'd suspect a bug first and hardware second. Make sure your BIOS is fully-updated, that your OS is current, and up-to-patch, especially as regards the kernel. If you do that and it still happens regularly, log a hardware support call.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.