5

On a semi-regular basis, I've seen GCE instances freezing with the following error message (from the serial console):

g[1375589.784755] BUG: unable to handle kernel NULL pointer dereference at 0000000000000078
g[1375589.786206] IP: [<ffffffff810a67d9>] check_preempt_wakeup+0xd9/0x1d0
g[1375589.787341] PGD 5da04067 PUD db83067 PMD 0 
g[1375589.788607] Oops: 0000 [#1] SMP 
g[1375589.788705] Modules linked in: veth xt_addrtype xt_conntrack iptable_filter ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip_tables x_tables nf_nat nf_conntrack bridge stp llc aufs(C) softdog crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 processor psmouse parport_pc parport i2c_piix4 i2c_core thermal_sys lrw virtio_net evdev pcspkr serio_raw gf128mul glue_helper ablk_helper cryptd button ext4 crc16 mbcache jbd2 sd_mod crc_t10dif crct10dif_common virtio_scsi scsi_mod virtio_pci virtio virtio_ring
g[1375589.788705] CPU: 1 PID: 1515 Comm: docker Tainted: G         C    3.16.0-0.bpo.4-amd64 #1 Debian 3.16.7-ckt9-3~deb8u1~bpo70+1
g[1375589.788705] Hardware name: Google Google, BIOS Google 01/01/2011
g[1375589.788705] task: ffff88006fffc110 ti: ffff880003ac4000 task.ti: ffff880003ac4000
g[1375589.788705] RIP: 0010:[<ffffffff810a67d9>]  [<ffffffff810a67d9>] check_preempt_wakeup+0xd9/0x1d0
g[1375589.788705] RSP: 0018:ffff880003ac7e30  EFLAGS: 00010002
g[1375589.788705] RAX: 0000000000000001 RBX: ffff880073112ec0 RCX: 0000000000000002
g[1375589.788705] RDX: 0000000000000001 RSI: ffff880009156d20 RDI: ffff880073112f38
g[1375589.788705] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
g[1375589.788705] R10: ffffffffffffffe0 R11: 0000000000000000 R12: ffff88006d2dcd00
g[1375589.788705] R13: ffff88006fffc110 R14: 0000000000000000 R15: 0000000000000000
g[1375589.788705] FS:  000000000323a880(0063) GS:ffff880073100000(0000) knlGS:0000000000000000
g[1375589.788705] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
g[1375589.788705] CR2: 0000000000000078 CR3: 0000000034bff000 CR4: 00000000000406e0
g[1375589.788705] Stack:
g[1375589.788705]  0000000000000000 ffffffff00000000 ffff88000000006e ffff880073112ec0
g[1375589.788705]  ffff8800091573a4 0000000000000286 0000000000012ec0 ffff880073112ec0
g[1375589.788705]  0000000000000002 ffffffff8109cef4 ffff880009156d20 ffffffff810a01a4
g[1375589.788705] Call Trace:
g[1375589.788705]  [<ffffffff8109cef4>] ? check_preempt_curr+0x84/0xa0
g[1375589.788705]  [<ffffffff810a01a4>] ? wake_up_new_task+0xf4/0x1b0
g[1375589.788705]  [<ffffffff8118516d>] ? mprotect_fixup+0x15d/0x250
g[1375589.788705]  [<ffffffff8106d10f>] ? do_fork+0xcf/0x340
g[1375589.788705]  [<ffffffff8154b779>] ? stub_clone+0x69/0x90
g[1375589.788705]  [<ffffffff8154b40d>] ? system_call_fast_compare_end+0x10/0x15
g[1375589.788705] Code: 00 00 83 e8 01 4d 8b 64 24 70 39 d0 7f f4 48 8b 7d 78 49 3b 7c 24 78 74 1d 66 0f 1f 84 00 00 00 00 00 48 8b 6d 70 4d 8b 64 24 70 <48> 8b 7d 78 49 3b 7c 24 78 75 ec 48 85 ff 74 e7 e8 f2 f9 ff ff 
g[1375589.788705] RIP  [<ffffffff810a67d9>] check_preempt_wakeup+0xd9/0x1d0
g[1375589.788705]  RSP <ffff880003ac7e30>
g[1375589.788705] CR2: 0000000000000078
g[1375589.788705] ---[ end trace 5fab7713cb2d171f ]---

The only way I've been able to restore them is to login to the web interface and manually reset them. Needless to say, it doesn't scale.

I've already tried setting up a watchdog device and setting kernel.panic = 10, which in theory should reboot the VM.

For these VMs, I'm using the 'container-vm' as the OS flavor (i.e. Debian with Docker-preinstalled more or less).

Have anyone else seen this?

11
  • check_preempt_wakeup() is in the fair scheduler. You may want to try using a different scheduler. Also, while that shouldn't be a problem, this seems to be somehow related to the docker process. Make sure that one is working properly (are the files okay on disk?). – user Jul 31 '15 at 9:53
  • Interesting lead. I know that in some instances, the disk did get corrupt as I had to attach the disk on a different VM to restore it with an fsck run. Am I correct in assuming that I'd have to recompile the kernel in order to switch scheduler? – vpetersson Jul 31 '15 at 10:02
  • I'm not sure; it might be possible to do through procfs or sysfs. You may want to have a look at How to verify that package-installed files match originals?, too. – user Jul 31 '15 at 11:04
  • It doesn't seem like it's possible to change during run-time. That said, I have a new lead. For these machines we used the --cpu-shares feature in Docker. I'm trying now to remove it to see if that changes anything. – vpetersson Aug 3 '15 at 10:20
  • 1
    Can you please file an issue on this link (code.google.com/p/google-compute-engine/issues/list) with details and steps to reproduce the issue. Once we receive the report I'll go ahead and try to reproduce it and will get in touch with engineering team. – Faizan Aug 14 '15 at 21:27
3

I don't have enough reputation to comment. So I put my comment here. I'm having the same issue. I checked the Internet for the bug reports and found the almost every kernel output contains do_fork() function in it. After that I found that:

http://www.serverphorums.com/read.php?12,1053418

And update version here:

https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/kernel/sched/core.c?id=ea86cb4b7621e1298a37197005bf0abcc86348d4

I hope it helps someone.

I'd like to have this fixed in my distro, but I don't know how to push the distro guys to put this patch to the default kernel.

1
  • I think it's an illness from 3.16 downwards and is completely solved from 3.19 onwards. A possible solution is to install a newer kernel (from BPO at Debian, for example). – emi May 26 '17 at 8:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.