5

On a semi-regular basis, I've seen GCE instances freezing with the following error message (from the serial console):

g[1375589.784755] BUG: unable to handle kernel NULL pointer dereference at 0000000000000078
g[1375589.786206] IP: [<ffffffff810a67d9>] check_preempt_wakeup+0xd9/0x1d0
g[1375589.787341] PGD 5da04067 PUD db83067 PMD 0 
g[1375589.788607] Oops: 0000 [#1] SMP 
g[1375589.788705] Modules linked in: veth xt_addrtype xt_conntrack iptable_filter ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip_tables x_tables nf_nat nf_conntrack bridge stp llc aufs(C) softdog crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 processor psmouse parport_pc parport i2c_piix4 i2c_core thermal_sys lrw virtio_net evdev pcspkr serio_raw gf128mul glue_helper ablk_helper cryptd button ext4 crc16 mbcache jbd2 sd_mod crc_t10dif crct10dif_common virtio_scsi scsi_mod virtio_pci virtio virtio_ring
g[1375589.788705] CPU: 1 PID: 1515 Comm: docker Tainted: G         C    3.16.0-0.bpo.4-amd64 #1 Debian 3.16.7-ckt9-3~deb8u1~bpo70+1
g[1375589.788705] Hardware name: Google Google, BIOS Google 01/01/2011
g[1375589.788705] task: ffff88006fffc110 ti: ffff880003ac4000 task.ti: ffff880003ac4000
g[1375589.788705] RIP: 0010:[<ffffffff810a67d9>]  [<ffffffff810a67d9>] check_preempt_wakeup+0xd9/0x1d0
g[1375589.788705] RSP: 0018:ffff880003ac7e30  EFLAGS: 00010002
g[1375589.788705] RAX: 0000000000000001 RBX: ffff880073112ec0 RCX: 0000000000000002
g[1375589.788705] RDX: 0000000000000001 RSI: ffff880009156d20 RDI: ffff880073112f38
g[1375589.788705] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
g[1375589.788705] R10: ffffffffffffffe0 R11: 0000000000000000 R12: ffff88006d2dcd00
g[1375589.788705] R13: ffff88006fffc110 R14: 0000000000000000 R15: 0000000000000000
g[1375589.788705] FS:  000000000323a880(0063) GS:ffff880073100000(0000) knlGS:0000000000000000
g[1375589.788705] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
g[1375589.788705] CR2: 0000000000000078 CR3: 0000000034bff000 CR4: 00000000000406e0
g[1375589.788705] Stack:
g[1375589.788705]  0000000000000000 ffffffff00000000 ffff88000000006e ffff880073112ec0
g[1375589.788705]  ffff8800091573a4 0000000000000286 0000000000012ec0 ffff880073112ec0
g[1375589.788705]  0000000000000002 ffffffff8109cef4 ffff880009156d20 ffffffff810a01a4
g[1375589.788705] Call Trace:
g[1375589.788705]  [<ffffffff8109cef4>] ? check_preempt_curr+0x84/0xa0
g[1375589.788705]  [<ffffffff810a01a4>] ? wake_up_new_task+0xf4/0x1b0
g[1375589.788705]  [<ffffffff8118516d>] ? mprotect_fixup+0x15d/0x250
g[1375589.788705]  [<ffffffff8106d10f>] ? do_fork+0xcf/0x340
g[1375589.788705]  [<ffffffff8154b779>] ? stub_clone+0x69/0x90
g[1375589.788705]  [<ffffffff8154b40d>] ? system_call_fast_compare_end+0x10/0x15
g[1375589.788705] Code: 00 00 83 e8 01 4d 8b 64 24 70 39 d0 7f f4 48 8b 7d 78 49 3b 7c 24 78 74 1d 66 0f 1f 84 00 00 00 00 00 48 8b 6d 70 4d 8b 64 24 70 <48> 8b 7d 78 49 3b 7c 24 78 75 ec 48 85 ff 74 e7 e8 f2 f9 ff ff 
g[1375589.788705] RIP  [<ffffffff810a67d9>] check_preempt_wakeup+0xd9/0x1d0
g[1375589.788705]  RSP <ffff880003ac7e30>
g[1375589.788705] CR2: 0000000000000078
g[1375589.788705] ---[ end trace 5fab7713cb2d171f ]---

The only way I've been able to restore them is to login to the web interface and manually reset them. Needless to say, it doesn't scale.

I've already tried setting up a watchdog device and setting kernel.panic = 10, which in theory should reboot the VM.

For these VMs, I'm using the 'container-vm' as the OS flavor (i.e. Debian with Docker-preinstalled more or less).

Have anyone else seen this?

Improve this question
11
  • check_preempt_wakeup() is in the fair scheduler. You may want to try using a different scheduler. Also, while that shouldn't be a problem, this seems to be somehow related to the docker process. Make sure that one is working properly (are the files okay on disk?).
    – user
    Jul 31, 2015 at 9:53
  • Interesting lead. I know that in some instances, the disk did get corrupt as I had to attach the disk on a different VM to restore it with an fsck run. Am I correct in assuming that I'd have to recompile the kernel in order to switch scheduler?
    – vpetersson
    Jul 31, 2015 at 10:02
  • I'm not sure; it might be possible to do through procfs or sysfs. You may want to have a look at How to verify that package-installed files match originals?, too.
    – user
    Jul 31, 2015 at 11:04
  • It doesn't seem like it's possible to change during run-time. That said, I have a new lead. For these machines we used the --cpu-shares feature in Docker. I'm trying now to remove it to see if that changes anything.
    – vpetersson
    Aug 3, 2015 at 10:20
  • 1
    Can you please file an issue on this link (code.google.com/p/google-compute-engine/issues/list) with details and steps to reproduce the issue. Once we receive the report I'll go ahead and try to reproduce it and will get in touch with engineering team.
    – Faizan
    Aug 14, 2015 at 21:27

1 Answer 1

Reset to default
3

I don't have enough reputation to comment. So I put my comment here. I'm having the same issue. I checked the Internet for the bug reports and found the almost every kernel output contains do_fork() function in it. After that I found that:

http://www.serverphorums.com/read.php?12,1053418

And update version here:

https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/kernel/sched/core.c?id=ea86cb4b7621e1298a37197005bf0abcc86348d4

I hope it helps someone.

I'd like to have this fixed in my distro, but I don't know how to push the distro guys to put this patch to the default kernel.

Improve this answer
1
  • I think it's an illness from 3.16 downwards and is completely solved from 3.19 onwards. A possible solution is to install a newer kernel (from BPO at Debian, for example).
    – emi
    May 26, 2017 at 8:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .