1

This happened on an Ubuntu 14.04 64-bit DigitalOcean Droplet:

Apr 22 02:17:02 localhost CRON[32514]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) 
Apr 22 03:16:59 localhost kernel: [211877.884132] show_signal_msg: 12 callbacks suppressed 
Apr 22 03:16:59 localhost kernel: [211877.884140] python[2376]: segfault at 24 ip 00000000004c278f sp 00007fbad163caa0 error 6 in python2.7[400000+2bd000]

Right around 3:16 AM, the CPU spiked up to 100%, and the machine became unresponsive to ping, SSH and (remote) console:

http://imgur.com/hMSH5SV.png

How could the "callback suppressed" throttling appear before the first segfault?

http://lxr.free-electrons.com/source/arch/x86/mm/fault.c#L727

In general, how could a few segfaults cause the kernel to lock up? The offending process was in 'while :; do python script.py; sleep 1; done' (NOTE sleep), was running with 'nice 10' and had an increased oom_adj so that the kernel would kill it first:

open('/proc/self/oom_adj', 'w').write('10')
os.nice(10)
  • If 12 callbacks from show_signal_msg were suppressed that means that something segfaulted 13 times in a row. Do you have coredumps enabled on that box? – SaveTheRbtz Apr 23 '14 at 6:56
  • ulimit -c is 0. You would suggest setting it to unlimited? How would I use the core dumps to figure out why the machine locks up? – Catalin P Apr 23 '14 at 18:23
  • No, I just suggested that "unresponsiveness" can be connected to codedumps being written to disk. – SaveTheRbtz Apr 24 '14 at 8:22
  • Ah, I doubt it - the CPU was pegged at 100% for several hours: imgur.com/hMSH5SV.png – Catalin P Apr 24 '14 at 21:15
-1

So far looks like this kernel bug:

http://blog.nitrous.io/2014/03/10/stability-and-a-linux-oom-killer-bug.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.