Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

This happened on an Ubuntu 14.04 64-bit DigitalOcean Droplet:

Apr 22 02:17:02 localhost CRON[32514]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) 
Apr 22 03:16:59 localhost kernel: [211877.884132] show_signal_msg: 12 callbacks suppressed 
Apr 22 03:16:59 localhost kernel: [211877.884140] python[2376]: segfault at 24 ip 00000000004c278f sp 00007fbad163caa0 error 6 in python2.7[400000+2bd000]

Right around 3:16 AM, the CPU spiked up to 100%, and the machine became unresponsive to ping, SSH and (remote) console:

http://imgur.com/hMSH5SV.png

How could the "callback suppressed" throttling appear before the first segfault?

http://lxr.free-electrons.com/source/arch/x86/mm/fault.c#L727

In general, how could a few segfaults cause the kernel to lock up? The offending process was in 'while :; do python script.py; sleep 1; done' (NOTE sleep), was running with 'nice 10' and had an increased oom_adj so that the kernel would kill it first:

open('/proc/self/oom_adj', 'w').write('10')
os.nice(10)
share|improve this question
    
If 12 callbacks from show_signal_msg were suppressed that means that something segfaulted 13 times in a row. Do you have coredumps enabled on that box? –  SaveTheRbtz Apr 23 '14 at 6:56
    
ulimit -c is 0. You would suggest setting it to unlimited? How would I use the core dumps to figure out why the machine locks up? –  Catalin Patulea Apr 23 '14 at 18:23
    
No, I just suggested that "unresponsiveness" can be connected to codedumps being written to disk. –  SaveTheRbtz Apr 24 '14 at 8:22
    
Ah, I doubt it - the CPU was pegged at 100% for several hours: imgur.com/hMSH5SV.png –  Catalin Patulea Apr 24 '14 at 21:15

1 Answer 1

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.