Today my Ubuntu 12.04 server started crashing once an hour. It's actually right at minute :32, every hour. It seems to be an actual kernel crash, i.e. there's nothing in /var/log/syslog, it just stops. I even wrote a script that printed out the time every second to try to nail down the exact time, and watched as the next crash was coming, and the best I can tell is it's about 32 minutes and 13 seconds after the hour.

But I have no CRON jobs running at that time, nothing. The ones I do have are the same Perl scripts I've had running for years that don't do anything special. I haven't changed anything on the server in probably weeks, and definitely nothing important. I've gone through every source of scheduled tasks I can find. Nothing special happens at that time except the server crashes without leaving any trace of why.

When it does crash, I have no choice but to reboot it through the hosting company's website, which has been my solution so far, but now it's 4am and I have to get some sleep sometime. But if I do, my site will go down within an hour. Any help would be appreciated, either a) how to find this task that is running and causing the crash, if that's what it is, or b) how to trace what's happening when it doesn't appear in syslog or dmesg.

SOLVED. I contacted my host. Apparently they installed new server monitoring software that flagged mine as using too much disk I/O. Instead of telling me, this software of theirs simply killed my server. It runs every hour at :32 minutes.

  • Is it physical machine or virtual? If physical, when it locks up, is it hardlock (ie. pressing numlock doesn't blink the light)? Do you run X11 on it or not, and if you do, with what driver? – Matija Nalis Sep 7 '16 at 11:39
  • Time to find a new hosting provider. – ewwhite Sep 7 '16 at 22:11

There might be some other reason.

  1. Is there any specific application (web server) running which consumes more memory (i.e, in case of bad tuning / optimization unable to release memory)

  2. Try to capture all processes which consumes more cpu and memory every minute (might help in your scenario to find the cause) - write some script to capture these information (redirect output to a file) and configure it into crontab.

  • I've tried that by watching top, and it's completely normal up until it stops responding. No abnormal CPU or memory, no strange processes. – felwithe Sep 7 '16 at 15:35
  • You should try process of elimination. So, at about 30 minutes after the hour, stop the cron. Yes, I know you don't think it is the cron, but still do it. If it doesn't crash at 32 minutes, than it was cron, and you can try eliminating half of the cron scripts that run at that time until you find the problematic one. If it STILL crashes with cron disabled, then for next time at 30 minutes you also disable half the daemons you're running, until you find the problematic one. (If system still crashes at 32 even when everything is disabled, than either your system is hacked badly, or you have electrical power problems at that time, or you are VM and physical host does something at that time)

  • install serial console (better) or netconsole (easier). That way when it crashes you might get a log on another machine.

  • as a temporary relief, install watchdog package with softdog module (or better yet hardware one, if your hardware supports it). It won't fix the problem, but might allow you some sleeping time until you fix it

It is probable that some of your old "always worked correctly" scripts are experiencing some problem with new data, or there is too much of it, so they eat up all the resources. You should always setup ulimit restrictions so it doesn't happen. It is also possible that scripts access damaged area of the disk or try to access some other problematic hardware and that this causes the crash. Serial Console should tell...

  • I've tried watching top and nothing strange or different happens right up until the moment it stop responding, so I don't think it can be resources. I also don't have anything running at the 32 minute mark-- nothing. – felwithe Sep 7 '16 at 15:39

Try this watch -n 10 top #(every 10 seconds repeat commands).

best regards

  • How is that supposed to help (especially given the fact that top itself updates itself when left running...)? – Sven Sep 7 '16 at 11:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.