4

A few months ago our server started to crash every 14 days, at the exact same time(around 11:04 every time). We're pretty sure this isn't some kind of hardware failure, as hardware failures tend to be random.

The server just stops responding in a sudden and reboots itself after a few seconds. None of the logs contain any related info and we're 100% sure there is no cron on the server that could cause this.

Has anyone ever faced this kind of problem? We're extremely frustrated about this wired behavior since there's not even a single clue of what's wrong...

I've also taken a video right before the server crashes, as you can see from it, nothing seemed wrong...

Update 11-Apr-2011 :

2 Weeks ago : In order to narrow down possibilities, the server was shutdown (shutdown -h now) 5 minutes before the next occurrence. And magically, the server booted by itself at the expected time. After that our DC moved the server to another PDU port, we thought that would finally solve our issue.

Today : The server still crashed, at the exact same time!! Our DC said other servers on the same PDU does not have this issue. Now we're really confused, if its not the PDU nor our server, what could it be?

Improve this question
9
  • does it reboot, or does it crash ? - you could try chmod -x reboot/halt/shutdown etc just to see if something is doing it deliberately.
    – Sirex
    Mar 21, 2011 at 12:54
  • Login an hour before it will happen the next time and run top or something?
    – Zoredache
    Mar 21, 2011 at 16:31
  • @Sirex Yep already done that. Please check the video I posted it didn't output anything before the server crashes/reboots, so it doesn't seem like a normal shutdown, more like a power-loss?
    – user75228
    Mar 21, 2011 at 18:14
  • @Zoredache Please check the video I posted, I took the video with top running 10 seconds before it crashes..
    – user75228
    Mar 21, 2011 at 18:16
  • 1
    I think the main problem of video is we cannot see whole end, because it is not tty, just remote.
    – Ency
    Mar 21, 2011 at 20:18

7 Answers 7

Reset to default
2

From the video it seems like a cold reboot. And as you said, nothing in the logs. All I can think of is the sysrq "magic" key or a faulty kvm card if no other servers using the same UPS are experiencing the same.

A bugged/faulty system monitoring process could be doing this at specific days/hours. This should be fun to track down.

First step would be to change the date and time of the OS and see if it reboots on its own so you can narrow it down.

Improve this answer
2

I had the same situation where both server-power-cords connected to the same ups. After looking at the ups logs the reset did happen when the ups did it`s selv test - every 14 days.

Solution: Plug one power-cord into another ups or connect it directly.

Improve this answer
0

What exactly do you mean by the "exact same time"?

Assuming you've eliminated any scheduled jobs (by changing the times thay all run at - not just looking at the logs) then top of my list would be looking at the UPS logs. You do have a UPS don't you?

Improve this answer
4
  • Although our server is placed behind a UPS, unfortunately we don't have access to the UPS logs since multiple servers (that don't belong to us) are sharing the same UPS. However we did inform our DC about our situation and they still insist there's nothing wrong on their end.
    – user75228
    Mar 21, 2011 at 18:10
  • Oh and "exact same time" means it *always crashes every 14 days on 11:04.
    – user75228
    Mar 21, 2011 at 18:12
  • So to diagnose, does that mean it crashes every x number of hours, which happens to look like 'every 14 days at 11.04' because that's how your restarts end up, or does it mean that you can switch it off after a crash, leave it for 14 days, bring it up at 10am and it will crash 1 hour and 4 mins later (or for that matter, shut it down at 11 and bring it up at 11.10 and not have a problem at all?) Because if its the latter then its environmental. If the former then probably something internal. If you can figure out which one it is then you might have a pointer to working out the solution.
    – Rob Moir
    Mar 21, 2011 at 19:38
  • Yes the server crashes and reboots approx every 336 hours (equal to 14 days), after the server stops responding (as you can see from the video), it then reboots and everything becomes back to normal for another 14 days, over and over again. We haven't tried shutting doen the server before it crashes, we'll try that and see what happens.
    – user75228
    Mar 22, 2011 at 1:23
0

I've actually seen someone do this in cron, just to drive up maintenance calls. You definitely need to check and make sure nothing is explicitly scheduled in the system that would cause this sort of issue.

What do the system logs say?

Improve this answer
1
  • Umm.. what log do you mean? We've already check all the logs we could think of, and none of them seemed suspicious. Well in other words, the server didn't log anything when it crashed/rebooted. We could possibly miss something however, please be more specific so we can see what we missed, thanks.
    – user75228
    Mar 21, 2011 at 18:22
0

I had an IBM server that would crash every 76 days. Drove me absolutely mad with frustration trying to figure it out. Turned out to be a clock issue on one of the system cards (http://communities.vmware.com/thread/9359). Be sure to check if anyone has experienced similar issues with the make and model server just in case.

Improve this answer
0

If server has external BMC, check BMC logs. Maybe BMC timer is configured for 24 hours and it is not being resetted by OS (still many BMCs try graceful shutdown first)

Improve this answer
0

First, try turning off crond the day of the crash. (I suspect that a cron job at 11am that takes four minutes to run triggers a kernel bug or hardware fault.)

Also, try retarding the system clock by three minutes, to test whether the problem is caused by something within the server or external to the server.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .