Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

A few months ago our server started to crash every 14 days, at the exact same time(around 11:04 every time). We're pretty sure this isn't some kind of hardware failure, as hardware failures tend to be random.

The server just stops responding in a sudden and reboots itself after a few seconds. None of the logs contain any related info and we're 100% sure there is no cron on the server that could cause this.

Has anyone ever faced this kind of problem? We're extremely frustrated about this wired behavior since there's not even a single clue of what's wrong...

I've also taken a video right before the server crashes, as you can see from it, nothing seemed wrong...

Update 11-Apr-2011 :

2 Weeks ago : In order to narrow down possibilities, the server was shutdown (shutdown -h now) 5 minutes before the next occurrence. And magically, the server booted by itself at the expected time. After that our DC moved the server to another PDU port, we thought that would finally solve our issue.

Today : The server still crashed, at the exact same time!! Our DC said other servers on the same PDU does not have this issue. Now we're really confused, if its not the PDU nor our server, what could it be?

share|improve this question
    
does it reboot, or does it crash ? - you could try chmod -x reboot/halt/shutdown etc just to see if something is doing it deliberately. –  Sirex Mar 21 '11 at 12:54
    
Login an hour before it will happen the next time and run top or something? –  Zoredache Mar 21 '11 at 16:31
    
@Sirex Yep already done that. Please check the video I posted it didn't output anything before the server crashes/reboots, so it doesn't seem like a normal shutdown, more like a power-loss? –  user75228 Mar 21 '11 at 18:14
    
@Zoredache Please check the video I posted, I took the video with top running 10 seconds before it crashes.. –  user75228 Mar 21 '11 at 18:16
1  
I think the main problem of video is we cannot see whole end, because it is not tty, just remote. –  Ency Mar 21 '11 at 20:18
show 2 more comments

9 Answers

I had the same situation where both server-power-cords connected to the same ups. After looking at the ups logs the reset did happen when the ups did it`s selv test - every 14 days.

Solution: Plug one power-cord into another ups or connect it directly.

share|improve this answer
add comment

From the video it seems like a cold reboot. And as you said, nothing in the logs. All I can think of is the sysrq "magic" key or a faulty kvm card if no other servers using the same UPS are experiencing the same.

A bugged/faulty system monitoring process could be doing this at specific days/hours. This should be fun to track down.

First step would be to change the date and time of the OS and see if it reboots on its own so you can narrow it down.

share|improve this answer
add comment

What exactly do you mean by the "exact same time"?

Assuming you've eliminated any scheduled jobs (by changing the times thay all run at - not just looking at the logs) then top of my list would be looking at the UPS logs. You do have a UPS don't you?

share|improve this answer
    
Although our server is placed behind a UPS, unfortunately we don't have access to the UPS logs since multiple servers (that don't belong to us) are sharing the same UPS. However we did inform our DC about our situation and they still insist there's nothing wrong on their end. –  user75228 Mar 21 '11 at 18:10
    
Oh and "exact same time" means it *always crashes every 14 days on 11:04. –  user75228 Mar 21 '11 at 18:12
    
So to diagnose, does that mean it crashes every x number of hours, which happens to look like 'every 14 days at 11.04' because that's how your restarts end up, or does it mean that you can switch it off after a crash, leave it for 14 days, bring it up at 10am and it will crash 1 hour and 4 mins later (or for that matter, shut it down at 11 and bring it up at 11.10 and not have a problem at all?) Because if its the latter then its environmental. If the former then probably something internal. If you can figure out which one it is then you might have a pointer to working out the solution. –  RobM Mar 21 '11 at 19:38
    
Yes the server crashes and reboots approx every 336 hours (equal to 14 days), after the server stops responding (as you can see from the video), it then reboots and everything becomes back to normal for another 14 days, over and over again. We haven't tried shutting doen the server before it crashes, we'll try that and see what happens. –  user75228 Mar 22 '11 at 1:23
add comment

I've actually seen someone do this in cron, just to drive up maintenance calls. You definitely need to check and make sure nothing is explicitly scheduled in the system that would cause this sort of issue.

What do the system logs say?

share|improve this answer
    
Umm.. what log do you mean? We've already check all the logs we could think of, and none of them seemed suspicious. Well in other words, the server didn't log anything when it crashed/rebooted. We could possibly miss something however, please be more specific so we can see what we missed, thanks. –  user75228 Mar 21 '11 at 18:22
add comment

only a comment.. I recall this same issue with a Windows physical server, it would reboot itself at exactly the same time every two weeks. (The client would not let us bring it down to see it takes every x hours coz it was a 'Production' box) Turns out there was this rogue 3rd party application which would run some scripts scheduled at that time and eventually crash the system.. sorry I dont recall what app it was though.

share|improve this answer
add comment

I had an IBM server that would crash every 76 days. Drove me absolutely mad with frustration trying to figure it out. Turned out to be a clock issue on one of the system cards (http://communities.vmware.com/thread/9359). Be sure to check if anyone has experienced similar issues with the make and model server just in case.

share|improve this answer
add comment

Maybe it's a Layer 8 Problem. I heard something similar from a friend where they had a crash every week at about the same time. They checked cron jobs and so on without any hints so they decided to stay and wait: About time the cleaner came, plugged the power cable of her vacuum into the plug previously used by the server...

share|improve this answer
add comment

If server has external BMC, check BMC logs. Maybe BMC timer is configured for 24 hours and it is not being resetted by OS (still many BMCs try graceful shutdown first)

share|improve this answer
add comment

First, try turning off crond the day of the crash. (I suspect that a cron job at 11am that takes four minutes to run triggers a kernel bug or hardware fault.)

Also, try retarding the system clock by three minutes, to test whether the problem is caused by something within the server or external to the server.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.