2

I've just had a server outage at 4.59AM on Sunday morning, and looking through our uptime records going back to 2006 all except for 4 outages out of 20 occured between 11PM and 6AM. (Only looking at unplanned downtime on web and database servers, not app-servers on an internal LAN.)

Do others also find similar behaviour for their servers? Is this just a fluke?

Edit: It was because so many outages (these are unplanned, not scheduled maintenance, and occured on our hardware, not the ISP's network) have occured between 11pm and 6AM that made me wonder is that just us...

The servers are busiest in terms of visitors between 1PM and about 10PM, while database backups happen throughout the day and a big backup (where compression uses more of the CPU) occurs around 4.30 every morning. But the outages have occured at any time during this window (also these 20 outages are events occuring on 1 of 5 servers or 2 firewalls - about a third of which were result of 2 different machines' hard-drives failing). There is nothing indicating that the server was doing anything specifically because it was the small hours of the morning.

5 Answers 5

16

Typical "working hours" are no more than 40 hours of a week. Less in some parts of the world. A week contains a total of 168 hours. 40/168 = less than 24% of the time of a week is 'working hours'.

That suggests that failures of systems that are running 24/7 will occur 3-times more often during non-working hours than working hours.

Obviously, there are many other considerations that could go into this; multiple shifts, peak times (which for many, might further bias failures toward non-working hours), etc.

4
  • 2
    Nice. Simplke low level school math to proove WHY most outages happen outside working hours
    – TomTom
    Nov 14, 2010 at 17:56
  • 1
    Nice answer, and I voted you up, but the question specifically mentions that 80% (16/20) of unplanned downtimes occurred within a specific 7 hour window (11PM to 6AM). 80% is much more than the expected 29% (7/24), which is probably why the question was asked in the first place. Nov 14, 2010 at 18:18
  • I think its easy to lose site of simple points like this while looking for the esoteric explanations.
    – Rob Moir
    Nov 14, 2010 at 18:18
  • 1
    40% of sick day times off occur on Mondays or Fridays.
    – mpez0
    Nov 14, 2010 at 20:28
10

Yes, we find it, and no, it's no fluke. Your servers hate you, I'm sure. I know my servers hate me, and whilst they'd happily see me dead, if they feel themselves flagging I'm sure they hold on until their ntp daemons whisper in their ears that it's the middle of the night, and now is a good time to die. They know that to fail at 1030h will ruin my day, but to fail at 0345 will ruin my night, drag me down to London in the dark, and ruin the next day as well. They love that.

After having had a corporate firewall fail on me at a most inconvenient time due to a failed HDD, I separated the disc controller board from the HDD, cut it into four, and thereafter wore - and still wear - a quarter of its board, like a scalp, hanging from my "chain of office" (the lanyard with all the various access tokens I use at all my various sites). I am sure the sight of this grisly relic, in their plain view, kept its brother and sister servers largely in line henceforth, the penalty for failure being thus clearly displayed.

(In case anyone suffers a sense of humour failure, this post is a joke; except the bit about the HDD controller, which is absolutely true, and works.)

5
  • I find that my sense of humor is "on" more often during non-working hours than working hours. I wonder if that's because I enjoy not being at work? :p Nov 14, 2010 at 8:40
  • But I love my servers.*
    – Dan
    Nov 14, 2010 at 21:56
  • * Actually, I definitely don't - although I do admit to having a favourite.
    – Dan
    Nov 14, 2010 at 21:57
  • I bet if I camped out at the data-centre we would never have an outage, at least until I realised a backup wasn't backing up that important file.
    – Dan
    Nov 14, 2010 at 21:58
  • +1 to ward off the SOH failures. Anyway, you say it's just a joke, but you're wrong of course. Anyone that knows anything understands that you have to totally keep the technology in terrified submission 100% of the time - the slightest let-up results in disaster when it comes back to bite your a*** (a** for our north American cousins).
    – ChrisA
    Nov 14, 2010 at 22:43
2

The time between 11PM and 6AM seems like a typical time to have nightly cron jobs running. Perhaps some of them put a bit of extra strain on your servers, increasing the risk of a pending failure to happen just then.

2

Overnight is when most infrastructure changes occur. Networks and other resources may go down. It you are using remote monitoring, you will see your site go down because it is not reachable. Knowing the maintenance windows for your various resources will help eliminate these outages from actual outages.

As other have noted, on average outages are more likely outside of office hours based on hours on the clock. Given weekday availability and an 8 hour workday, only 1/3 of the outages should occur during office hours. Add in weekends and even less of the outages occur during workdays.

Track the reasons for the outages, and how they were detected. You will find some outages due to resources like the network being down. These may appear as mysterious outages where the site disappeared for a few minutes and came back without intervention. I would expect many of your overnight outages were infrastructure changes.

Infrastructure changes are usually scheduled, so you should be able to arrange to be notified of them. You can then adjust your response accordingly. Your outage log should reflect that the outage was due to the change. Also record any intervention that was required. You may need to add recovery code to your application to handle database restarts or other such resource changes.

Knowing the maintenance windows for various resources can help identify which resources are causing unplanned outages. You may need to trace your resource dependencies, as networked disk and databases will depend on the network infrastructure. Likewise, the database may depend on networked disk storage.

1

I had a Voip server die on me in the last 3 months. Die, perhaps isn't the best word, since the machine would be bootable after a kernel panic. Typically, the machine would function flawlessly between 7am and 7pm. Then, at random intervals separated by 1-30 days, it would be locked-up and unresponsive at the system console when I returned to the office at 7am.

After about 12 iterations of this situation... which invariably happened between 11pm and 7am, it was determined that the motherboard had failed, and specifically, the capacitors were to blame. I think I read somewhere that temperature extremes can hasten this death. I suppose my small office is not unusual, but I typically allowed the temperatures to swing as much as 15 degrees F above and 20 degrees below 75 degrees during the off hours. So, I believe, that small-time operations, that aren't using a chilled data center, are likely to suffer temperature induced failures during the wee hours of the morning.

My recollection, again, is that the logs showed failure during the 8 hours before we opened our shop in the morning -- always.

2
  • Incidentally, a tag of 'reliability' would make this topic/question more findable. Nov 14, 2010 at 21:45
  • Thanks Robert. I hadn't thought about temperature, but our hardware is installed in a data-centre and that shouldn't be a problem (hopefully).
    – Dan
    Nov 14, 2010 at 22:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .