2

Edit: While this appears to be due to a power outage at Linode's Fremont facility, I've still had a problem with this in the past. Any help would be appreciated.


I want you to teach me to fish.

Yes, I'm asking "Why is my VPS crashing?" I'm sure if I gave you root access to my server you could find out in 2 minutes. I'm sure you could tell me to post my error logs, and I'd hunt them down without much thought. But that's not what I want.

I want you to teach me to fish.

I'm running into a problem that has no root cause apparent to me. See my Linode graphs to see what I mean. The crash is happening at a time when network traffic and disk IO is at a minimum.

alt text

alt text

What are the first, most common, most basic places I should look when this happens, and what should I be looking for?

How can I be alerted when this happens (so I don't miss it for 8 hours while the server's down)?

2
  • where on the graphs do we see a crash / outage? What do your system logs show? Hard to teach someone how to fish if you don't know what tools they already have at hand. Are we to assume you already checked with Linode support (who have been pretty fantastic when I had issues) ?
    – Jakub
    Nov 21, 2010 at 5:13
  • 1
    @Jakub Well this particular time it was an issue with their Fremont facility having an outage. You can see on the right-hand side of the graph where the marker just stops (no activity = down). I'm not sure what system logs to look into.
    – Josh Smith
    Nov 21, 2010 at 5:37

1 Answer 1

4

If your entire server is crashing due to non-server related events, I would go with a better provider. The fact that a power outage caused the machines to go down indicates they don't have any backup providers--which is troubling, especially since it appears that they didn't alert you until well after the fact. I am surprised, as Linode has a reputation for being fairly reliable.

What are the first, most common, most basic places I should look when this happens, and what should I be looking for?

What kind of things are you talking about? /var/log/* or similar are great places to start general debugging when you don't know what happened. Unfortunately, there's no real catchall for unknown server crashes. You may want to investigate enabling more verbose logging across many of your services, although be forewarned--this can cause your log files to balloon!

How can I be alerted when this happens (so I don't miss it for 8 hours while the server's down)?

For this, I've actually found the easiest solution to simply be external services. For my company's forward facing web sites, we use utilities like Pingdom, as well as several internally hosted tools such as Nagios and Ganglia. Attacking this on all fronts is the best bet--having multiple sources to check your uptime, along with a variety of geographical locales, is the standard practice for monitoring.

Remember, as tempting as it is to maintain your own versions, having something that's (a) outside your network and control, and (b) has multiple servers around the country and/or world, will give you much better return. Additionally, most of these services are not expensive at all.

I hope this helps, or at least gets you on the right track!

2
  • Thanks for the help on this. I actually ended up signing up for Pingdom after someone on #linode suggested it. Nagios I've heard about but never fully understood.
    – Josh Smith
    Nov 21, 2010 at 18:09
  • Oh, and in Linode's defense, they're still looking into what happened, but it appears that a severe lightning storm coupled with UPS failures at their new HE facility caused the outage. They're not sure yet why the redundancies failed.
    – Josh Smith
    Nov 21, 2010 at 18:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .