5

My company builds various web sites for its customers with a dedicated hosting provider.

This hosting provider accidentally shutdown the devices responsible for Border Gateway Protocol (BGP) announcements for a small range of IPs. Since I'm lucky like that, one of these IPs happened to be the public IP address assigned to the load balancer for all of the web traffic for our customers. As a result, the BGP routing advertisement for this range was withdrawn and quickly became unreachable worldwide.

The hosting provider remedied the problem once alerted to it, but that cost us over 15 minutes of downtime, which we're anxious to avoid in the future.

  1. How could we monitor this? It's a lot lower level than our normal monitoring, which just checks apache httpd status, JVMs, etc. We have internal monitoring which uses Advent AppEngine to check server processes, apache server status responses, application home page responses.

  2. Are we able to take steps to fix this ourselves; e.g. by making our own BGP announcements somehow?

I'm happy for pointers / suggested reading rather than just straight answers, since this level of the stack is completely new to me and I'd like to fill in the gaps of my knowledge.

Improve this question

6 Answers 6

Reset to default
4

You're unlikely to be able to work around this, unless your address space is large enough for you to be able to run your own BGP. Even then, you're vulnerable to BGP failures by your peers.

If you're using multiple DNS servers in separate ASes, you may be able to have some kind of work around by setting a low TTL and failing over to a separate web server in a different netblock/data centre by changing DNS once problems are noted. Even this will take several minutes however, at the least.

EDIT: as pointed out by Chris, if you're running BGP, you need all of your peers to fail before you become unreachable.

Improve this answer
2
  • That's why you have 2 providers and you peer with both.
    – chris
    Jul 29, 2009 at 3:26
  • If you have different datacenters you could also load-balance at DNS level. In case of routing holes affecting one of your DC the users will still have partial connectivity even during the failover.
    – Federico
    Jul 30, 2009 at 10:28
3

You're unlikely to be able to run BGP unless you have at least a /23 of Provider Independant address space and have an ASN number. As such, you need to trust your hosting company. Router changes tend to be fairly rare, so the likelihood of this problem happening again is slim. You could investigate any SLA you have with them, but this is probably just going to involve getting a refund on your hosting fees.

As far as monitoring is concerned, we have a dedicated server outside our network, which we use as an external Nagios server. You could buy a cheap VPS server and use that to monitor things from the PoV of an external user. For example, we check SMTP and HTTP work, rather than checking that exim and apache are running, which we do on our internal monitoring.

Improve this answer
4
  • You can probably advertise a /24 as easily as a /23 -- your providers may summarize that, but likely you'll be getting IP addresses from them anyhow, so it isn't a crisis...
    – chris
    Jul 29, 2009 at 3:18
  • If you're using PA space, it's very unlikely that you'd need to or want to run BGP. Quite a lot of places will also filter out networks smaller than /23 routes to reduce their routing tables.
    – David Pashley
    Jul 29, 2009 at 7:27
  • People with smaller routers may filter to /23 but providers won't and they're the ones you care about. If you're dual homed and connected to provA and provB and have address space in provB's assignments, you'll still be covered because provA and provB peer upstream of you and even if almost everyone else summarizes routes down to /20 (for the sake of argument) you can be damned sure that provA and provB don't summarize away a customer's route. A remote session will hop onto provB's network right up to the break then, because they peer with provA, use that for the rest of the trip.
    – chris
    Jul 29, 2009 at 12:02
  • Also, monitoring from an outside network will have other advantages over BGP monitoring: it will detect non-BGP failures.
    – bortzmeyer
    Jul 30, 2009 at 8:00
2

For the record, it exists several gratis BGP monitor and alarm systems. None provide a resolution of 15 mn as you want. And, since you can have many other causes of outage, monitoring the IP connectivity from the outside is the only real solution.

A general article about BGP monitoring, in french.

Improve this answer
0

Depending on how things are setup, the size of the netblock being advertised, and how things get agregated upstream, you may be able to use one of the looking glass scripts to monitor BGP announcements for the block your server is in.

It may be easier to just ping both your host and the router one step back from your server from the outside. You can use traceroute to determine which address to use.

There is very little you can do to prevent your hosting company from doing this again. In order to do so you would need to have a router or other host running BGP connected to your provider at a minimum. Unless you also have another provider, it wont help if they accidentally turn off the peering router.

A better solution may be to have a failover site as mentioned by another answer. Depending on your risk tolerance you can setup failover to happen in a very short time but it involves complete control of your DNS.

Improve this answer
1
  • He was talking about 15 mn of downtime. Reading the looking glasses will take more time than that. Pinging from the outside is indeed the best solution.
    – bortzmeyer
    Jul 30, 2009 at 7:59
0

Your options are pretty limited. You can yell and scream at your provider, you can move to another provider, you can get 2 different IP ranges and advertise services on both and have short TTLs on your DNS entries.

But

If you really want to solve this, move to a colo facility with a meetme room and buy bandwidth and IP addresses from a couple providers. Then, register an ASN with arin (or whatever registrar is correct for wherever you live) and peer with the providers yourself.

If you're buying enough bandwidth, it won't be hard to get them to cough up a /24 or /23. The peering is also going to be pretty easy, depending on the size of the colo facility and the amount of bandwidth you're going to be asking for.

If you're writing big checks and you act like you know exactly what you want (and what you want is reasonable) it isn't hard to get this stuff done. If you farm it out to your "provider" you'll always be on the dumb end of the stick.

Improve this answer
8
  • 1
    Be wary of providers that will "cough up" a /24 or /23 of address space based on how much you are spending rather than demonstrable need. Chances are they are breaking the rules that they agreed to when becoming an LIR. Provision of Provider Independant address space is out of your service provider's hands. AFAIK no RIRs will make a PI assignment based on invoice size...
    – Russell Heilling
    Jul 29, 2009 at 9:37
  • Providers have lots of addresses. They own them, you use them, you change providers you give the addresses back. No need to talk to a registrar except for the AS number. The need you provide the provider is that you want it to be reliable. The only real hazard is that it is slightly harder to change away from the provider who is loaning you their address space, but that's always the case unless you get your addresses directly from a registry, which as you say, is slightly harder (you do need to demonstrate need, as described in one of the docs I linked to).
    – chris
    Jul 29, 2009 at 11:29
  • 1
    Like Russell says - I'd be very wary if an LIR is willing to bypass the application/documentation process just to get a sale. It could place the assignments and the LIR in jeopardy. You need to demonstrate need to the LIR and the LIR must make that available to the RIR. Renumbering is always possible but it certainly isn't easy.
    – Dan Carley
    Jul 29, 2009 at 11:54
  • ISPs give out netblocks all the time. How is this any different, except that you're associating an AS with that netblock and advertising it out to to providers? There isn't any funny business going on -- you have a legitimate need to dual home your enterprise and you don't need a giant allocation of addresses so you just get the addresses from the provider, which I assume is kinda what their business is. Chances are you can even just get a /29 from them and your two providers are happy to route those. They may not appear in the global route tables, but that's not a problem.
    – chris
    Jul 29, 2009 at 12:20
  • If the /29 is part of ProviderA's allocation and is not showing up through ProviderB from the core and ProviderA then suddenly no longer announce it, how does the (non-announced) network via ProviderB actually help?
    – Vatine
    Jul 29, 2009 at 12:24
0
  1. You can monitor your provider announcements by asking public route-servers (http://www.traceroute.org/#Route%20Servers) about the prefix you're using. You can automate this kind of monitoring by telneting these route servers.
  2. If you use enough bandwidth, have the budget and the skills for such deployment, you can ask for an AS number and an IP address range. However, this is costly, and since RIRs are going out of IPv4 addresses, you'll have to give real proof of your needs.
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .