Overall it has been working fairly well, although I'd like to hear some suggestions from gurus who have been working with these sort of circumstances for longer than I have. I'm sure those working in the web hosting business have had their fair share of dealing with these situations. So I'm wondering what the best practices are to approaching these sorts of problems in a corporate environment?
Preventing a DDoS is mostly about not being a target. Don't host game servers, gambling/porn sites, and other things that tend to get people annoyed.
Mitigating a DDoS attack comes in two forms:
- being able to ignore traffic and shed excess load, which is useful when you're under an attack that tries to take you down by overloading your machines (and also comes in handy if you ever get "Slashdotted";
- being able to reject abusive network traffic upstream of you, so that it doesn't clog your links and take out your connectivity.
The former is somewhat dependent on what exactly you're serving, but usually comes down to some combination of caching, overflow handling (detecting when the servers are "full" and redirecting new connections to a low-resource-usage "sorry" page), and graceful degradation of request processing (so not doing dynamic rendering of images, for example).
The latter requires good communications with your upstreams -- have the phone number of your upstreams' NOCs tattooed to the inside of your eyelids (or at the very least in a wiki somewhere that isn't hosted in the same place as your production servers...) and get to know the people who work there, so when you call you'll get immediate attention as someone who actually knows what they're talking about rather than just being some random johnny.
You don't mention what kind of perimeter security you have in place. With Cisco firewalls you can limit the number of embryonic (half sessions) that your firewall will allow before it cuts them off, while still allowing full sessions to go through. By default it's unlimited, which offers no protection.
Hardware-assisted load-balancers such as Foundry ServerIron's and Cisco ACEs are great for dealing with huge numbers of the main types of DOS/DDOS attacks but aren't so flexible as software solutions which can 'learn' newer techniques quicker.
One good source for information is at this site. One measure which they only mention in passing (and which is worth researching further) is enabling SYN cookies. This prevents an entire class of DoS attacks by preventing an attacker from opening a large number of 'half-open' connections in an attempt to reach the maximum number of file descriptors permitted per process. (See the bash manpage, look for the 'ulimit' builtin with the '-n' option)
Disclaimer: I'm not a DDoS protecion guru.
I think it depends on the budget you have for it, what your uptime terms of conditions are and how you or your customers are exposed to this kind of risk.
Proxy-based DDoS protection could be an option. In most cases it is not a cheap option, but I think it's the most effective. I would ask my hosting provider for a solution. RackSpace, for example, provides this multi-tier mitigation tool. I'm sure all large hosters have similar solutions.