Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

One of the servers I use is hosted on the Amazon EC2 cloud. Every few months we appear to have a DDOS attack on this sever. This slows the server down incredibly. After around 30 minutes, and sometimes a reboot later, everything is back to normal.

Amazon has security groups and firewall, but what else should I have in place on an EC2 server to mitigate or prevent an attack?

From similar questions I've learned:

  • Limit the rate of requests/minute (or seconds) from a particular IP address via something like IP tables (or maybe UFW?)
  • Have enough resources to survive such an attack - or -
  • Possibly build the web application so it is elastic / has an elastic load balancer and can quickly scale up to meet such a high demand)
  • If using mySql, set up mySql connections so that they run sequentially so that slow queries won't bog down the system

What else am I missing? I would love information about specific tools and configuration options (again, using Linux here), and/or anything that is specific to Amazon EC2.

ps: Notes about monitoring for DDOS would also be welcomed - perhaps with nagios? ;)

share|improve this question
    
Preventing it is nearly impossible but you could certainly decrease your vulnerability. –  Belmin Fernandez Dec 13 '11 at 1:35
    
Very true. And I would be happy with answers that discuss decreasing vulnerability :) –  cwd Dec 13 '11 at 1:46
    
In order to know how to prevent it, one would need to know more about the nature of the attack. Was it a syn-flood? Was there a web page that is expensive that was being hit? was it some other service? –  stew Dec 13 '11 at 1:58
    
I think it was a webpage but am not sure. I would also welcome tips on monitor and investigating these. Some of the access logs have already been deleted - what else should I be looking for? –  cwd Dec 13 '11 at 2:03
    
and actually, if a reboot fixes the problem, then I don't know what it could be, except perhaps your server has a resource leak somewhere. A reboot certainly cannot stop a DDoS on its own. How do you know this is a DDoS at all? –  stew Dec 13 '11 at 2:25

6 Answers 6

up vote 23 down vote accepted

A DDOS (or even a DOS), in its essence, is a resource exhaustion. You will never be able to eliminate bottlenecks, as you can only push them farther away.

On AWS, you are lucky because the network component is very strong - it would be very surprising to learn that the upstream link was saturated. However, the CPU, as well as disks I/O, are way easier to flood.

The best course of action would be by starting some monitoring (local such as SAR, remote with Nagios and/or ScoutApp) and some remote logging facilities (syslog-ng). With such setup, you will be able to identify which resources get saturated (network socket due to Syn flood ; CPU due to bad SQL queries or crawlers ; ram due to …). Don’t forget to have your log partition (if you don’t have remote logging enable) on an EBS volumes (to later study the logs).

If the attack come through a web pages, the access log (or the equivalent) can be very useful.

share|improve this answer
    
Thanks - all of this looks very good. +1 –  cwd Dec 13 '11 at 2:56

You can also further isolate your EC2 instances by putting them behind an Elastic Load Balancer and only accepting traffic from the ELB instance. This puts more of the onus on Amazon to manage DDOS attacks.

I assume that you'll still have SSH open to all, so it's likely you'll still see some rogue traffic coming in there, unless you can lock down that port to some static IPs. You could change the SSHd port to something more obscure (i.e., something other than 22) to further reduce DDOS hits (most bots only check known ports).

I'll also mention fail2ban, which can monitor logs and temporarily modify your ip tables to block specific IPs (for instance, if there have been 6 failed attempts to SSH into your host from a single IP address, it can block that IP for 30 minutes or so).

share|improve this answer
    
+1 thank you. actually i did have ssh closed down ;) –  cwd Dec 13 '11 at 21:35

@cwd:

If you are using Apache, I suggest mod_security. Packaged by most vendors, the core rules set do an fantastic job.

Another hardening step, is ;imiting the requests at the webserver level. Nginx., Apache can throttle and limit incoming requests.

share|improve this answer
    
awesome - those look like fantastic options thanks! –  cwd Mar 8 '12 at 1:15

Config server firewall is the best I've seen for DDoS mitigation in software based VM's in EC2. If you combine the syslog feature it can protect against a load balanced environment.

http://configserver.com/cp/csf.html

share|improve this answer
    
Welcome to Server Fault! Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. –  Scott Pack Oct 10 '13 at 17:14

The solution I use for blocking realtime bad activity IP's coming out of AWS and others is this... In my CSF Firewall in the config for LFD Blocklists I use a list found here - http://myip.ms/browse/blacklist/Blacklist_IP_Blacklist_IP_Addresses_Live_Database_Real-time

Download Blacklist for CSF Firewall » http://myip.ms/files/blacklist/csf/latest_blacklist.txt

No more outrageously obnoxious AWS traffic.

share|improve this answer
    
This does not answer the question. –  kasperd Aug 14 at 6:47

I am biased, as I work for a content delivery network as a security Presales engineer.

However, leveraging a Ddos mitigation solution on a content delivery network ensures that you never run out of resources at the origin. It's similar to putting an F5 load balancer in front of your site, but spread to thousands of locations around the world.

A good cdn will allow you to cloak the origin with a whitelist which you install on the aws firewall. So when attackers perform their reconnaissance on Amazon, your IP address will come up empty as everything will be blocked.

So the Ddos attacks are blocked when the traffic hits a node as close as possible to the attacker. This ensures you mitigate Ddos attacks as far away from the asset you are trying to protect.

A good cdn can also perform health checks and failover traffic to other locations eg another ego on in ass, Azure, rack space, soft layer, a physical dc etc. It should also have a WAF to ensure you can block application layer exhaustion attacks like RUDY, slowpost, slowloris as well as sqli, xss, rfi, lfi etc.

By default the cdn also blocks network layer attacks like teardrop, icmp attacks, synfloods etc. A cdn is able to mitigate Ddos attacks because trey have vast amounts of capacity to accept the requests, filter out bad traffic and pass on the good traffic. So amplifying attacks like ntp, DNS, ssdp, chargen and snmp volumetric attacks can be blocked.

The largest attack I have seen to date has been 321gbps in July 2014. During this attack there was also a DNS protocol attack at 20gbps. So you will need to ensure you DNS infrastructure is also resilient to withstand a huge number of requests.

From the description you provided, it looks like you were subject to an exhaustion attack, where thr attacker opened up lots of threads such that all the threads were consumed up on the web server, app server or firewall. It's similar to something like a slowpost, slowloris or RUDY.

To block against application layer exhaustion attacks you will need to get a web application firewall (WAF). A typical network firewall (including amazons firewall and next generation firewalls) won't be able to block it. Sent work firewalls these days can only block about 30% of all the attacks these days (Nov 2014).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.