Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This question already has an answer here:

I am running a site with 2 GB ram on centos. Using LAMP

Everyday for a few hours (about 3 to 4 hours) apache goes down. It takes too long to load a page and most of time it shows Request time out. But other ports are okay like ftp also 7778, as I am using kloxo.

I am not a server expert but I have googled and done a bit research and I think this is a ddos syn attack.

I used netstat -n | grep :80 | grep SYN |wc -l to look into syn connections

During the attack it shows a number more than 3000. But when my site backs to normal it shows a number around 100.

So please help me. and tell me how to stop this this is really pissing me off.

share|improve this question

marked as duplicate by Tom O'Connor Aug 25 '13 at 21:09

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

    
Try one of the solutions from this question: mod_evasive or fail2ban. –  Russ Amos Aug 9 '12 at 20:02
    
<3 mod_evasive! –  Dave Drager Aug 9 '12 at 20:05
2  
What state are the 3000 connections in? Your grep doesn't distinguish between SYN_SENT and SYN_RECV. The first means you are making a lot of connections on port 80 to other people, the second means other people are making a lot of connections on port 80 to you. (Also, technically either of these could be from port 80 but that would be a bit weird.) If you take off the | wc -l from the end you will get this information and also the list of IP addresses. –  Ladadadada Aug 9 '12 at 20:54
    
That's not a SYN DDoS, it's just your site being overly popular. A real SYN flood would knock out all TCP ports on the machine. Tune your Apache config and system resources to be able to handle the traffic you're receiving. –  womble Aug 9 '12 at 23:38
    
Is it helpful to suggest that the most effective way to prevent is any DDoS attack is not to piss off Anonymous or other script kiddies? (And/or not be on the same network as anyone who does...) –  HopelessN00b Aug 10 '12 at 0:31
show 4 more comments

4 Answers

The first thing to do is to determine what's happening. All you know at the moment is that Apache is unresponsive sometimes and during these times you have a lot of half-open TCP connections. You don't know if the TCP connections are the cause of the problem or just a symptom.

Do you have any performance monitoring system such as Cacti, Munin, Zabbix, Observium or any of the other options in this space? If not, get one of them now. Configure them to graph all the normal things such as memory usage, load average, CPU usage, free disk space, IOPS, network usage, etc but it also might be worth adding custom metrics such as requests per minute or TCP states. Also add whatever templates match the services you are using, such as Apache, MySQL, memcached, etc.

By analysing the graphs that these tools produce you should be able to find the resource that is hitting 100% during these downtimes. From there, you can trace the cause-and-effect chain back through to the initial trigger that caused the problem. It's possible that the resource that is being exhausted is not being measured by you and may not even be in your control.

As a guess, if your connections are in the SYN_SENT state, you are probably using someone else's API over HTTP and either they are down or they temporarily blocked you. If they are in the SYN_RECV state, there might be a firewall problem that is blocking the SYN-ACK responses from being received by the client or the response from the client being received by you. It could also be the SYN-flood you suggested.

During a real SYN-flood, you will see the bandwidth and the number of packets per second jump significantly. Use tcpdump -w packet-capture-file.cap or -j LOG in iptables to log these packets and see if you can spot a pattern. Maybe the source address is always the same, maybe it's always in a small range, maybe a strange TCP flag (such as URG or PSH) is set. Failing that, see if the IP addresses involved make multiple connection attempts. If they do, you can add them to a DROP rules in your firewall so that your OS won't have to deal with them. Depending on how many different IP addresses there are, you may need to split your rules into several tables to reduce the size of the list iptables has to scan through.

The DDoS may be big enough that it's actually exhausting your hosting provider's bandwidth. If that is the case, they or you will probably need to go to a dedicated DDoS mitigation company. These guys use either DNS or BGP to route your traffic to them, filter out the DDoS traffic and send the rest on to you. They're generally not cheap and you will have to weigh the cost of downtime against the cost of the mitigation service. There are also services like CloudFlare that can prevent DDoS attacks up to certain limits and have more affordable pricing plans.

share|improve this answer
    
+1 for explaining that the OP has no idea exactly what the problem is without ay monitoring analysis. –  Jim B Aug 9 '12 at 21:52
    
Thank you for your reply. It never uses a high resource, either it is under attack or not. cpu and ram uses always under 40%. For a first few days I was really confused I found nothing odd in the server, but apache was responding very slow for those few hours. And after that I have noticed during that time SYN_RECV connections goes very high. Do u think this is a ddos attack ? more suggestions needed. –  Suman Biswas Aug 10 '12 at 6:57
    
There's rarely any doubt when you get a real DDoS. The bandwidth spike is massive and you usually can't access your server at all unless you have an out-of-band access method. Keep investigating and check those monitoring graphs. –  Ladadadada Aug 10 '12 at 20:18
add comment

Are they distributed or coming from a single IP? Either way, I would recommend a firewall such as csf for restricting TCP syn attacks.

share|improve this answer
    
I am not able to trace those ips. Please tell me a better way to trace them ? –  Suman Biswas Aug 9 '12 at 20:19
add comment

A quick and cheap solution is to:

  • Reduce the timeout on the syn packets timeout

    Edit /etc/sysctl.conf

net.netfilter.nf_conntrack_tcp_timeout_syn_recv=30

  • Limit the amount of syn packets you can receive

create new chains

iptables -N syn-flood

limits incoming packets

iptables -A syn-flood -m limit --limit 10/second --limit-burst 50 -j RETURN

log attacks

iptables -A syn-flood -j LOG --log-prefix "SYN flood: "

silently drop the rest

iptables -A syn-flood -j DROP

Source

share|improve this answer
    
Depending on how much traffic your site gets or if your site is in constant communication with various servers (RSS feeds, etc), this rule could actually stop legitimate traffic. Also, just because you limit how many SYN pps you receive will not eliminate a DoS situation. Even if you --limit 10/second your system is still waiting for the ACK from the requester. So really, you are just delaying the DoS not mitigating or eliminating it. For a holistic approach, Id look into blacklisting based on the logged IP's. I still like the rules though Ponsfonze! –  Blake Aug 9 '12 at 22:52
add comment

Who cares about tracing the IP's. 90% of DoS/DDoS attacks will be proxied! You need to setup a blacklist period based on the "malicious traffic".

http://perishablepress.com/eight-ways-to-blacklist-with-apaches-mod_rewrite/

In a SYN attack your system is flooded with a series of SYN packets (obviously). Each packet causes your system to issue a SYN-ACK response. Your system waits for an ACK that follows the SYN+ACK (3 way handshake). Since the attack never sends back an ACK, your system resources get consumed. Once the queue is full your system will ignor incoming requests from legitimate users for services (http/mail etc). Hence it is necessary to stop this attack with iptables, ipfw, etc.

As far as a iptables rule:

iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP

share|improve this answer
    
Turning on syn-cookies is the appropriate response to the OS not having enough memory to keep track of the half-open connections. Blocking with mod_rewrite is ineffective for SYN-floods as Apache doesn't see a request until after the TCP connection has been completely opened. –  Ladadadada Aug 10 '12 at 20:13
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.