1

I am running a site with 2 GB ram on centos. Using LAMP

Everyday for a few hours (about 3 to 4 hours) apache goes down. It takes too long to load a page and most of time it shows Request time out. But other ports are okay like ftp also 7778, as I am using kloxo.

I am not a server expert but I have googled and done a bit research and I think this is a ddos syn attack.

I used netstat -n | grep :80 | grep SYN |wc -l to look into syn connections

During the attack it shows a number more than 3000. But when my site backs to normal it shows a number around 100.

So please help me. and tell me how to stop this this is really pissing me off.

10
  • Try one of the solutions from this question: mod_evasive or fail2ban.
    – Russ Amos
    Aug 9, 2012 at 20:02
  • <3 mod_evasive! Aug 9, 2012 at 20:05
  • 2
    What state are the 3000 connections in? Your grep doesn't distinguish between SYN_SENT and SYN_RECV. The first means you are making a lot of connections on port 80 to other people, the second means other people are making a lot of connections on port 80 to you. (Also, technically either of these could be from port 80 but that would be a bit weird.) If you take off the | wc -l from the end you will get this information and also the list of IP addresses.
    – Ladadadada
    Aug 9, 2012 at 20:54
  • That's not a SYN DDoS, it's just your site being overly popular. A real SYN flood would knock out all TCP ports on the machine. Tune your Apache config and system resources to be able to handle the traffic you're receiving.
    – womble
    Aug 9, 2012 at 23:38
  • Is it helpful to suggest that the most effective way to prevent is any DDoS attack is not to piss off Anonymous or other script kiddies? (And/or not be on the same network as anyone who does...) Aug 10, 2012 at 0:31

4 Answers 4

8

The first thing to do is to determine what's happening. All you know at the moment is that Apache is unresponsive sometimes and during these times you have a lot of half-open TCP connections. You don't know if the TCP connections are the cause of the problem or just a symptom.

Do you have any performance monitoring system such as Cacti, Munin, Zabbix, Observium or any of the other options in this space? If not, get one of them now. Configure them to graph all the normal things such as memory usage, load average, CPU usage, free disk space, IOPS, network usage, etc but it also might be worth adding custom metrics such as requests per minute or TCP states. Also add whatever templates match the services you are using, such as Apache, MySQL, memcached, etc.

By analysing the graphs that these tools produce you should be able to find the resource that is hitting 100% during these downtimes. From there, you can trace the cause-and-effect chain back through to the initial trigger that caused the problem. It's possible that the resource that is being exhausted is not being measured by you and may not even be in your control.

As a guess, if your connections are in the SYN_SENT state, you are probably using someone else's API over HTTP and either they are down or they temporarily blocked you. If they are in the SYN_RECV state, there might be a firewall problem that is blocking the SYN-ACK responses from being received by the client or the response from the client being received by you. It could also be the SYN-flood you suggested.

During a real SYN-flood, you will see the bandwidth and the number of packets per second jump significantly. Use tcpdump -w packet-capture-file.cap or -j LOG in iptables to log these packets and see if you can spot a pattern. Maybe the source address is always the same, maybe it's always in a small range, maybe a strange TCP flag (such as URG or PSH) is set. Failing that, see if the IP addresses involved make multiple connection attempts. If they do, you can add them to a DROP rules in your firewall so that your OS won't have to deal with them. Depending on how many different IP addresses there are, you may need to split your rules into several tables to reduce the size of the list iptables has to scan through.

The DDoS may be big enough that it's actually exhausting your hosting provider's bandwidth. If that is the case, they or you will probably need to go to a dedicated DDoS mitigation company. These guys use either DNS or BGP to route your traffic to them, filter out the DDoS traffic and send the rest on to you. They're generally not cheap and you will have to weigh the cost of downtime against the cost of the mitigation service. There are also services like CloudFlare that can prevent DDoS attacks up to certain limits and have more affordable pricing plans.

3
  • +1 for explaining that the OP has no idea exactly what the problem is without ay monitoring analysis.
    – Jim B
    Aug 9, 2012 at 21:52
  • Thank you for your reply. It never uses a high resource, either it is under attack or not. cpu and ram uses always under 40%. For a first few days I was really confused I found nothing odd in the server, but apache was responding very slow for those few hours. And after that I have noticed during that time SYN_RECV connections goes very high. Do u think this is a ddos attack ? more suggestions needed. Aug 10, 2012 at 6:57
  • There's rarely any doubt when you get a real DDoS. The bandwidth spike is massive and you usually can't access your server at all unless you have an out-of-band access method. Keep investigating and check those monitoring graphs.
    – Ladadadada
    Aug 10, 2012 at 20:18
1

Are they distributed or coming from a single IP? Either way, I would recommend a firewall such as csf for restricting TCP syn attacks.

1
  • I am not able to trace those ips. Please tell me a better way to trace them ? Aug 9, 2012 at 20:19
1

A quick and cheap solution is to:

  • Reduce the timeout on the syn packets timeout

    Edit /etc/sysctl.conf

net.netfilter.nf_conntrack_tcp_timeout_syn_recv=30

  • Limit the amount of syn packets you can receive

create new chains

iptables -N syn-flood

limits incoming packets

iptables -A syn-flood -m limit --limit 10/second --limit-burst 50 -j RETURN

log attacks

iptables -A syn-flood -j LOG --log-prefix "SYN flood: "

silently drop the rest

iptables -A syn-flood -j DROP

Source

1
  • 1
    Depending on how much traffic your site gets or if your site is in constant communication with various servers (RSS feeds, etc), this rule could actually stop legitimate traffic. Also, just because you limit how many SYN pps you receive will not eliminate a DoS situation. Even if you --limit 10/second your system is still waiting for the ACK from the requester. So really, you are just delaying the DoS not mitigating or eliminating it. For a holistic approach, Id look into blacklisting based on the logged IP's. I still like the rules though Ponsfonze!
    – Blake
    Aug 9, 2012 at 22:52
0

Who cares about tracing the IP's. 90% of DoS/DDoS attacks will be proxied! You need to setup a blacklist period based on the "malicious traffic".

http://perishablepress.com/eight-ways-to-blacklist-with-apaches-mod_rewrite/

In a SYN attack your system is flooded with a series of SYN packets (obviously). Each packet causes your system to issue a SYN-ACK response. Your system waits for an ACK that follows the SYN+ACK (3 way handshake). Since the attack never sends back an ACK, your system resources get consumed. Once the queue is full your system will ignor incoming requests from legitimate users for services (http/mail etc). Hence it is necessary to stop this attack with iptables, ipfw, etc.

As far as a iptables rule:

iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP

1
  • 1
    Turning on syn-cookies is the appropriate response to the OS not having enough memory to keep track of the half-open connections. Blocking with mod_rewrite is ineffective for SYN-floods as Apache doesn't see a request until after the TCP connection has been completely opened.
    – Ladadadada
    Aug 10, 2012 at 20:13

Not the answer you're looking for? Browse other questions tagged .