Yesterday our webservers running Apache2.2 got DDoSed from a botnet which caused nine (9) hours downtime. The attacker was just creating too many "GET /" connections with result our servers to reach cpu load 100 and finally rejecting further connections. In front of those webserver there is a pfsense machine with haproxy that load-balances connections to webserver.
We tried to block the ips from pfsense but the connections were still getting through. It was not after the DDoS has stopped when we realized that the connections were still getting through because they were already established. When we created the rules we did not specify that it should also apply on established connections. We still dont know if that would work since the attach was finished.
We implemented modsecure on apache but we realized that it was not working as expected due to X-Forwarded-For. Modsecure was correctly identifying the ips but when was coming to blocking face, it was rejecting the ip of load balancer (need to install mod_remoteip). By the time we used mod_evase on webservers the attack was finished.
I did some homework on DDoS countermeasures but i still have some questions that i haven't managed to answer myself.
1) Can you block "GET /" requests from pfsense/frontend in Layer4 level so the connections wont reach the backend?
I saw that there is a possibility to rate limit the connections of an ip but will this help when the attacker creates a single connection (keep-alive) and then sends all "GET" requests through this connection?
2) Is it better to block such attacks on the frontend (say firewall - loadbalancer) or the backend?