Rate limiting using haproxy per url per ip

Question

Our servers are being hit by someone constantly. Its not DDOS. Just one ip hits one url more than 200 times a second. Currently I am blocking the user by ip tables. We are using HAproxy for load balancing. Is there a way to limit a user based on his ip and the url being hit?

I don't want to block just based on IP as NAT'ed users might be affected. Lets say I want to show 503 error page if a particular ip is hitting same url like www.example.com/somepage.php?some=option&other=option2 more than 3000 times in 5 minutes? This means the same ip can access other urls like www.example.com/somepage.php?another=someotheroption.

Answer 1

This solution requires at least haproxy 1.6.

First, add the following to the frontend:

http-request set-header X-DOS-Protect %[src];%[req.fhdr(host)]%[capture.req.uri]

Then, add the following to the backend:

stick-table type integer size 1m expire 5m store http_req_rate(5m)
tcp-request inspect-delay 5s
tcp-request content track-sc0 req.fhdr(X-DOS-Protect),crc32(1) if HTTP
http-request tarpit if { sc0_http_req_rate gt 3000 }

I was not able to find a way to do the tracking in the frontend as I didn't find a way to apply a converter on that concatenated string that makes up the X-DOS-Protect header.

I am applying the hash function in order to make sure that you don't store a huge string in the stick-table as it could easily lead to a denial of service. If you think that this hash function is not suitable for you due to too many possible collisions, you could also make it bigger by applying crc32 to each of the concatenated components (and, of course, removing it when storing the data and switching to a bigger stick-table storage), like this:

http-request set-header X-DOS-Protect %[src,crc32(1)];%[req.fhdr(host),crc32(1)]%[capture.req.uri,crc32(1)]

stick-table type string len 30 size 1m expire 5m store http_req_rate(5m)
tcp-request inspect-delay 5s
tcp-request content track-sc0 req.fhdr(X-DOS-Protect) if HTTP
http-request tarpit if { sc0_http_req_rate gt 3000 }

Please note that this last solution will use more than 7 times more memory than the first one, for each entry in the stick table. Of course, the collision risk would be way smaller too.