Programatically blocking a url to rate limit, unblocking or expiring the block

Question

My requirements are to be able to rate limit or temporarily hard block any requests to our server for a particular URL if the client sends more than x requests per hour.

There are about 250K urls that could potentially be blocked.

Clients have unique urls that they make a http post too, so something like:

customer1.example.com/post/location/36A107F4-DD68-47F2-A093-753F7A427CF6/

Where customer1 is the customer name, and 36A107F4-DD68-47F2-A093-753F7A427CF6 is their unique location guid.

I want my application to be able to make some sort of an API call to block a given url, and then either use an API call to unblock or have the block expire after x minutes.

Is this possible with any open source service like haproxy or nginx?

Answer 1

Here are a few ideas...these aren't specific to nginx or HAProxy, but possibly you could use the same ideas.

This article discusses how they are using Varnish to implement per-URL rate limiting for an application. The application communicates with Varnish using a custom HTTP header, rather than an API call, but the idea is similar.

Using Apache, you could do something similar using a programmatic RewriteMap...look up requests in a database, and for those that are currently rate-limited, rewrite them to an error page, otherwise pass them through unmodified. Turning rate-limiting on/off for a particular URL would be simple one-row changes in the database.