Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Is there a way I can make Nginx to notify me if hits from a referrer goes beyond a threshold?

e.g If my website is featured at Slashdot and all of sudden I have 2K hits coming in an hour I want to be notified when goes beyond 1K hits an hour.

Will it be possible to do this in Nginx? Possibly without lua? (since my prod is not lua compiled)

share|improve this question
4  
What's "Slashdot"?? –  ewwhite Sep 27 '12 at 16:57
    
I did something like this to detect ddos on ngix. I achieved it by parsing access log. I did a cron job to parse access log and count unique ip connections per hour. –  Hex Sep 27 '12 at 16:57
8  
You mean you want nginx to be able to detect if you've been bought by Dice? –  MDMarra Sep 27 '12 at 16:58
1  
@Hex That (and maybe a few snippets from your script) would make an excellent answer to this question :) –  voretaq7 Sep 27 '12 at 16:58
3  
Probably no need to worry about getting Slashdotted anymore. Your webserver should be able to handle an extra 4 connections an hour. Might want to worry about getting Redditted, though... –  HopelessN00b Sep 27 '12 at 16:59
show 1 more comment

2 Answers

I think this would be far better done with logtail and grep. Even if it's possible to do with lua inline, you don't want that overhead for every request and you especially don't want it when you have been Slashdotted.

Here's a 5-second version. Stick it in a script and put some more readable text around it and you're golden.

5 * * * * logtail -f /var/log/nginx/access_log -o /tmp/nginx-logtail.offset | grep -c "http://[^ ]slashdot.org"

Of course, that completely ignores reddit.com and facebook.com and all of the million other sites that could send you lots of traffic. Not to mention 100 different sites sending you 20 visitors each. You should probably just have a plain old traffic threshold that causes an email to be sent to you, regardless of referrer.

share|improve this answer
    
The problem is to be proactive. I need to know from any site. Another question is where do I put the threshold? Did you mean some additional log parsing? Also I didn’t find –o in fourmilab.ch/webtools/logtail –  Quintin Par Sep 28 '12 at 4:24
    
The threshold will depend on how much traffic your server(s) can handle. Only you can set that. If you want quicker notification, run it every five minutes instead of every hour and divide the threshold by 12. The -o option is for an offset file so it knows where to start reading next time. –  Ladadadada Sep 28 '12 at 7:20
add comment

The nginx limit_req_zone directive can base its zones on any variable, including $http_referrer.

http {
    limit_req_zone  $http_referrer  zone=one:10m   rate=1r/s;

    ...

    server {

        ...

        location /search/ {
            limit_req   zone=one  burst=5;
        }

You will also want to do something to limit the amount of state required on the web server though, as the referrer headers can be quite long and varied and you may see an infinte variet. You can use the nginx split_clients feature to set a variable for all requests that is based on the hash of the referrer header. The example below uses only 10 buckes, but you could do it with 1000 just as easily. So if you got slashdotted, people whose referrer happened to hash into the same bucket as the slashdot URL would get blocked too, but you could limit that to 0.1% of visitors by using 1000 buckets in split_clients.

It would look something like this (totally untested, but directionally correct):

http {

split_clients $http_referrer $refhash {
               10%               x01;
               10%               x02;
               10%               x03;
               10%               x04;
               10%               x05;
               10%               x06;
               10%               x07;
               10%               x08;
               10%               x09;
               *                 x10;
               }

limit_req_zone  $refhash  zone=one:10m   rate=1r/s;

...

server {

    ...

    location /search/ {
        limit_req   zone=one  burst=5;
    }
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.