Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Many large scale sites block access if too many requests are made from one origin. But large provider proxies would exhibit similiar access. So how should a server differ between those requests? And what behaveour is needed for legitimate proxy need to not trigger blocking?

share|improve this question
    
I just remember an example of a fail: A proxy using institution inside that Google Maps' sometimes lacks all map images due to overuse. Thus either Google Maps or their proxy setup doesn't work for that case. –  dronus Mar 18 '12 at 23:14
    
Well, how do you normally identify abuse? Too many requests per second? Too much load generated by one ip? And is a large proxy different to abuse? These are the questions you need to ask first. –  Mark Henderson Mar 18 '12 at 23:14
1  
Everyone does something different, certainly. You're asking for a lot of speculation here. –  Bill Weiss Mar 19 '12 at 1:37

3 Answers 3

Most sites check for open proxies, maybe crawl some proxy lists and check various blacklists.

Then as time goes and abuses go the admins just blacklist and/or throttle IP blocks. For example if you get a lot of shady traffic from China and your site is not even targeted at them then it's a fairly common practice to block traffic from their IP ranges (IANA and here's a list that resulted from someone digging through APNIC records).

After this it boils down to what site do you run and how important those users might be for your business. Google just uses CAPTCHAs to limit resource abuse, and I'm sure they'd utilize more drastic measures (even to the point of altering AS PATHs via BGP) if must.

So if you can authenticate the users behind the IP and their behavior is consistent with normal usage, then it's a probably just a transparent enterprise/corporate proxy for some office. (For monitoring usage patterns you might try using some kind of IPS/IDS with custom HTTP filters tailored to your needs.)

share|improve this answer

They would probably add the IP Addresses of the proxies to a whitelist of some sort so they are ignored by the traffic filtering software.

share|improve this answer

A more general question is "How do I deal with with receiving both malicious and benign traffic from the same IP address(es)?"

Besides open (and paid) proxies, you can have a legitimate user who unknowingly also hosts a spam bot. Or a company with a large NAT hiding spam bots. Or an entire country that sends you both spam and real users.

These can also come in different ratios. Some examples from the site I run: So far, 100% of all traffic I have received from WebSense IP address has been spammers and apart from the one customer I have from Senegal, the rest have been spammers. On the other hand China sends me about 25% real traffic and 75% spam bots.

Blocking all requests from these IPs or netblocks will always have a false positive rate, however you don't have to block all requests when you block an IP address. If you are having a problem with blog comment or forum spam, you could try limiting POST requests from the offending IP address.

<Limit POST>
  Deny from 192.0.2.1
  Deny from 198.51.100.2
  Deny from 203.0.113.3
</Limit>

Rather than blocking the IP address(es) you could use the IP address as part of your spam scoring system. If you have a Bayesian scoring system for spam, just add the user's IP address into the tokens. Addresses that always spam will get a high score, addresses that don't will get a low score and addresses that have both types of users (such as proxies) will hover around the middle so that they are judged on the content of their messages and not their IP address.

You could even ignore IP addresses entirely and just judge all submitted content on the content itself and not the reputation of the IP address it came from.

Although I haven't done it myself, you could probably configure fail2ban to read your spam scoring logs and drop the IP addresses of the worst offenders into a .htaccess file so that the IP address bans will expire after a certain time.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.