Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have the following settings in my pf.conf to dissuade abusive clients (I get a few a day that are obviously just scraping content indiscriminately and slow everything down):

pass in on $ext_if proto tcp to { $loadbalancer1_int } port $web_ports flags S/SA keep state (max-src-conn 80, max-src-conn-rate 200/5, overload <web_flood> flush)
pass in on $ext_if proto tcp to { $loadbalancer2_int } port $web_ports flags S/SA keep state (max-src-conn 80, max-src-conn-rate 200/5, overload <web_flood> flush)

Unfortunately some of my customers are getting themselves blocked. I don't know what these customers are doing to trigger such network IO. I keep upping the values and they keep getting themselves blocked. Aren't these values (80 connections & 200 connections within 5 seconds) well within what would generally be considered "abusive"?

share|improve this question

1 Answer 1

up vote 1 down vote accepted

"abusive" is entirely relative, but in terms of web access setting a rate limit on number of connections/second is almost certain to catch legitimate users.

Bear in mind what happens when you visit a web page

  1. A HTTP connection is opened and the main page (let's say index.html) is downloaded.
  2. Your browser parses index.html and figures out what other resources it needs:
    • CSS
    • Javascript
    • Images
    • Stuff in a or (more "main pages")
    • Probably more stuff I'm forgetting
  3. Your browser grabs all of those items.
    It can do this one of two ways"
    • Serially: Downloading each item in sequence (one connection at a time)
    • In parallel: Downloading several items at once (multiple open connections).

HTTP keepalive theoretically keeps the number of connect() cycles down, but some browsers may not support HTTP keepalive (or may decide not to use it for whatever reason), in which case every item being downloaded means another connection gets opened to the server.

One web page is therefore potentially a huge number of connections - easily enough to trip your firewall, especially if you have more than one user on an IP or your users browse actively.


Rather than blanket blocking you have a couple of choices:

  1. Tag matching packets in pf and bandwidth-limit them with altq.
    This isn't really ideal - you're still going to catch legit users - but at least they're throttled (slow), not blocked (broken).

  2. Use mod_evasive, mod_security or similar
    These modules are more work for Apache, but are more configurable and will help you avoid catching legit users.

share|improve this answer
    
Good insights. I appreciate your alternatives, as well. I suppose I'll have to forgo the limiting for now. –  myferalprofessor Feb 15 '13 at 19:58
    
@myferalprofessor If you're really having trouble with web scrapers mod_evasive is probably your best option. Doing throttling right (slowing down as requests pour in, then speeding back up after idle time) is a lot of work, and pf isn't really up to the task (it's outside it's design scope). –  voretaq7 Feb 16 '13 at 3:11

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.