Trying to find the offender by looking at the volume of traffic over time is likely to be too late. Remember that a user expect web responses in seconds. Limiting a user by analyzing the traffic flow is likely to be too apply the limit long after the browsing user has given up.
There are some things you can do with iptables (see e.g. hashlimit) but your users will probably notice the slowdown anyway. If you want to make sure that everyone gets a share, you need to play with prioritizing traffic. See the tc command. For an easier time, I recommend tcng. No matter which method you use, you will need some criteria to differentiate "bad" traffic from "good" traffic (other than quantity).
However, my experience is that well-behaved routers (be they linux boxen or enterprise-level routers) nearly always do a reasonable job of prioritizing traffic such that you don't get a total breakdown like OP describes. Before trying to do fancy stuff, I would try to see if there is some badly configured or designed hardware on the network. My presumption of guilt falls on ADSL routers, old Cisco PIX routers and cheap switches.