What is the best solution for traffic control in a large system (ca. 2000 users)?

Question

following situation: We're a group of students administering the internet connection for the local resident halls, with a total of about 2000 end users.

We have a traffic point system, every MB down- or upload costs points, new points are added by the hour. At the moment, we block a user's internet access when he has spent all his points (by placing him into a REJECT policy in iptables on our Debian gateway router).

We would like only to limit a user's bandwidth. What is the best way to do this?

The simple answer would be to set a rate-limit on the user's switch port (mostly Cisco Catalyst 3550s). However, this is undesirable, as traffic inside our own network and to the university network should remain unlimited. Is there a way to limit bandwidth only for packets with a certain destination or source IP range (so both egress and ingress) in Cisco IOS? I could not find anything.

The other way would be to control the traffic on our gateway router. Several solutions come to my mind:

• tc or tcng - seems like both have a rather arcane syntax and neither offer good features for doing per-IP traffic control. A dedicated QDisc for so many people would probably slow down the router quite a lot. Furthermore, documentation on both is pretty outdated.

• shorewall - seems to have a rather neat syntax for configurations, however, I'm unsure whether it can handle this amount of traffic and users and whether it's suitable for per-IP traffic limiting

• pfSense - looks like an OS intended for purposes like ours. However, it would require us to compeletely reinstall our gateway router. We don't have other BSD systems and pfSense would need to have very good traffic accounting capabilities (we're using fprobe-ulog and ulog-acctd there at the moment), too.

What is your experience? Which solution suits our needs and can be most easily maintained? Do you have other ideas?

If you need any additional information about our system, please don't hesitate to ask.

Thanks in advance.

---

EDIT: I have implemented the system with iptables and tc.

Every user has a /28-subnet, a VPN IP (both from 10.0.0.0/8) and an external IP, all are steered through one iptables chain. This chain has only one rule, a simple RETURN.

Every five minutes, a Python script reads out the byte counters of these rules. It resets the counters and updates the user's traffic point account in our PostgreSQL database.

If a user's point balance decreases below a certain threshold, two tc classes are created for this user (one for the incoming, one for the outgoing interface on our gateway router), the IPs are entered into tc filters belonging to these classes. The classes are speed-limited by an HTB.

Compared to the previous system with fprobe-ulog and ulog-acctd this is much faster as the byte counting is done by iptables.

Network speed has improved considerably for our users.

Answer 1

I'm not sure how interested you are in reconfiguring your entire setup (i.e., replace Debian), or how feasible it would be to place something like this behind your gateway, but FreeBSD has a feature in ipfw known as dummynet. Since it sounds like you don't feel like getting a dedicated hardware traffic shaper, this may be an option for you. We currently use it to choke SMTP traffic inbound and outbound through one of our proxy gateways to keep an aging NFS backend system from being overwhelmed and subsequently becoming unresponsive.

With some scripting and intelligent configuration of your rulesets, it would be feasible to be able to traffic control thousands of individual IP addresses.

Answer 2

I'll have to ask what we're using, but our ResTek people have an appliance at the Internet border that does quality of service functions. It has a default priority for unclassified streams, and prioritizes other traffic based on rules. It's the priority feature that really sells it, as they also operate a Squid-based caching cluster that is listed as highest-priority in the border appliance. People on their network then have the choice of surfing the web at standard (and somewhat sucky) priority, or use the proxy and get very zippy response. They also take gaming server priority requests, since they're latency sensitive but low bandwidth, as a matter of course.

There is a fair amount of work handling gaming server requests, but those are slowly going by the wayside as more and more games move away from private servers. Overall I hear it works pretty well for them. They can keep things like bittorrent from swamping out all other traffic, and still keep people's YouTube load times fast.

Answer 3

With that many users most of the simple software based stuff will start to break down.

Consider looking at packeteer and similar devices. These days they're almost more expensive then just buying more bandwidth (In the US & Europe, us poor folks in Australia still be very high rates).

Answer 4

dummynet is a very good suggestion in my opinion. But I am sure iptables are capable of traffic shaping too, so you can just do it on your debian box.

Answer 5

tc should work fine.. I believe what you want to do is change your current iptables rules to mark rather than REJECT traffic, then you can use a small number of tc rules to apply traffic shaping to those marked flows.

also, look at ipset for managing the list of throttled users http://ipset.netfilter.org/

Answer 6

Trafficpanel seems to be a solution. Among other features:

• Limiting max HTTP speed per connection
• Limiting total amount traffic of web resources per local network host
• Log internet traffic per local network host
• Limiting max traffic speed per local network host
• Limiting total amount internet traffic per local network host

Actually, never tried this thing, but looks ok.