I run a Linux router (more precisely OpenWRT) on an internet connection with very limited bandwidth, around 1 MBit/s downstream and some dozen kBit/s upstream.

There are several machines on the net that do low-bandwidth stuff, like playing web radio or sending measurement data. Other machines may start normal downloads for software updates occasionally.

Whenever a machine starts a download, the low-bandwidth stuff gets choppy. I suppose that the bandwidth of the stream gets reduced because there's another connection on the router, although it woudl "fit" nicely into the WAN b/w. This is a bit against my intuition, and I would like to configure the router to allocate bandwidth more fairly.

By "fairly" I mean:

Suppose there is 1 MBit/s downstream bandwidth, and 64 kBit/s of it are used. The next client that accesses the WAN should get at most (1 MBit - 64 kBit)/s bandwidth. If and only if the downstream bandwidth is all used up, the individual connections' bandwidth should be lowered, and it should be adapted so that connections are throttled proportionally to their size (the smaller, the less).

First of all, is my understanding of the problem correct? If so, what can I do to influence the router's bandwidth allocations? Note: I do not want to what is usually recommended in the literature, namely limit the bandwidth of each client to a fraction of the total bandwidth available. There's just too little WAN speed at my site to do that.

  • Running openwrt is a bit of a problem. There are application level solution for http if you are using a proxy like Squid. The delay_pools configuration gives you some flexibility. – Zoredache Jan 30 '12 at 22:59
  • 1
    @ErikA -- you're right, I just noticed it myself. My questions here were a bit open-ended and I didn't get as detailed answers as I would have liked, but meanwhile, I can accept some of the not-so-complete ones with good conscience. – jstarek Jan 30 '12 at 23:03
  • 1
    @jstarek - right on. Thanks for "giving back"! – EEAA Jan 30 '12 at 23:04
  • @Zoredache - Not all my traffic passes through proxies (a lot of it is also handled by approx), so I'd prefer a more centralized approach. – jstarek Jan 30 '12 at 23:05
  • I just noticed wiki.openwrt.org/doc/howto/packet.scheduler/packet.scheduler -- this looks promising, I never found it since I did not know the term to search for was "packet scheduler". I'll look further into this. – jstarek Feb 3 '12 at 0:25

Your understanding of the problem is generally correct, but the kind of solution you propose is VERY complicated to implement. The questions of "What is a client?" and "What is a connection?" come up, and can be difficult to answer well.

The more typical bandwidth-limiting strategy is something like this:

  • Define the limit of the upstream bandwidth (say 1Mbit/sec)
  • Optionally Reserve some amount of that bandwidth for "system administration" (say 64Kb/sec)
  • Optionally "guarantee" a certain amount of bandwidth to a specific purpose (say 192Kb/sec for VOIP)
  • Allow everyone to use the remaining bandwidth (768Kbit/sec).

VOIP may use more than 192Kbit/sec, so we may let it borrow from the "everyone" pool (or vice-versa). When the "everyone" pool is saturated start dropping packets just like we would if the upstream link were REALLY saturated (using, e.g. Random Early Detection to pick your drop victims.

Typically this is done on UPSTREAM traffic. Downstream traffic can be limited the same way, but you can't avoid link saturation from downstream traffic (the packets still have to come to your firewall before the drop decisions can be made, so they're still traveling down your wire. The result is a surge of traffic followed by a natural decay as the TCP protocol senses "link saturation" congestion and the remote side backs off its send rate until the packet loss stops).

Also note that this doesn't guarantee "fairness" to the client machines, except to the extent that Random Early Detection will drop packets "randomly" (randomly enough that it won't always be client A that gets packets dropped when the link is saturated). What you're counting on is that the "random" drops will naturally shape traffic to an extent that you don't have to worry about one client being starved out while another hogs all the bandwidth.

An out-of-the-box solution in your particular case might be to limit the bandwidth available for updates (presumably these come from known subnets, so limit those), but this is still subject to the caveats I mentioned above.

Alternatively, if you have hardware available, you can distribute your updates from a local server (WSUS, Local apt mirror, etc.) -- This would let you schedule those updates to be pulled locally during off-hours when nobody is using your network, and ultimately would save you a lot of bandwidth transferring individual updates for each machine.
Since the updates are already local it doesn't matter when the individual client machines pick them up - they're not going out to the internet, so as long as you aren't saturating your local network (pretty tough!) you won't suffer significant performance issues. The downside of course is you need to invest time and hardware in setting up the update server.

  • Thanks for your detailed answer. I already proxy everything I can (web traffic as well as software updates for different OSes) and have the proxies speculatively fetch data that will probably be needed the next day between 3 and 5 AM. -- Can you elaborate a bit on how you propose to put traffic into those categories? Would you consider "system administration" traffic simply all SSH packets inbound to the main server, and "system update" traffic everything where the dst-range corresponds to my Linux mirrors? – jstarek Jan 31 '12 at 8:29
  • Most decent firewalls (cough*pfSense*cough) have the ability to assign traffic to groups (in pfSense "queues") based on some characteristics. The OpenBSD pf documentation talks about this in some detail. "System Administration" traffic is whatever you define it to be: All connections to, All SSH (port 22/TCP) connections, All SSH connections to, All connections from *insert subnet here*, etc. – voretaq7 Jan 31 '12 at 15:47
  • OK, I'm afraid that's too coarse a distinction for my case. What problem is there (and I really ask this from a technical standpoint) with dealing with the connection between localhost:someport and foreignhost:80 as one entity? – jstarek Feb 2 '12 at 20:32

No, there is no good way to do this right now. The basic problem is that your ISP decides which packets to put on your link, and it doesn't have any information that it could use to make that decision. The short, sad answer is that consumer Internet access just isn't set up to do this.

  • Aren't congestion mitigation techniques the same for every TCP/IP-connection? Delayed ACKs and so on should work, shouldn't they? – jstarek Feb 2 '12 at 20:31
  • @jstarek Assuming that's true, what's the good way to delay ACKs right now? – David Schwartz Feb 2 '12 at 23:45
  • That's part of what I'm asking here :-) I was reacting to your statement that "my ISP decides which packets to put on my link" -- the speed at which my communication partners talk to my router should be controllable by the usual congestion control mechanisms. – jstarek Feb 3 '12 at 0:11
  • Perhaps it should be, but so far as I know, nobody's gone and done it. I don't believe there's any good way to do this right now. – David Schwartz Feb 3 '12 at 0:15

There is this fairly old script that does just this. I have been using it for years now.

It does quite complicated stuff behind the scenes, but the actual user-facing configuration is pretty simple.

My fork: https://github.com/kalmi/FairNAT (It has some small fixes allowing it to run on recent versions of Linux)

There is a version of the script for openwrt, but it have not used it.

(I had to turn off TOS support, because it caused random TCP resets for some sites (wikia, imgur, etc..). It's probably due to some weirdness at this particular upstream ISP, but I don't know.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.