8

Assume an office of people, they want to limit HTTP downloads to a max of 40% bandwidth of their internet connection speed so that it doesn't block other traffic.

We say "it's not supported in your firewall", and they say the inevitable line "we used to be able to do it with our Netgear/DLink/DrayTek".

Thinking about it, a download is like this:

HTTP GET request
Server sends file data as TCP packets
Client acknowledges receipt of TCP packets
Repeat until download finished.

The speed is determined by how fast the server sends data to you, and how fast you acknowledge it.

So, to limit download speed, you have two choices:

1) Instruct the server to send data to you more slowly - and I don't think there's any protocol feature to request that in TCP or HTTP.

2) Acknowledge packets more slowly by limiting your upload speed, and also ruin your upload speed.

How do devices do this limiting? Is there a standard way?

1
  • Rate limiting how fast received packets are allowed out of the firewall onto the LAN causes slower acknowledgements and the congestion handling in the sending server's TCP stack throttles back sending speed. Thanks. I can get it to work like that with tessting. I have upvoted several answers, but I can only mark one as an answer. Nov 30, 2011 at 18:20

5 Answers 5

12

TCP itself implements the congestion control.

These rate limiters will simply throw packets away over the limit. TCP handles this, ensuring that the packets all arrive and all arrive in order; the client doesn't ACK for the dropped packets, and they are resent by the server.

The server's TCP stack will resend the packets, and it will also dial back a bit on its send rate because it figures there's congestion between it and the client. It'll speed back up until the rate limiter drops packets again, and so on.

7
  • So I should apply a QoS policy rate limiting how fast the firewall spits HTTP traffic out on the /LAN/ interface? And then just let TCP handle it. "it will also dial back a bit on its send rate" is another part I was missing. Nov 30, 2011 at 16:57
  • 2
    Yup, that's right. The server could just keep throwing data at your link, saturating it before the QoS was applied - but, as long as it's a good TCP citizen, its send rate will establish an approximate equilibrium with the rate at which its data is actually getting through the rate limiting. Nov 30, 2011 at 18:12
  • Yes, TCP implements it's own congestion control, but it's not necessarily that effective. Anyone with any experience with torrents knows this. Basically, most implementations of TCP congestion control break down when there are hundreds of active connections on the network.
    – user606723
    Nov 30, 2011 at 19:08
  • 1
    @user606723 If torrents are a problem, you should be using a packet shaper at your egress to discard that traffic. This will cut the torrenter off from the tracker and keep other people downloading the same torrent from flooding your ingress with connections. Problem solved.
    – MDMarra
    Dec 1, 2011 at 19:12
  • 1
    @user606723 Why yes, congestion control will have an uphill struggle when thousands of connections are being initiated all the time with quick-starting TCP, since new connections know nothing of the state of the connection they're building. Hundreds of active connections, though? Maybe that'll bog down a slow home link.. Dec 1, 2011 at 19:12
5

The best description I've ever heard that made sense of TCP's inherent throttling method was off a recent Security Now podcast. To quote Steve Gibson:

So by universal agreement, TCP being this very clever protocol, it does something called a "slow start." It's generally given permission to send some number of packets without acknowledgment. So the idea being, let's just get things moving here. And typically that number is two. And so when TCP starts, it can send two packets off, one after the other. Without having the first one acknowledged, it'll send the second. But then it waits. And then the rule for throttling is we allow the number of unacknowledged packets to increase by one for every acknowledgment we receive.

So let's think about that. We allow the number of unacknowledged packets to be increased by one for every acknowledgment we receive. So we first send off two packets as our agreed-upon start. They get acknowledged. So we have our first acknowledgment. We were allowing ourselves to send two. Now, with the receipt of this first acknowledgment, we increase that by one to three. So we can now send three more packets off without any further acknowledgment. When an acknowledgment comes back for whatever we've sent before, we increase that to four. This is known as the "congestion window." It's not a window that's ever sent on the line, that is, it's not like the receive window, which is 16 bits of the TCP header that tells us how much data we're able to send ahead. This one is - it's a window. It's basically a counter maintained by the TCP stack for the purpose of avoiding connection, the idea being that at some point this is going to break.

If we keep increasing the number of unacknowledged packets we're allowed to send by one every time we receive an acknowledgment, at some point we're going to reach a limit. And the beauty of this system is that it will, as we start trying to send packets faster than the weakest link, literally link, between routers, at some point we find the point where the weakest link breaks. It drops the packets we're trying to send because we're trying to send them too fast. So acknowledgments from the other end stop because the data is no longer getting through.

And what TCP does is, if it has failed to receive - and this varies in strategies. Over time, the strategy, the actual congestion avoidance strategy has varied a lot. There's names like Tahoe and Reno, and a whole bunch of other ones that you'll see if you do some Googling and Wikipediaing, which specific exactly what the behavior is. But the idea is that, when the sender realizes that its data is no longer getting through because it's missing acknowledgments, it cuts back its sending rate quickly. Typically, it divides it in half. So it dramatically scales it back, and then goes back to increasing it.

So essentially what this means is that losing packets is the signaling function for "We can't send the data any faster," and that the TCP senders at each end of a connection, all over the Internet, are always sort of - they're trying to go faster than the maximum speed that is available between the two endpoints, that is, that weakest link, wherever that is, and they're always pushing it to the limit. So given that there is a point somewhere that is weaker than their ability to send packets, they're going to find it because they will pump packets out. As long as there's data to be sent and they've got a high bandwidth connection, the sender will increase the rate of sending, that is, the number of outstanding packets, the packets that are allowed to be out there on the fly as acknowledgments come back, aggressively keeps moving that number upwards until it pushes it too far. Then it backs off a lot, and then again moves forward.

So this is what's actually going on between TCP connections which are, like, probably, I don't know what percentage, but the vastly greatest percentage of traffic on the Internet are over TCP connections. All of our operating systems down in the kernel, in the so-called TCP stack, have these counters. And when we're sending a file, when we're uploading a big file or we're receiving a web page, the server at the other end is doing the same thing. It's pushing, on an individual connection basis, as many packets that have not yet been acknowledged out as it can, increasing the packet rate until it hits the point where it starts to fail or stutter. Then it backs off, to sort of allow things to recover, and then starts working back up again.

And so that ends up being a sort of a self-throttling system which, given the constraints, I mean, it really seems kind of funky and crude."

3

So, to limit download speed, you have two choices:

1) Instruct the server to send data to you more slowly - and I don't think there's any protocol feature to request that in TCP or HTTP.

2) Acknowledge packets more slowly by limiting your upload speed, and also ruin your upload speed.

3) You router / firewall device puts incoming data into a QoS bucket and only empties that bucket at the rate you requested. Incoming data will adapt to that speed as computers inside will only see acknowledge receipt at that speed. Also, the occasional (purposefully) dropped packet works really well for slowing down a connection.

When trying to find a device that handles this, look for QoS (Quality of Service) in the configuration / documentation. Linux (or BSD) boxes are also handy for this.

2
  • That almost makes sense - so the technique is to limit the rate of acknowledgements and to do so by pretending to the LAN device that the server is sending more slowly than it really is? So there will be a burst filling the connection at first, then not after that? Nov 30, 2011 at 16:50
  • 1
    @TessellatingHeckler Yes, that's it. Also, the "burst" shouldn't be a very large burst courtesy of en.wikipedia.org/wiki/Slow-start. Nov 30, 2011 at 17:01
2

You use a firewall or device that supports QoS (quality of service) limiting.

You could build a Linux system to act as the office gateway and have it use traffic shaping to achieve this. Just needs multiple NICs installed and then every machine points to is as a gateway.

As a bonus, you could configure a proxy server on it to help ease traffic too. Something like Squid. There may be turnkey routing appliance distributions that can do this too.

5
  • How does QoS help? By the time your device can apply QoS to a download, the incoming traffic has already arrived over the internet connection and potentially blocked it up while doing so. QoS can apply to outgoing traffic, but it can't do anything about incoming traffic because by the time it sees the traffic, it's too late. Nov 30, 2011 at 16:48
  • 3
    It can shape traffic where you can control it. There's no way you can tell a remote server to limit the speed at which you get the data, but you can lower it at your ingress point. Otherwise, you'd have to talk to your provider about traffic shaping in their network to your feed. Nov 30, 2011 at 16:55
  • Also the proxy server can help ease some congestion as well. But otherwise, you're going to have to shape it at the ingress point. Nov 30, 2011 at 16:55
  • My original question was: it appears you can't shape at ingress point, because any control you can apply happens after the traffic has passed through the bottleneck, how does any amount of technology deal with that? "Apply QoS e.g. with Linux" and "use traffic shaping" may be practically what to do, but I was looking for an explanation of how that can possibly help. (and now have some in other replies). Nov 30, 2011 at 17:49
  • @TessellatingHeckler: The method that I like also enables the use of ECN which actually does tell the sending server to slow down without dropping packets. This method is to apply the rate limiter such as RED to packets leaving on the LAN interface, instead of trying to use an ingress filter on the WAN interface.
    – Zan Lynx
    Nov 30, 2011 at 21:13
2

The HTTP protocol doesn't provide facilities to limit the used bandwidth, and even if it did, that would be a client-side setting, on which network administrators couldn't have any control.

Bandwidth limiting (also known as "Quality Of Service") is usually managed on routers/firewalls, which handle all the incoming and outgoing traffic to/from a network; the ones that support this usually let you configure policies such as "let any single client computer use at most 10% of all the available bandwidth", or "give SMTP priority over FTP so that emails can flow even when someone is doing an heavy download".

How exactly this is accomplished depends on the router/firewall used, but the most basic way is to just throw away packets that exceed the configured limits; TCP will ensure they get retransmitted, and will eventually be able to pass through the bottleneck.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .