2

I have a strange setup where the ISP provides a very fast connection (10 Gbps) but limits each connection to 50 Mbps. This is fine for multi-threaded applications where I can just boost the number of threads. But I would like to solve this problem for single-threaded applications as well. I am running Linux and this is all TCP traffic (has to be) - I feel like there is a good way to do this using iptables but this is a little out of my depth. I am limited to one NIC card on the device. Is there a way to make a load balancer that just create multiple connections over the same NIC and then round robin the packets? (Basically I am already doing this in software for multithreaded apps.. but want to do it at the OS level).

problem solved what happened was.. not limited by the ISP at all. I was trying to connect to devices that have a fixed TCP window size. changing those devices to dynamic TCP window size increased the per-connection throughput to the max. more here:

https://en.wikipedia.org/wiki/Bandwidth-delay_product

2
  • Did they say why they did that? Oct 23 at 22:11
  • This is a hosted environment - you can access your servers over the public internet for relatively cheap. But our org has opted for dark fiber. We actually pay for 10 Gbps. So I can get 10 Gbps... with a zillion threads. I think the ISP didn't realize how many people would opt for dark fiber so they underprovisioned it big time.. and now they have to speed limit people on a per-connection basis (without violating their total speed limit that we paid for.. thats in the contract). This whole thing is really expensive.. trying to get our moneys worth : )
    – mohotmoz
    Oct 24 at 11:49
2

I don't think there's a point.

Of course, you can use multiple source IPs (whether from multiple hosts, multiple NICs or even a single NIC) but then again, you'd likely NAT all of those to the public IPv4 address you've got - so there's no difference from the outside. You can just as well just use multiple threads.

But obviously, you can just try binding multiple source IP addresses to a single NIC, not much to it.

2
  • Thanks for the quick reply - I am going to try to create a bunch of IPs on my host (same NIC), then use iptables to round robin to them.. and then NAT out the other end (all on the same host). Maybe this will somehow trick the system into thinking they are separate connections. I am not sure HOW they are tracking.. but it can be the exact same source and exact same destination.. as long as it is in a different thread, it gets its own bandwidth!
    – mohotmoz
    Oct 24 at 11:45
  • Likely, multiple threads / socket connections are all that you require. Throwing in different IP addresses before NAT can't really make a difference (unless they're required to make the application use ECMP).
    – Zac67
    Oct 24 at 14:00
1

There's two ways this has been limited - on a per-VM basis or on a per-vNIC-basis. If it's on a per-VM basis there's nothing you can do other than threaten to move to a different provider or actually move providers, obviously the former might not work. If it's on a per-vNIC basis then you could either add more yourself if that's possible or again ask the provider to add them, and again if they refuse threaten to leave.

Ultimately you have a business need that it sounds like isn't being met by your provider, so be prepared to pay a little more to get what you need.

2
  • Thanks so much for the quick reply - it is really weird for sure. They are somehow tracking the each connection and then limiting it - but they don't care if the source and destination are completely identical. My multi-threaded code is literally doing the exact same thing in parallel (same source, same destination - in this case over HTTPS but have tried others and it doesn't matter). I guess I'm wondering if there is a way to abstract this out to the OS level so every TCP connection turns into.. say.. 50 of them.. as opposed to having to re-tool each piece of software to do this internally.
    – mohotmoz
    Oct 24 at 11:43
  • Said differently.. it is not on a per-VM or a per-vNIC basis... it is definitely on a per-connection basis... I am just not sure HOW this is tracked and if there is an elegant way to get around it : ) We are guaranteed a certain speed (it is dark fiber).. see my comment to the other poster above.. but they are limiting our per-connection speed I think to avoid congestion. I have tested the exact same setup with a different ISP that doesn't thread limit (we have redundant ISPs.. redundant data centers..etc) and it is like 50x faster per thread.
    – mohotmoz
    Oct 24 at 11:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.