I have a pretty good handle on load balancing web applications with nginx/HAProxy/etc. As I understand it, in these cases you're mostly limited by things like the number of concurrent connections and TLS handshakes, but each request is a relatively small amount of data transferred.

I'm currently working on a service that transfers a lot of data per request. Think video streaming or peer-to-peer file transfers proxied through my server.

I'm wondering what the typical way to load balance something like this is? Even if HAProxy could handle the bandwidth, having everything going through a single VPS would saturate its network fairly easily (at least on DigitalOcean; maybe an AWS 25Gbps instance would be enough). I'm thinking redirects might be the way to go, but I'd like to avoid that and wanted to see if there's a better way.

One other piece of information about my service is that requests to the same URL must go to the same upstream server. But it only cares about the path. Query params, headers, etc don't matter.

I did a quick check on youtube, and it looks like they use redirects to almost random-looking domains like r5---sn-qxo7rn7l.googlevideo.com, r1---sn-qxoedn7e.googlevideo.com.

EDIT: Additional details at Tim's request:

The data should be considered uncacheable. Imagine peer1 has a 4GB video file they want to share with peer2. peer1 connects to lb.example.com/path and waits for peer2 to connect. peer2 connects to lb.example.com/path, and and the data is streamed through the server from peer1 to peer2.

The way I would do this with redirects is peer1 connects to lb.example.com/path. path is hashed, and the value of the hash is used to determine whether to redirect peer1 to instance1.example.com or instance2.example.com. When peer2 connects with the same path, it ends up on the same upstream instance. The hash space would be evenly divided between instance1 and instance2.

I agree AWS egress is too expensive which is a big part of why I'm trying to design a scalable solution that doesn't depend on a single huge network pipe.

  • Can you please edit your question to add a bit more detail about the content being served, cachability, how you see redirects working, etc. Using a CDN is the obvious solution, but I guess you've considered that. AWS bandwidth is really expensive btw - I like AWS but for egress high bandwidth applications you'll pay a lot of money. – Tim Nov 23 '19 at 18:04
  • I added some details. – anderspitman Nov 23 '19 at 20:18

For peer to peer you may be able to connect directly to the other node, removing the bottleneck of your infrastructure completely. And the costs of ingress and egress bandwidth on every session. Especially viable with IPv6 which brings back global addressing. Matchmaking and firewall traversal may be tricky.

haproxy can scale up to hundreds of thousands of connections, but you probably want to scale out too. 25 Gb Ethernet is relatively inexpensive to own, but rather a premium to rent. So many nodes is a given.

For load balancers, keep as little state as possible. Stateless level 4 routing. Direct server return so the backends respond directly to clients. Both make some load balancer tricks difficult, but that's the price of simple stupid-fast stateless.

Consider Vincent Bernat's lab Multi-tier load-balancing with Linux. IPVS + keepalived provides stateless L4. haproxy provides L7 termination, with direct server return due to send-proxy.

Whatever your design, the total bandwidth to your users needs to exceed your peak transfer. Either you spin up several load balancer instances with fast interfaces, or hire load balancer as a service from your cloud.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.