We are planning to have multiple high-capacity servers with the same application configuration (data analysis pipeline) and would like to send requests including a data stream to each server, based on its available capacity (i.e., starting with the least-utilized server and stopping requests when utilization is over 70% in all servers). I'm familiar with options for AWS, but I'm interested in the best practices for physical (non-cloud) infrastructure.
1 Answer
95% of your question depends on the loadbalancer you're attempting to use. If you haven't decided on a specific loadbalancer, then you'll need to define what it is you want to balance. Memory? CPU? Network Traffic? requests-per-second? IO? etc... There is no "magic-bullet" that can balance everything perfectly.
In large clusters, most of the load-balancing is done with a combination of round-robin DNS and several different types of load-balancers. There are HTTP load-balancers, database-server load balancers, application load balancers, network load balancers, etc... they all monitor and balance based on specific requirements.
To successfully do any high-quality load-balancing, you need to have a plan from the beginning, and evaluate every piece in the entire puzzle for potential failure points and bottlenecks. Once again, there is no such thing as a "one-size-fits-all" solution. Depending on the scale of your framework... designing and maintaining a loadbalancing solution can be a career all by itself.
