Suppose I have 4 servers and I'd like to send the first 10 requests to the first one, and then send the next ten requests to the second, and so on. Unweighted round robin would send one request to the first, then one to the second, and so on.

If I apply server weights of 10 to each of the 4 servers, would I get this behavior? That the first ten requests go to the first server, the next ten go to the second, and so on back until the first server? Or would the weights 'cancel out', and Nginx would round robin as if they were all set to weight 1?

If not, is there some other way to get this behavior?

EDIT: I was asked why this question might be useful to others. I think it will be for specialized use cases.

What I'm trying to achieve is this: I have four inferences server for an ML model, fronted by Nginx. They can each enable server-side batching: that is, they collect input data from multiple request bodies, then perform a single inference on a whole batch of inference data, and respond to each request with the inferences they asked for. This "batching" behavior is extremely useful for inference using GPUs, which can perform many operations in parallel, and where overhead from data copying between device and host limits throughput when one request is handled at a time.

We can get 10x the throughput with batching with one inference server compared to no batching. The problem is: if we want to do batching on multiple inference servers, we need to be able to distribute requests in such a way that we don't lose the benefits of batching. With round robin, each GPU will be waiting for requests. With the kind of round robin I described above, I can send 10 requests to an inference server / GPU, and while it performs inference, move on to the next one, and so on. If this is done right, I think we can achieve closer to ~35-40x the performance compared to one GPU without batching.


  • I do think the question is useful, I edited my question. – Andre Apr 9 at 12:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.