0

Setup

  1. 4x blade setup in a single chassis each with 48 cores and 1 TB memory. VMs managed with Proxmox.
  2. A "parent" service with 7 VMs running Linux (Debian)/Apache/PHP calling several microservices using cURL (multi) from PHP. Each of these are assigned 4 cores and 4 GB memory
  3. 1 load balancer/cache for one of the microservices (NLP) running Linux (Debian) and Varnish (configuration: https://pastebin.com/prE3ZtmN) with 16 CPUs and 16 GB memory.
  4. 6 Vms of the NLP microservice each with 6 CPUs and 12 GB memory.

Problem

When load testing the parent service (with several hundreds concurrent requests) the 7 instances will generate thousands (> 10k) of requests to the NLP balancer per second.

During this load test the load balancer becomes slow/unresponsive - requests that usually take 1-1000 ms to complete (depending on the complexity of the request and always <10 ms if cached) may take several seconds to complete if not timing out entirely.

Things I've checked/ruled out as the source of the issue

  • The NLP VMs: When load testing these separately they can handle ~10k RPS in each VM before 100% CPU utilization. During the parent service load test they do not reach 100% CPU.
  • The network bandwidth: Using iperf I've tested the bandwidth between VMs on the same blade (~10 Gbps) and different blades (~3 Gbps). Those transfer rates are significantly higher than what is measured in Proxmox for the load balancer (and the blade the load balancer VM is running on).
  • Disk bandwidth: Proxmox reports < 10 MB/s of disk activity on the load balancer and NLP VMs combined.
  • Varnish (at least as far as I'm aware): Calling the load balancer from a separate VM (with a request I know to be cached) I measure response times of several seconds, but Varnish reports serving the response in <1 ms (varnishlog output: https://pastebin.com/kQSdC7p1). CPU utilization during the parent service test is also only ~60%. I also tried running Varnish on several ports simultaneously with no success.

My conclusion

I think it's a Linux TCP connection issue - in particular something related to maximum concurrent connections and wait time. That would also explain why other network activity (like using PuTTY) is slow during the load test.

Is it likely that I am correct in my assumption? What options do I have to solve the problem?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.