- 4x blade setup in a single chassis each with 48 cores and 1 TB memory. VMs managed with Proxmox.
- A "parent" service with 7 VMs running Linux (Debian)/Apache/PHP calling several microservices using cURL (multi) from PHP. Each of these are assigned 4 cores and 4 GB memory
- 1 load balancer/cache for one of the microservices (NLP) running Linux (Debian) and Varnish (configuration: https://pastebin.com/prE3ZtmN) with 16 CPUs and 16 GB memory.
- 6 Vms of the NLP microservice each with 6 CPUs and 12 GB memory.
When load testing the parent service (with several hundreds concurrent requests) the 7 instances will generate thousands (> 10k) of requests to the NLP balancer per second.
During this load test the load balancer becomes slow/unresponsive - requests that usually take 1-1000 ms to complete (depending on the complexity of the request and always <10 ms if cached) may take several seconds to complete if not timing out entirely.
Things I've checked/ruled out as the source of the issue
- The NLP VMs: When load testing these separately they can handle ~10k RPS in each VM before 100% CPU utilization. During the parent service load test they do not reach 100% CPU.
- The network bandwidth: Using
iperfI've tested the bandwidth between VMs on the same blade (~10 Gbps) and different blades (~3 Gbps). Those transfer rates are significantly higher than what is measured in Proxmox for the load balancer (and the blade the load balancer VM is running on).
- Disk bandwidth: Proxmox reports < 10 MB/s of disk activity on the load balancer and NLP VMs combined.
- Varnish (at least as far as I'm aware): Calling the load balancer from a separate VM (with a request I know to be cached) I measure response times of several seconds, but Varnish reports serving the response in <1 ms (varnishlog output: https://pastebin.com/kQSdC7p1). CPU utilization during the parent service test is also only ~60%. I also tried running Varnish on several ports simultaneously with no success.
I think it's a Linux TCP connection issue - in particular something related to maximum concurrent connections and wait time. That would also explain why other network activity (like using PuTTY) is slow during the load test.
netstat -anoutput during parent service load test (had to split it into 4 parts to avoid Pastebin max file size):
netstat -soutput during parent service load test: https://pastebin.com/eN3ANb7s
Is it likely that I am correct in my assumption? What options do I have to solve the problem?