I have a web application that looks like this:
WEB SERVER (PHP, Apache) -> MYSQL NODE
This works perfectly without any issues, and has done for a long time.
However, when I introduce a load balancer (I have used both HAProxy and ProxySQL, both with the same resulting behaviour) to manage requests to my database cluster, I have some issues with Apache stalling.
The load balancer is installed on the web server alongside apache and accessed via the system's loopback device. To be explicit, the web application makes database calls to 127.0.0.1:3306 and this is where the load balancer listens.
To simplify things, this is my new setup:
[WEB SERVER (PHP, Apache) -> LOAD BALANCER] -> MYSQL NODE
When load becomes heavy, this is what what I see:
PROMPT~] apachectl status
...
...
WWWWWWWWWWWWWWWWWRWWRRWWWWWWWWWWWWWRWRWWWWWWWWWWRWWWWWRWWWWWWWWR
WWWWWWWWWWWWWWWWWWWWWWWWWWRWWRWWWWWWWWRWWWWWWWWWWCWWWWWRWWWWWWWW
WWWWWWWWRWWWWRWRWWRRWWWCWWWRWWWWWWWWRWWWWWWWWWWWWWWRWWWRWWWWWWWW
WWWWWWWWWWRWRWWCWWWWWWRRWRWWWWRCWWWWWWWWWWWRWWWWWWWRWRWWWW
The fact that this is occuring with multiple load balancers suggests that the issue is more general and possibly operating system related.
I see no errors reported by Apache, the load balancers or the database.
OS metrics such as CPU and Memory appear to be okay. I do see a spike in the following around the time the issue strikes:
- Number of TCP sockets
- Number of Apache processes
I feel like I am probably missing something elementery here and would appreciate any advice.
Thanks
