I have a 2 apache with mod_php web servers behind a load balancer, the OSs are CentOS 6.x. They are supposed to be IDENTICAL. They are supposed to receive about the same requests using round-robin. One of them was performing a lot worst than the other.
I thought it was receiving more requests for some reason:
- That is not the case.
- The requests were about the same URLs and about the same quantity of requests at about the same time. Still the performance is worst.
- Well I decided to help the server while I found a solution. so I re-configure the Load balancer.
- My configuration change of the load-balancer was to send the requests to the server with less connections open. After this the server started receiving less requests than the second server and performance was still bad compared to the other. Just to stress the difference, at this point requests are distributed about ~35 to ~65 %. In general about the same URLs requests ratio.
- Now, I thought we had some other user software level difference between the two servers. That was NOT the case.
- We manage configurations and code automatically. PHP has the same modules loaded, apache also. No difference between code.
- hardware is the same.
- As additional information we DO NOT have sticky sessions. Requests get randomly directed to both servers. We also tested that. The difference now after the config change is that the server with the issue receives much less requests.
- The servers do not have any meaningful time difference on processing requests or connection to other components like DB.
Given this background check current CPU images. The server with the problem: web1 image CPU performance Notice RED means system CPU time not user, apache is user space. Please also notice that these images are from after the change was done to get less requests to web1(with the problem). Before we have serious CPU and load average problems Then here is the other server.Web2 image CPU performance. No IO, RAM or other hardware problem was detected.
This made me suspect of kernel issues.
top seems to say the same.Top
Web1 vs Web2 image performance
As I can not find any other difference. I actually have enough with this information as the servers are supposed to be the same. BOSS want me to be able to link this to the actual problem before removing kernel modules. I tried a few things, included perf, but have not managed.
perf record -F 99 -a -g -- sleep 30) is yielding a result like this, but I do not really understand how to use it or how it can help me define results.web1 on the left and web2 on the right. if you see it lists the calls that take CPU time/Percentage, but I do not really know what they are.
Do you agree that this should be probably related to the kernel modules? Why yes or not?
Can someone give me some idea about how to get statistics on modules usage and time/percentage of CPU usage per module, etc?
Any further thing that I am not thinking about?