We have recently replaced an RHEL 5.6 web server with RHEL 6.1. For both environments, the (stock) Redhat shipped Apache httpd has been used (i.e. yum install httpd). The server is serving PHP content and is busy (serving approximately 2500 - 4000 page requests per minute). The specifications of both servers are identical in terms of memory, storage and network connection. What we are seeing is significantly higher load averages on the RHEL 6.1 box - where the load average will (at times) spike to over 40 (all httpd processes) resulting in substantially degraded performance of the site. We have monitored the RHEL 5.6 environment and the load average does not exceed approximately 5 concurrent httpds. How can we investigate this issue ? Please bear in mind that this is a production environment, but we can compare "apples with apples" by switching between the 5.6 and 6.1 server.
Your 2500 - 4000 requests per minute translates to 40-60 requests per second. That kind of load hardly needs kernel level tuning, but more likely there is something wrong with your Apache or PHP setup. Some typical reasons include
You need to figure out what's different compared to your old server. Did you copy the configuration values from the old server to new one? Was the old server fine-tuned years ago and you have forgotten something vital?
What does Apache's server-status page show you?
If everything else fails, you can always use PHP's
Upgrading from 5.6 to 6.1 is a big step - you're probably running different versions of both apache, php, and perhaps a bytecode cache as well.
If you're using a bytecode cache (which I believe you should, with the kind of load you're describing), you should verify that it's working (I know apc has changed some configuration syntax between versions, for example). Which one are you in that case using?
You should have some metrics explaining what kind of cpu time is spent - if it's IOWait, System or User time. The same applies to memory usage, etc. How does these things look?
I will again recommend some kind of monitoring tool, like for example Munin.
Perhaps you can compare the output of
Edit: I assume apache and php are configured the same way.
Question: Is the both apache configurations the same i.e. are you using thread invokations or process invokations for each request?
If you are using process invokations, then a process is created for each request (with a few remaining behind) and thus a higher load factor.
If you are using thread invokations, then a number of requests will be handled by threads within a far fewer number of processes.
Thus, there might be nothing wrong.
While a lot depends on what the PHP is actually doing, IMHO the load average seems to be rather high even on the old machine - but obviously the pressing issue is the difference in behaviour between old and new.
Are you sure that the configs are same? Have you checked the apache error logs to see if its having any problems with any of the processing directives? Have you checked the permissions on the eaccelerator dir?
Does the box use external resources (such as DNS, database)? If so, is it on the same network as the old one? Are the RTT times to the other servers similar?
Can you easily downgrade the box to RH5.6? If so, does this resolve the problem?
Are you seeing a change in the bandwidth or request response times (%D)?
The difference isn't really all that great - since the bulk of the memory is in TXT segments marked as COW. Indeed, some benchmarks show pre-fork to be more scalable than worker threads on Linux.