Optimize Apache for high concurrent requests

Question

I am running a Apache2 Ubuntu server which serves the API request from mobile apps.

Recently I am facing a bottleneck that when the request getting higher (more concurrent users), the request response is getting slow. Previously ~1 or 2 seconds will do, but once the concurrent users increase (at certain peak hour), it can delay up to 10 seconds or more, even-though the server load, CPU and Memory remains very low.

My target is to increase the capability of Apache2 and Ubuntu Server to serve as much concurrent users as possible at lowest response time. The memory and CPU is not a major consideration factor as the VPS specification can be scaled up if it hits the limit. How can I do that?

What I have done so far:

1. Configure ulimit /etc/security/limits.conf and add the following:

   • soft nofile 40000
   • hard nofile 40000
   • soft nproc 40000

2. Add the following line to /etc/pam.d/common-session

   session required pam_limits.so

3. Configure Apache2 /etc/apache2/mods-enabled/mpm_prefork.conf

    <IfModule mpm_prefork_module>
            StartServers              20
            MinSpareServers           25
            MaxSpareServers          100
            MaxRequestWorkers         150
            MaxConnectionsPerChild   0

            MaxClients              8192
            MaxRequestsPerChild     0
            ServerLimit             8192
     </IfModule>

4. Add the following line to /etc/sysctl.conf

   fs.file-max = 2097152

The response time seems to be improved (though some requests still delayed), but not satisfactory as it is still much slower than non-peak hour.

Answer 1

Stop using prefork module and start using event mpm in apache. Prefork is the old method and moving forward you want to install event mpm.

Answer 2

This may not be your web server but rather something else in the chain.

1. DNS. In situation, when it's not obviously the database, there is a good chance it is DNS. Do a nslookup and see how long the command takes. Then lookup a couple of domains.
2. Its the network, do a traceroute and see if any of the hops on your network are taking longer then usual.
3. Netstat -i 5 to see if there are any errors on the network interface.
4. Make sure KeepAlive is on . Especially if you something like a ajax heavy site you will improve performance.

I don't believe switching MPM's will help you. Your situation isn't resource constraint. its something else.