Why are there tons of PHP processes open on my server?

Question

Today I saw that a website of mine isn't working so I ssh'd to the server and executed ps -eF. I see about 200 PHP processes that are running all for 4 hours.

Apache is built with mpm event and mod fcgid.

I killed all the PHP processes and now it's running fine, why does this happen? is this expected behavior?

I don't really understand how processes how Apache keeps track of the number of PHP processes and their process IDs, so it would be nice if someone can also give some reference when I can read about this.

Also, I used the "ab" command (Apache Benchmark) to see if this happens all the time, so I ran it about 4-5 times in a row with 30 concurrent requests and again there are like 150 PHP processes running, when I keep running "ab" now it doesn't spawn more processes and the website is still working. To be clear the 4-5 times I executed "ab" were not a the same time, as soon as one "ab" process finished I executed another one, but each time I executed "ab" I did 30 concurrent requests.

Please shed some light on this! Thank you :)

Answer 1

Yes this is expected behaviour. It would appear you had 5 copies of ab running at the same time. This resulted in your 150 PHP processes to handle the 30 requests for each of your 5 ab processes.

Apache works best when all its threads can run in memory. There will be a load at which the threads will need to be swapped to disk. Response time will degrade rapidly when this occurs. Killing processes as you did will improve performance. Server performance may become poor in this case. If the maximum threads are configured too high it is relatively easy to perform a DOS (Denial Of Service) using this behavior.

Another reason you can end up with problems is if two processes deadlock on two resources. If the other processes need either of these resources, they may fail to respond. Apache will limit the number of processes created in this case. Killing either of the deadlocked processes should clear the problem. Depending on the normal locking order it may require a specific process to be killed to prevent another deadlock. This will have little or no impact on server performance.

Programming errors may also cause threads to hang. This can result in the theads sticking as you saw. Finding what happened is difficult after the fact, but the contents of your apache server's logs may help. Check the error log for problems. The access log may have entries for the requests that hung around the time you killed the processes.

Apache should kill off some of the processes if they are not used for a period of time. This is controlled by the MaxSpareThreads parameter. Check your control file which should have some comments about the parameters for Threads, Servers, and Clients. The default values are usually good.

If there are problems with code, setting MaxRequestsPerChild to somewhere in the range 5000 to 100 often helps. The smaller the number the more often new threads need to be created and the harder the apache server needs to work.

EDIT: Apache will open up a number of threads at startup. This depends on the interaction of several parameters. See the MPM configuration documentation for details on the parameters. This is independent of any loading of the server, and normally be the minimum number of threads a running server will keep running.