I'm running a CentOS 6 64bit server with 1.2 GHz CPU and 2.4 GB RAM. The server runs nginx and php-fpm with the APC extension and also mysql.
A couple of weeks ago the server starting continuously throwing 500 errors on HTTP requests; Google WMT was reporting around daily 450 server errors in Crawl Errors. If you were browsing the website an error was thrown every 3-4 pages or so.
This is how php-fpm was configured:
pm = dynamic pm.max_children = 45 pm.start_servers = 15 pm.min_spare_servers = 5 pm.max_spare_servers = 15 pm.max_requests = 1000
After checking the logs I found in the php-fpm error log that these were the 2 most frequent errors:
ERROR: fork() failed: Cannot allocate memory (12) WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 30 total children WARNING: [pool www] child 27455 exited on signal 11 (SIGSEGV) after 8393.397161 seconds from start
I followed the log's recommendation and increased the values of the above to this:
pm = dynamic pm.max_children = 70 pm.start_servers = 25 pm.min_spare_servers = 15 pm.max_spare_servers = 25 pm.max_requests = 1000
It decreased the frequency of errors, but not significantly.
After doing more research I discovered that one of the reasons for the "Cannot alocate memory" error could be an incompatibility between the installed PHP 5.4 and APC 3.1.9 so I went ahead and updated APC to version 3.1.13 which reported compatibility with PHP 5.4. This made the "Cannot alocate memory" error disappear completely but it was still throwing code 11 segmentation faults.
I discovered then the value of pm.max_requests could be too high so I decreased it to this:
pm.max_requests = 600
Again it lowered the frequency of errors thrown but not significantly.
After reading a bit more recommendations I installed Varnish as front end for the web server. This marked a significant decrease in errors but the logs show that the code 11 segfault error is still thrown once every 2-3 hours or so. This means there are still users who get server errors when they try to view my website. I would like to know if there are any other optimizations I can do to my web server.
Other info that might be useful: I'm getting around 2000 daily unique visitors but I also get a high level of spam attacks on my website. I have set up Cloudflare as proxy in front of my server and ever since the spam level decreased significantly. I also use MaxCDN for static files such as scripts or images so these are usually not requested from the web server.