2

I'm running a CentOS 6 64bit server with 1.2 GHz CPU and 2.4 GB RAM. The server runs nginx and php-fpm with the APC extension and also mysql.

A couple of weeks ago the server starting continuously throwing 500 errors on HTTP requests; Google WMT was reporting around daily 450 server errors in Crawl Errors. If you were browsing the website an error was thrown every 3-4 pages or so.

This is how php-fpm was configured:

pm = dynamic
pm.max_children = 45
pm.start_servers = 15
pm.min_spare_servers = 5
pm.max_spare_servers = 15
pm.max_requests = 1000

After checking the logs I found in the php-fpm error log that these were the 2 most frequent errors:

ERROR: fork() failed: Cannot allocate memory (12)
WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 30 total children

WARNING: [pool www] child 27455 exited on signal 11 (SIGSEGV) after 8393.397161 seconds from start

I followed the log's recommendation and increased the values of the above to this:

pm = dynamic
pm.max_children = 70
pm.start_servers = 25
pm.min_spare_servers = 15
pm.max_spare_servers = 25
pm.max_requests = 1000

It decreased the frequency of errors, but not significantly.

After doing more research I discovered that one of the reasons for the "Cannot alocate memory" error could be an incompatibility between the installed PHP 5.4 and APC 3.1.9 so I went ahead and updated APC to version 3.1.13 which reported compatibility with PHP 5.4. This made the "Cannot alocate memory" error disappear completely but it was still throwing code 11 segmentation faults.

I discovered then the value of pm.max_requests could be too high so I decreased it to this:

pm.max_requests = 600

Again it lowered the frequency of errors thrown but not significantly.

After reading a bit more recommendations I installed Varnish as front end for the web server. This marked a significant decrease in errors but the logs show that the code 11 segfault error is still thrown once every 2-3 hours or so. This means there are still users who get server errors when they try to view my website. I would like to know if there are any other optimizations I can do to my web server.

Other info that might be useful: I'm getting around 2000 daily unique visitors but I also get a high level of spam attacks on my website. I have set up Cloudflare as proxy in front of my server and ever since the spam level decreased significantly. I also use MaxCDN for static files such as scripts or images so these are usually not requested from the web server.

1
  • Static files should be no issue anyway since they should be processed by nginx. Whatever load I had, I have never seen PHP segfaulting because of it (usually PHP-FPM will be really busy and upstream requests coming from nginx will time out). Try disabling modules one after another and see when the segfaults stop. You might want to reinstall all your extensions as well (don't know how that goes on CentOS) and check that their versions match PHP. Did you update anything before the segfaults started?
    – Lukas
    May 25, 2014 at 3:02

1 Answer 1

3

More than likely some module php is including is causing your segfaults. You will need to do some debugging to figure out what could be causing it. Here's a good place to start:

Turn on core files: http://aplawrence.com/Linux/limit_core_files.html

Read the core dump with GDB: https://stackoverflow.com/questions/5115613/core-dump-file-analysis

That'll give you a general idea of what might be causing it. You may need to upgrade/downgrade any PHP modules you have installed.

3
  • Thanks, I enabled core dumps and killed a child with sig 11 to test it and the output said: Program terminated with signal 11, Segmentation fault. #0 0x00007f5c7070488d in accept () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install php-fpm-5.4.3-1.el6.remi.x86_64 However I could not find a debuginfo package for php-fpm, will this info be enough?
    – Daniel
    Dec 5, 2012 at 15:04
  • Thats not really enough to know whats going on. You'll need a full backtrace with the debug info to really see where the problem is.
    – Kyle
    Dec 6, 2012 at 18:48
  • So any idea where I can find the debug packages for php-fpm. I have really searched and couldn't find anything. Thanks.
    – Daniel
    Dec 6, 2012 at 22:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .