Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a situation in which the following is taking place:

  • We are on linode with 8-core, 8gb of ram , 2.6 ghz - using nginx + php-fpm - we are getting extremely high graphs of cpu usage (which we don't want to be such a bad VPS neighbor)...

  • We have around less than 100 users on the site at a time - so this situation is also incredibly embarassing - that our cpu usage is very high.

  • We are using a very unknown, possibly cpu intensive php-wise, questionably horrible Framework instead of well-known, well-documented, well-crafted other frameworks like wordpress or drupal in which there is LOTS of documentation about caching (as well as plugins that handle caching) php on a nginx + php_fpm platform.

  • Thus, we have about 6 open php-fpm processes that when RUNNING, consume individually LARGE (30+, and often near 99%) amounts of cpu - and I haven't really the slightest idea how to stop them from using so much cpu. I can't tell which php scripts are causing these spikes because they are happening all the time... usually only 1 or 2 are running - but when all 6 run we maximize all 8 cpus.

  • My pool.d/www.conf file has the following settings:

    pm = dynamic
    pm.max_children = 10
    pm.start_servers = 4
    pm.min_spare_servers = 2
    pm.max_spare_servers = 6
    
  • We did this ^ setup because, in the way that I am interpretting it, our memory is actually amazing (htop showing 472/7000+mb used, no swapping etc) and we could handle many more processes and break down the line waiting to get processed - BUT unfortunately, since each process is too intense on our cpu when running - we end up driving our CPU through the roof - so we can't handle enough processes.

  • The question - what on earth can we do to reduce the process php-fpm cpu usage so that we can increase the settings in that pool conf file for php-fpm - and also yes, the /var/log/php5-fpm.log is yelling at us to increase our children and adjust/increase our min/max/start servers. But doing so makes our load average crazy as previously stated. How can we do so without necessarily using a cache or what are our options?

  • My idea? I've read things about using cpulimit to ensure no process takes more than an allotted amount of cpu - but will that slow things down to be unusable? Or in doing so we could increase our ability to run more than a few processes - I also thought running two pools - one for our forward facing website (what customers experience) and another for a backend (which is affecting our forward facing site when time-consuming reports are being ran).

  • I have been spending a few days researching, googling, etc on this topic - and it is difficult because everyone's situation is so unique to their system - the trouble is being on such a specific unheard-of, possibly poorly written - framework - is making it hard to find a solution. We can't just scrap this framework just yet either - I have to find a solution of some sort.


UPDATE: I have implemented memcache to store php sessions - because the framework heavily relies on user sessions and the nature of our system is that employees often use several tabs at a time - each checking back to the sessions to confirm abilities / user data / etc... so I am hoping to see some increase in performance from this - welcome to comment on that if you'd like - I'll see how it goes tomorrow when we get through our higher volume peak times.

share|improve this question
    
Nginx is not very good for a cpu intensive web application - but our high cpu is bad - really bad - and we are working to fix it. There's no great max clients setup to have because it SHOULD be able to support a decent number of clients - but the high cpu usage per process skews that capability. We switched to apache only because it does a LITTLE better with high cpu usage - but ultimately this problem is more indicative of a web app problem and it may take a while to address but there's no time like the present to start fixing it. –  amurrell Apr 18 at 18:16
    
When you go to the doctor, and he tells you to take certain meds - because he knows you wont listen to the "stop drinking soda and eating fastfood" statements - this is exactly why there was no great answer for me - because the truth is, no setup or quick fix really applied - only the sad truth that we have to dramatically alter our web app itself. –  amurrell Apr 18 at 18:17
    
Luckily if you have this problem with a popular framework, you may have the option to leverage caching and plentiful documentation about it - but we are on some obscure random thing that we cannot change except the framework itself. Yay! –  amurrell Apr 18 at 18:19
1  
So from what I understand of it, opcache is storing your PHP code as binary and can be consumed by php-fpm much quicker (caching) but you should also utilize object caching.. an example is storing an entire page output as an "object" in something like memcached. This would actually cache the page output (or other things you want like the php sessions etc)... Next, you could also use varnish which is a reverse proxy - but basically it's a middle man between the request and your server so that your server isn't being hit with requests directly - and it works from memory to serve the cached urls. –  amurrell Dec 3 at 20:19
1  
Varnish is awesome for this - storing a cached copy of what it got from the server in memory - so varnish takes much of the load. My current employer is on nginx and we use varnish and memcached. Fortunately the framework we are on now has it's own caching mechanism to determine the page cache (outputted page data). At my last job I had to write it into the framework myself - it was not fun, but worked.Apache- I would not switch back, unless you don't have time to fix nginx.. I hated going back but it was the only solution to not totally killing our project while I wrote the caching mechanism. –  amurrell Dec 3 at 20:22

2 Answers 2

A couple of things to consider (apologies in advance if you have already considered these): First of all, make sure to optimize your nginx config and invoke php-fpm only when absolutely necessary. The last thing you want to do is let php handle things like static HTML pages (which it will happily do).

Secondly, since you're using php-fpm, I suggest to be more aggressive with how long php-fpm's children are allowed to live. You need to find the sweet spot between shortly lived threads/children and stability. The php-fpm defaults are way to generous for any production system, IMHO. The longer a worker is allowed to furnish requests, more unstable it will get. There's also a higher risk of memory leaks, and if this framework you refer to has bugs like infinite loops, which may be causing you grief with CPU load, this shouldn't hurt.

I'd reduce the number for pm.max_requests for your production pools. I think the default is 200. I'd start from 50 and see where that takes you.

Failing/complementary to that, you could also try these global options (AFAIK they are all disabled by default):

emergency_restart_threshold 3
emergency_restart_interval 1m
process_control_timeout 5s

What does this mean? If 3 PHP-FPM child processes exit with SIGSEGV or SIGBUS (i.e. crash) within 1 minute then PHP-FPM is supposed to restart automatically. The child processes wait 5s for a reaction on signals from master.

Here's a nice overview of all the config options I mentioned here, as well as others: http://myjeeva.com/php-fpm-configuration-101.html

Hope these tips help you! Remember to tweak and observe, unfortunately there doesn't seem to be a rule of thumb for all this, as you yourself observed, there are too many variables that affect PHP's behaviour and stability.

Finally, the CPU limiting facility you inquired about is documented here, but I'd only resort to it if you exhaust every other option. If you do choose this path, I'd definitely watch out for possible interactions between PHP-FPM tweaks and your limits.conf configuration. At that point etckeeper may be a lifesaver! :)

Good luck!

Rouben

share|improve this answer
    
I am going to try limiting the max_requests tomorrow - It seems like each process we allow wants to eat up the cpu so this might be a good idea. We are utilizing the emergency restart thresholds, but my numbers were slightly higher - I'll try yours and see how things go. Thanks for your thoroughness - very appreciated. Did want to know your thoughts on caching? I am wondering if utilizing php caching means that the framework would have to be tailored to handle it. I am pretty new to that concept. –  amurrell Feb 17 at 10:20

You are running opcode caching, right?

It used to be APC that was the go-to here, but it's been a buggy piece of for quite a while, and has been superseded by Zend Opcache, which is now part of PHP since 5.5, and has a backport in PECL for 5.3 and 5.4.

share|improve this answer
    
I'm interested in this Zend OpCache - We are on 5.3 - If you could expand on this answer I'd really appreciate it! –  amurrell Feb 17 at 10:11
    
We had xcache - but now I've opted for Zend Opcache and I have installed it and confirmed it is up and running in phpinfo(). I'll let you know if our performance is better as a result crosses fingers –  amurrell Feb 17 at 10:48
    
I had to disable zend opcache for now - everything was working except any dynamic-php-generated css or js file. I tried to blacklist those files from opcache - but either the blacklist didn't work or I can't tell why opcache is causing nginx to get 502 bad gateway errors for ONLY those files. Which I need them for the template and they are part of the bad framework. In the nginx logs I received tons of readv failed - 104 peer connection error. –  amurrell Feb 17 at 13:11

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.