2

I am reaching out for some help here, I have spent quite a few hours on this site searching the posts and also on google trying to see what the issue may be and trying to solve my problem without much success. I have the following setup. We are hosting with Linode.com.

(1) - One load balancer doing a round robin with least connections to the following two nodes (2) - 2GB nodes each running Nginx + php-fpm + APC and connected to yet another instance of mySQL running on a dedicated machine over a local IP address. Each of the nodes have (4) dedicated cores.

I am running a worpdess site on each node, basically each node runs an identical copy of the same wordpress PHP code.

Everything is running as it should but the load on the CPU is very high, we are averaging the following load on the nodes:

Node 1: Load average: 4.11 3.84 3.95 Node 2: Load average: 4.20 3.94 4.95

Why are the cores spiking and holding the load. The whole websites gets about 2.1 million requests a day balanced over the (2) nodes.

Is this an nginx configuration or php-fpm issue, or it's just a matter of needing to add one or two more nodes?

Thank you in advance for your help. Dave

My nginx.conf

[ddavtian@mobilefood-1 nginx]$ more nginx.conf
#######################################################################
#
# This is the main Nginx configuration file.  
#
# More information about the configuration options is available on 
#   * the English wiki - http://wiki.nginx.org/Main
#   * the Russian documentation - http://sysoev.ru/nginx/
#
#######################################################################

#----------------------------------------------------------------------
# Main Module - directives that cover basic functionality
#
#   http://wiki.nginx.org/NginxHttpMainModule
#
#----------------------------------------------------------------------

user              nginx;
worker_processes  4;
worker_rlimit_nofile 30000;

error_log  /var/log/nginx/error.log;
#error_log  /var/log/nginx/error.log  notice;
#error_log  /var/log/nginx/error.log  info;

pid        /var/run/nginx.pid;


#----------------------------------------------------------------------
# Events Module 
#
#   http://wiki.nginx.org/NginxHttpEventsModule
#
#----------------------------------------------------------------------

events {
    worker_connections  1024;
}


#----------------------------------------------------------------------
# HTTP Core Module
#
#   http://wiki.nginx.org/NginxHttpCoreModule 
#
#----------------------------------------------------------------------

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  15;

    #gzip  on;

    # Load config files from the /etc/nginx/conf.d directory
    # The default server is in conf.d/default.conf
    include /etc/nginx/conf.d/*.conf;
}

My fastcgi_params

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

fastcgi_connect_timeout 60;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
fastcgi_buffer_size 128k;
fastcgi_buffers 4 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_intercept_errors on;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS    200;

My default.conf

[ddavtian@mobilefood-1 conf.d]$ more default.conf

#
# The default server
#
server {
    listen       80;
    server_name  mobilefoodblog.com;

    #charset koi8-r;

    access_log off;
    #access_log  logs/host.access.log  main;

    location /nginx_status {
     stub_status on;
     access_log   off;
     allow 67.180.226.49;
     deny all;
    }

    location / {
        root   /var/www/html/blog;
        index  index.php index.html index.htm;
        try_files $uri $uri/ /index.php?q=$uri&$args;
    }

    error_page  404              /404.html;
    location = /404.html {
        root   /usr/share/nginx/html;
    }

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    #location ~ \.php$ {
    #    root           html;
    #    fastcgi_pass   127.0.0.1:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    location ~ \.php$ {
       root           /var/www/html/blog;
       fastcgi_pass   127.0.0.1:9000;
       fastcgi_index  index.php;
       fastcgi_param  SCRIPT_FILENAME  /var/www/html/blog$fastcgi_script_name;
       include        fastcgi_params;
       set_real_ip_from  192.168.255.0/24;
       real_ip_header X-Forwarded-For;
    }


    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}

My php-fpm.conf

[ddavtian@mobilefood-1 conf.d]$ more default.conf

#
# The default server
#
server {
listen       80;
server_name  mobilefoodblog.com;

#charset koi8-r;

access_log off;
#access_log  logs/host.access.log  main;

location /nginx_status {
 stub_status on;
 access_log   off;
 allow 67.180.226.49;
 deny all;
}

location / {
    root   /var/www/html/blog;
    index  index.php index.html index.htm;
    try_files $uri $uri/ /index.php?q=$uri&$args;
}

error_page  404              /404.html;
location = /404.html {
    root   /usr/share/nginx/html;
}

# redirect server error pages to the static page /50x.html
#
error_page   500 502 503 504  /50x.html;
location = /50x.html {
    root   /usr/share/nginx/html;
}

# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
#    proxy_pass   http://127.0.0.1;
#}

# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
#    root           html;
#    fastcgi_pass   127.0.0.1:9000;
#    fastcgi_index  index.php;
#    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
#    include        fastcgi_params;
#}

location ~ \.php$ {
   root           /var/www/html/blog;
   fastcgi_pass   127.0.0.1:9000;
   fastcgi_index  index.php;
   fastcgi_param  SCRIPT_FILENAME  /var/www/html/blog$fastcgi_script_name;
   include        fastcgi_params;
   set_real_ip_from  192.168.255.0/24;
   real_ip_header X-Forwarded-For;
}


# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
#    deny  all;
#}

}

My php-fpm.conf

[ddavtian@mobilefood-1 php-fpm.d]$ more www.conf

; Start a new pool named 'www'.
[www]

; The address on which to accept FastCGI requests.
; Valid syntaxes are:
;   'ip.add.re.ss:port'    - to listen on a TCP socket to a specific address on
;                            a specific port;
;   'port'                 - to listen on a TCP socket to all addresses on a
;                            specific port;
;   '/path/to/unix/socket' - to listen on a unix socket.
; Note: This value is mandatory.
listen = 127.0.0.1:9000

; Set listen(2) backlog. A value of '-1' means unlimited.
; Default Value: -1
;listen.backlog = -1

; List of ipv4 addresses of FastCGI clients which are allowed to connect.
; Equivalent to the FCGI_WEB_SERVER_ADDRS environment variable in the original
; PHP FCGI (5.2.2+). Makes sense only with a tcp listening socket. Each address
; must be separated by a comma. If this value is left blank, connections will be
; accepted from any ip address.
; Default Value: any
listen.allowed_clients = 127.0.0.1

; Set permissions for unix socket, if one is used. In Linux, read/write
; permissions must be set in order to allow connections from a web server. Many
; BSD-derived systems allow connections regardless of permissions. 
; Default Values: user and group are set as the running user
;                 mode is set to 0666
;listen.owner = nobody
;listen.group = nobody
;listen.mode = 0666

; Unix user/group of processes
; Note: The user is mandatory. If the group is not set, the default user's group
;       will be used.
; RPM: apache Choosed to be able to access some dir as httpd
user = apache
; RPM: Keep a group allowed to write in log dir.
group = apache

; Choose how the process manager will control the number of child processes.
; Possible Values:
;   static  - a fixed number (pm.max_children) of child processes;
;   dynamic - the number of child processes are set dynamically based on the
;             following directives:
;             pm.max_children      - the maximum number of children that can
;                                    be alive at the same time.
;             pm.start_servers     - the number of children created on startup.
;             pm.min_spare_servers - the minimum number of children in 'idle'
;                                    state (waiting to process). If the number
;                                    of 'idle' processes is less than this
;                                    number then some children will be created.
;             pm.max_spare_servers - the maximum number of children in 'idle'
;                                    state (waiting to process). If the number
;                                    of 'idle' processes is greater than this
;                                    number then some children will be killed.
; Note: This value is mandatory.
pm = dynamic

; The number of child processes to be created when pm is set to 'static' and the
; maximum number of child processes to be created when pm is set to 'dynamic'.
; This value sets the limit on the number of simultaneous requests that will be
; served. Equivalent to the ApacheMaxClients directive with mpm_prefork.
; Equivalent to the PHP_FCGI_CHILDREN environment variable in the original PHP
; CGI.
; Note: Used when pm is set to either 'static' or 'dynamic'
; Note: This value is mandatory.
pm.max_children = 60

; The number of child processes created on startup.
; Note: Used only when pm is set to 'dynamic'
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 10

; The desired minimum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.min_spare_servers = 5

; The desired maximum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.max_spare_servers = 35

; The number of requests each child process should execute before respawning.
; This can be useful to work around memory leaks in 3rd party libraries. For
; endless request processing specify '0'. Equivalent to PHP_FCGI_MAX_REQUESTS.
; Default Value: 0
;pm.max_requests = 500

; The URI to view the FPM status page. If this value is not set, no URI will be
; recognized as a status page. By default, the status page shows the following
; information:
;   accepted conn    - the number of request accepted by the pool;
;   pool             - the name of the pool;
;   process manager  - static or dynamic;
;   idle processes   - the number of idle processes;
;   active processes - the number of active processes;
;   total processes  - the number of idle + active processes.
; The values of 'idle processes', 'active processes' and 'total processes' are
; updated each second. The value of 'accepted conn' is updated in real time.
; Example output:
;   accepted conn:   12073
;   pool:             www
;   process manager:  static
;   idle processes:   35
;   active processes: 65
;   total processes:  100
; By default the status page output is formatted as text/plain. Passing either
; 'html' or 'json' as a query string will return the corresponding output
; syntax. Example:
;   http://www.foo.bar/status
;   http://www.foo.bar/status?json
;   http://www.foo.bar/status?html
; Note: The value must start with a leading slash (/). The value can be
;       anything, but it may not be a good idea to use the .php extension or it
;       may conflict with a real PHP file.
; Default Value: not set 
;pm.status_path = /status

; The ping URI to call the monitoring page of FPM. If this value is not set, no
; URI will be recognized as a ping page. This could be used to test from outside
; that FPM is alive and responding, or to
; - create a graph of FPM availability (rrd or such);
; - remove a server from a group if it is not responding (load balancing);
; - trigger alerts for the operating team (24/7).
; Note: The value must start with a leading slash (/). The value can be
;       anything, but it may not be a good idea to use the .php extension or it
;       may conflict with a real PHP file.
; Default Value: not set
;ping.path = /ping

; This directive may be used to customize the response of a ping request. The
; response is formatted as text/plain with a 200 response code.
; Default Value: pong
;ping.response = pong

; The timeout for serving a single request after which the worker process will
; be killed. This option should be used when the 'max_execution_time' ini option
; does not stop script execution for some reason. A value of '0' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
;request_terminate_timeout = 0

; The timeout for serving a single request after which a PHP backtrace will be
; dumped to the 'slowlog' file. A value of '0s' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
;request_slowlog_timeout = 0

; The log file for slow requests
; Default Value: not set
; Note: slowlog is mandatory if request_slowlog_timeout is set
slowlog = /var/log/php-fpm/www-slow.log

; Set open file descriptor rlimit.
; Default Value: system defined value
;rlimit_files = 1024

; Set max core size rlimit.
; Possible Values: 'unlimited' or an integer greater or equal to 0
; Default Value: system defined value
;rlimit_core = 0

; Chroot to this directory at the start. This value must be defined as an
; absolute path. When this value is not set, chroot is not used.
; Note: chrooting is a great security feature and should be used whenever 
;       possible. However, all PHP paths will be relative to the chroot
;       (error_log, sessions.save_path, ...).
; Default Value: not set
;chroot = 

; Chdir to this directory at the start. This value must be an absolute path.
; Default Value: current directory or / when chroot
;chdir = /var/www

; Redirect worker stdout and stderr into main error log. If not set, stdout and
; stderr will be redirected to /dev/null according to FastCGI specs.
; Default Value: no
;catch_workers_output = yes

; Pass environment variables like LD_LIBRARY_PATH. All $VARIABLEs are taken from
; the current environment.
; Default Value: clean env
;env[HOSTNAME] = $HOSTNAME
;env[PATH] = /usr/local/bin:/usr/bin:/bin
;env[TMP] = /tmp
;env[TMPDIR] = /tmp
;env[TEMP] = /tmp

; Additional php.ini defines, specific to this pool of workers. These settings
; overwrite the values previously defined in the php.ini. The directives are the
; same as the PHP SAPI:
;   php_value/php_flag             - you can set classic ini defines which can
;                                    be overwritten from PHP call 'ini_set'. 
;   php_admin_value/php_admin_flag - these directives won't be overwritten by
;                                     PHP call 'ini_set'
; For php_*flag, valid values are on, off, 1, 0, true, false, yes or no.

; Defining 'extension' will load the corresponding shared extension from
; extension_dir. Defining 'disable_functions' or 'disable_classes' will not
; overwrite previously defined php.ini values, but will append the new value
; instead.

; Default Value: nothing is defined by default except the values in php.ini and
;                specified at startup with the -d argument
;php_admin_value[sendmail_path] = /usr/sbin/sendmail -t -i -f [email protected]
;php_flag[display_errors] = off
php_admin_value[error_log] = /var/log/php-fpm/www-error.log
php_admin_flag[log_errors] = on
;php_admin_value[memory_limit] = 32M

UPDATE 1: Adding the output of htop which shows that php-fpm is consuming most of the CPU and the box is not using much memory nor swapping

UDPATE 2: Here's the latest shapshot of htop, getting more traffic now and more CPU usage. I have now even installed memcached but so far not seeing an improvement.

  1  [|||||||||||||||||||||||||||||||||||||||85.5%]     Tasks: 62, 9 thr; 18 running
  2  [|||||||||||||||||||||||||||||||||      72.7%]     Load average: 6.18 5.46 4.23 
  3  [||||||||||||||||||||||||||             57.0%]     Uptime: 2 days, 03:40:49
  4  [|||||||||||||||||||||||                52.0%]
  Mem[|||||||||||||||||||               463/1997MB]
  Swp[                                    0/2047MB]

  PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
28892 apache    20   0  511M 33864 21484 R  9.0  1.7  0:52.62 php-fpm: pool www
28972 apache    20   0  511M 33808 21188 R  9.0  1.7  0:30.08 php-fpm: pool www
28918 apache    20   0  510M 32512 20544 R  8.0  1.6  0:52.00 php-fpm: pool www
28899 apache    20   0  510M 29320 17488 S  8.0  1.4  0:52.53 php-fpm: pool www
28900 apache    20   0  510M 29280 17492 S  8.0  1.4  0:52.60 php-fpm: pool www
28919 apache    20   0  510M 29512 17568 R  8.0  1.4  0:51.53 php-fpm: pool www
28914 apache    20   0  510M 29300 17468 R  8.0  1.4  0:52.12 php-fpm: pool www
28971 apache    20   0  510M 32648 20868 R  8.0  1.6  0:31.66 php-fpm: pool www
28904 apache    20   0  512M 38544 25284 R  8.0  1.9  0:51.70 php-fpm: pool www
28916 apache    20   0  511M 33636 21092 R  8.0  1.6  0:51.89 php-fpm: pool www
28905 apache    20   0  510M 29868 18108 S  8.0  1.5  0:52.06 php-fpm: pool www
28898 apache    20   0  512M 38608 25156 S  7.0  1.9  0:52.57 php-fpm: pool www
28891 apache    20   0  512M 34824 21420 R  7.0  1.7  0:52.73 php-fpm: pool www
28912 apache    20   0  510M 32724 20904 R  7.0  1.6  0:52.14 php-fpm: pool www
28894 apache    20   0  512M 34280 21388 S  7.0  1.7  0:53.14 php-fpm: pool www
11744 apache    20   0  508M 28104 16648 S  0.0  1.4  0:08.72 php-fpm: pool www
6
  • Wordpress is slow. Your problem likely boils down to PHP (watch (h)top and see if it is the php-fpm processes that are using all your CPU). If that is the case, a good starting point would be to implement caching. You could try a plugin (e.g. W3TC), fastcgi caching (at nginx), or an extra layer such as Varnish (which might offer the best performance).
    – cyberx86
    Jan 16, 2012 at 17:26
  • I have updated my original post and added htop which shows php-fpm using the CPU the most. As for caching, APC is already installed. I am afraid if I install W3TC caching for wordpress it is also going to cache our php based banner which would impact the impressions, etc. Will look into fastcgi caching from Nginx and also Vamish, thank you for your advice.
    – ddavtian
    Jan 16, 2012 at 21:17
  • That htop output doesn't really show high CPU - most php-fpm processes are using 2% or less, and each core is only being used 8-12% which is certainly not overloaded. I also notice your load average has dropped to something better (on a multi-core CPU numbers above 1 are acceptable). See if you can get a sample of htop when the CPU usage is high - perhaps get some CPU usage data from sar, etc.
    – cyberx86
    Jan 16, 2012 at 21:24
  • I have updated the htop and now it shows more CPU usage since I am getting more traffic now. I have also installed memcached but the CPU usage has not improved thus far.
    – ddavtian
    Jan 16, 2012 at 22:39
  • 1
    It simply seems like your application is CPU bound, which isn't necessarily a bad thing by itself. The excessive CPU usage seems to be as a result of too many simultaneous requests being served - reduce your pm.* values and see if it makes a difference. It may also be worth looking a bit into I/O problems - your load average seems a good bit higher than your CPU usage, which might suggest some blocking. You mention the use of APC - ensure you have your APC setup to use memory (and not files), increase the shm_size, and check your hitrate. Also, make your ads load asynchronously if you can.
    – cyberx86
    Jan 17, 2012 at 12:44

0

You must log in to answer this question.

Browse other questions tagged .