0

I'm on a server with the following config:

32 GB RAM 12 core processor

and it's running single wordpress on nginx, varnish, php5-fpm, fastcgi. Everything was running fine, but suddenly after yesterday night the whole site is extremely slow and unresponsive. Upon inspection, I found that mysql was using a lot of CPU. After a lot of research online, I disabled XML-RPC and mysql CPU load came down drastically.

But the problem still persists, my CPU was almost 100%. I see that php5-fpm processes are taking up all the CPU.

Here's the result of top

top - 10:30:33 up  7:10,  1 user,  load average: 31.35, 31.87, 31.85
Tasks: 238 total,  27 running, 179 sleeping,  32 stopped,   0 zombie
%Cpu(s): 84.9 us, 11.7 sy,  0.0 ni,  3.1 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem:  32948544 total, 11695168 used, 21253376 free,   408404 buffers
KiB Swap:        0 total,        0 used,        0 free.  3151748 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
12179 mysql     20   0 6233684 998836   7632 S 109.2  3.0 371:56.58 mysqld
18533 nginx     20   0 1007708 769740  26008 R  57.5  2.3   3:33.24 php5-fpm
18487 nginx     20   0  600672 361312  26212 R  51.7  1.1   4:13.51 php5-fpm
18657 nginx     20   0  280024  42640  25436 R  51.7  0.1   0:46.94 php5-fpm
18459 nginx     20   0  996592 753804  25764 R  40.2  2.3   4:53.79 php5-fpm
18469 nginx     20   0 1010296 768676  25704 R  40.2  2.3   4:22.98 php5-fpm
18545 nginx     20   0  277216  40060  25740 R  40.2  0.1   2:53.74 php5-fpm
18547 nginx     20   0  684952 441796  25616 R  40.2  1.3   2:58.57 php5-fpm
18629 nginx     20   0  280452  42752  25444 R  40.2  0.1   1:22.99 php5-fpm
18460 nginx     20   0  282796  45920  25828 R  34.5  0.1   5:22.28 php5-fpm
18461 nginx     20   0  276952  40200  25636 R  34.5  0.1   4:13.64 php5-fpm
18462 nginx     20   0  278752  41504  25672 R  34.5  0.1   4:16.16 php5-fpm
18540 nginx     20   0  277744  41096  25740 R  34.5  0.1   3:02.92 php5-fpm
18624 nginx     20   0  278648  41060  25588 R  34.5  0.1   1:27.54 php5-fpm
18452 nginx     20   0  278752  41696  25732 S  28.7  0.1   4:25.82 php5-fpm
18458 nginx     20   0  282488  45512  25684 S  28.7  0.1   4:44.26 php5-fpm
18472 nginx     20   0  282652  45976  25948 R  28.7  0.1   4:34.61 php5-fpm
18486 nginx     20   0 1007732 769472  25800 R  28.7  2.3   4:03.61 php5-fpm
18491 nginx     20   0  282488  45584  25764 R  28.7  0.1   4:06.75 php5-fpm
18495 nginx     20   0  280744  43216  25744 R  28.7  0.1   3:37.82 php5-fpm
18496 nginx     20   0  282488  44924  25908 R  28.7  0.1   3:38.28 php5-fpm
18562 nginx     20   0  282488  45412  25584 R  28.7  0.1   2:53.19 php5-fpm
18575 nginx     20   0  282760  46068  26192 R  28.7  0.1   2:34.81 php5-fpm
18455 nginx     20   0  278904  41532  25716 S  23.0  0.1   4:22.13 php5-fpm
18468 nginx     20   0  278776  42120  25732 R  23.0  0.1   4:09.41 php5-fpm
18490 nginx     20   0  280840  44148  26172 R  23.0  0.1   3:39.08 php5-fpm
18492 nginx     20   0  282232  45260  25652 R  23.0  0.1   4:07.21 php5-fpm
18514 nginx     20   0  282608  45716  25732 R  23.0  0.1   3:44.15 php5-fpm
18576 nginx     20   0  279416  41964  25496 R  23.0  0.1   2:09.75 php5-fpm
18656 nginx     20   0  279560  42428  25508 S  23.0  0.1   0:49.84 php5-fpm
18465 nginx     20   0  284492  47860  26252 R  17.2  0.1   5:09.60 php5-fpm
10165 nobody    20   0 11.233g 2.437g  83232 S  11.5  7.8  23:56.78 varnishd
18686 root      20   0   24948   1656   1080 R  11.5  0.0   0:00.04 top
    8 root      20   0       0      0      0 S   5.7  0.0   2:48.56 rcuos/0
18122 nginx     20   0   36700   7428   1040 S   5.7  0.0   1:04.16 nginx
18129 nginx     20   0   33836   4568   1040 S   5.7  0.0   0:25.52 nginx

As you can see Load Average is a whopping 31.35, 31.87, 31.85 What can cause such high CPU usage. I've tried tweaking everything from mysql to nginx to varnish, but nothing seems to be working. I've tried playing around with pm value in my php5 conf file too, but here I'm, tearing my hair apart.

Please help.

The website gets a lot of traffic, say 2000 to 3000 concurrent users on an average. :(

UPDATE:

Including the nginx conf files: /etc/nginx/nginx.conf

user  nginx;
worker_processes  12;
worker_rlimit_nofile 20000;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  3000;
    use epoll;
    multi_accept on;
}


http {
    server_names_hash_bucket_size  64;
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    limit_req_zone $binary_remote_addr zone=one:100m rate=3r/m;

    sendfile        on;
    tcp_nopush     on;
    tcp_nodelay on;

    keepalive_timeout  65;
    client_header_timeout 180;
    client_body_timeout 180;
    fastcgi_read_timeout 180;
    client_body_buffer_size 10M;
    client_max_body_size 32m;
    gzip  on;
    gzip_disable "msie6";
        gzip_vary on;
        gzip_proxied any;
        gzip_comp_level 6;
        gzip_buffers 16 8k;
        gzip_http_version 1.1;
        gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

    include /etc/nginx/conf.d/*.conf;
    port_in_redirect off;
}

/etc/nginx/conf.d/default.conf

server {
    listen 8080;
    server_name mysite.com;
    return 301 $scheme://www.mysite.com$request_uri;
}
server {
    ## Your website name goes here.
    server_name www.mysite.com;
    ## Your only path reference.
    root /var/www/;
    listen 8080;
    ## This should be in your http block and if it is, it's not needed here.
    index index.html index.htm index.php;

    include conf.d/drop;

        location / {
                # This is cool because no php is touched for static content
                        try_files $uri $uri/ /index.php?q=$uri&$args;
        }
        location ~ \.php$ {
             try_files $uri =404;
             # proxy buffers - no 502 errors!
             proxy_buffer_size               128k;
             proxy_buffers                   4 256k;
             proxy_busy_buffers_size         256k;

            fastcgi_buffers 256 16k;
            fastcgi_buffer_size 128k;
            fastcgi_max_temp_file_size 0;
            fastcgi_intercept_errors on;
            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_pass unix:/dev/shm/php-fpm-www.sock;
            #fastcgi_pass 127.0.0.1:9000;

             location ~* wp\-login\.php {
                limit_req   zone=one  burst=1 nodelay;
                include fastcgi_params;
                fastcgi_pass unix:/dev/shm/php-fpm-www.sock;
              }

        }
        location /xmlrpc.php {
          deny all;
        }

}
  • 1
    Can you post your nginx configuration for us? Specifically your fpm settings. – Nathan C May 18 '16 at 15:12
  • @NathanC I've added the nginx config. Kindly help. – LittleLebowski May 18 '16 at 15:42
  • Hmm, a similar question is here: stackoverflow.com/questions/13732436/… ...try playing with the php-fpm configuration. Those child processes are running for a long time so pm.max_requests could be lowered so they recycle more. – Nathan C May 18 '16 at 18:41
  • 1
    running single wordpress, but in that WordPress can many slow (read: bad) queries? Can you install Query Monitor plugin and check how many bad queries. – prosti May 18 '16 at 19:10
  • You are actually not having the cached content from the perspective of WordPress, right? Not using caching plugins. – prosti May 18 '16 at 19:19
0

From another point of view I noticed something. If you don't use Memcache you don't need Varnish. Varnish is "pasato", since not planning to support spdy for a long time as Varnish author said once.

Also I see that your caching layer is actually on application layer. The W3TC plugin you are using.

Still you may add to your configuration. I haven't saw that.

location ~*  \.(jpg|jpeg|png|gif|ico|css|js)$ {
        expires 365d;
}

Also improve what you allow on your web server.

## Only allow these request methods ##
     if ($request_method !~ ^(GET|HEAD|POST)$ ) {
         return 444;
     }
## Do not accept DELETE, SEARCH and other methods ##

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.