Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have 3 servers.

First server (CPU - model name: 06/17, 2.66GHz, 4 cores, 8GB RAM) have nginx as load balancer with next config

upstream  lb_mydomain  {
    server   mydomain.ru:81 weight=2;
    server   66.0.0.18 weight=6;
}
server {
    listen 80;
    server_name ~(?!mydomain.ru)(.*);

    client_max_body_size 20m;

    location / {
        proxy_pass  http://lb_mydomain;
        proxy_redirect     off;
        proxy_set_header  Connection  close;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass_header Set-Cookie;
        proxy_pass_header P3P;
        proxy_pass_header Content-Type;
        proxy_pass_header Content-Disposition;
        proxy_pass_header Content-Length;
     }
  }

And configs from nginx.conf:

user www-data;
worker_processes  5;
# worker_priority -1;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
   worker_connections  5024;
   # multi_accept on;
}

http {
   include       /etc/nginx/mime.types;

   access_log  /var/log/nginx/access.log;

   sendfile        on;
   default_type  application/octet-stream;
   #tcp_nopush     on;

   keepalive_timeout  65;
   tcp_nodelay        on;

   gzip  on;
   gzip_disable "MSIE [1-6]\.(?!.*SV1)";

    # PHP-FPM (backend)
    upstream php-fpm {
       server 127.0.0.1:9000;
    }


   include /etc/nginx/conf.d/*.conf;
   include /etc/nginx/sites-enabled/*;
}

And config php-fpm:

listen = 127.0.0.1:9000
;listen.backlog = -1
;listen.allowed_clients = 127.0.0.1
;listen.owner = www-data
;listen.group = www-data
;listen.mode = 0666
user = www-data
group = www-data
pm = dynamic
pm.max_children = 80
;pm.start_servers = 20
pm.min_spare_servers = 5
pm.max_spare_servers = 35
;pm.max_requests = 500
pm.status_path = /status
ping.path = /ping
;ping.response = pong
request_terminate_timeout = 30s
request_slowlog_timeout = 10s
slowlog = /var/log/php-fpm.log.slow
;rlimit_files = 1024
;rlimit_core = 0
;chroot = 
chdir = /var/www
;catch_workers_output = yes
;env[HOSTNAME] = $HOSTNAME
;env[PATH] = /usr/local/bin:/usr/bin:/bin
;env[TMP] = /tmp
;env[TMPDIR] = /tmp
;env[TEMP] = /tmp
;php_admin_value[sendmail_path] = /usr/sbin/sendmail -t -i -f www@my.domain.com
;php_flag[display_errors] = off
;php_admin_value[error_log] = /var/log/fpm-php.www.log
;php_admin_flag[log_errors] = on
;php_admin_value[memory_limit] = 32M

In top I see 20 php-fpm processes which use from 1% - 15% CPU. So it's have high load averadge:

top - 15:36:22 up 34 days, 20:54,  1 user,  load average: 5.98, 7.75, 8.78
Tasks: 218 total,   1 running, 217 sleeping,   0 stopped,   0 zombie
Cpu(s): 34.1%us,  3.2%sy,  0.0%ni, 37.0%id, 24.8%wa,  0.0%hi,  0.9%si,  0.0%st
Mem:   8183228k total,  7538584k used,   644644k free,   351136k buffers
Swap:  9936892k total,    14636k used,  9922256k free,   990540k cached

Second server(CPU - model name: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz, 8 cores, 8GB RAM). Nginx configs from nginx.conf:

user www-data;
worker_processes  5;
# worker_priority -1;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
   worker_connections  5024;
   # multi_accept on;
}

http {
   include       /etc/nginx/mime.types;

   access_log  /var/log/nginx/access.log;

   sendfile        on;
   default_type  application/octet-stream;
   #tcp_nopush     on;

   keepalive_timeout  65;
   tcp_nodelay        on;

   gzip  on;
   gzip_disable "MSIE [1-6]\.(?!.*SV1)";

    # PHP-FPM (backend)
    upstream php-fpm {
       server 127.0.0.1:9000;
    }


   include /etc/nginx/conf.d/*.conf;
   include /etc/nginx/sites-enabled/*;
}

And config of php-fpm:

listen = 127.0.0.1:9000
;listen.backlog = -1
;listen.allowed_clients = 127.0.0.1
;listen.owner = www-data
;listen.group = www-data
;listen.mode = 0666
user = www-data
group = www-data
pm = dynamic
pm.max_children = 50
;pm.start_servers = 20
pm.min_spare_servers = 5
pm.max_spare_servers = 35
;pm.max_requests = 500
;pm.status_path = /status
;ping.path = /ping
;ping.response = pong
;request_terminate_timeout = 0
;request_slowlog_timeout = 0
;slowlog = /var/log/php-fpm.log.slow
;rlimit_files = 1024
;rlimit_core = 0
;chroot = 
chdir = /var/www
;catch_workers_output = yes
;env[HOSTNAME] = $HOSTNAME
;env[PATH] = /usr/local/bin:/usr/bin:/bin
;env[TMP] = /tmp
;env[TMPDIR] = /tmp
;env[TEMP] = /tmp
;php_admin_value[sendmail_path] = /usr/sbin/sendmail -t -i -f www@my.domain.com
;php_flag[display_errors] = off
;php_admin_value[error_log] = /var/log/fpm-php.www.log
;php_admin_flag[log_errors] = on
;php_admin_value[memory_limit] = 32M

In top I see 50 php-fpm processes which use from 10% - 25% CPU. So it's have high load averadge:

top - 15:53:05 up 33 days,  1:15,  1 user,  load average: 41.35, 40.28, 39.61
Tasks: 239 total,  40 running, 199 sleeping,   0 stopped,   0 zombie
Cpu(s): 96.5%us,  3.1%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.4%si,  0.0%st
Mem:   8185560k total,  7804224k used,   381336k free,   161648k buffers
Swap: 19802108k total,       16k used, 19802092k free,  5068112k cached

Third server is server with database postgresql. Also i try ab -n 50 -c 5 http://www.mydomain.ru/ And I get next info:

Complete requests:      50
Failed requests:        48
(Connect: 0, Receive: 0, Length: 48, Exceptions: 0)
Write errors:           0
Total transferred:      9271367 bytes
HTML transferred:       9247767 bytes
Requests per second:    1.02 [#/sec] (mean)
Time per request:       4882.427 [ms] (mean)
Time per request:       976.486 [ms] (mean, across all concurrent requests)
Transfer rate:          185.44 [Kbytes/sec] received

Please advise how can I make lower level of load average?

share|improve this question
    
If PHP is consuming excessive resources, reduce the number of PHP processes running by lowering pm.max_children (try 20 to start with, and adjust from there). (You will also need to reduce pm.max_spare_servers so it remains less than pm.max_children) –  cyberx86 Feb 7 '12 at 18:15
    
Yes, reducing of pm.max_children = 20 helps for me, load average becomes 15. –  Dmitro Feb 8 '12 at 10:45
    
First you need to find your limiting factor. Then start optimization. I dumped nginx for Cherokee. Cherokee is serving 800 request/s whereas nginx is serving only 150 request/s for me. Please note i have used nginx for 2 years and i still love it! –  BigSack Aug 23 '13 at 6:31

2 Answers 2

You can see that your first server is doing a lot of I/O wait (top shows 24.8%wa).

That means that somebody is doing I/O - maybe your PHP scripts? Long-running I/O work can slow down any webserver that uses PHP-FPM and lead to high load numbers.

share|improve this answer

Installing PHP-APC (the Alternative PHP Cache module) will speed up large PHP apps 5-10x.

If you have a large percentage of read-only users (ie, users who are just browsing the site, not interacting with it) then you might want to try using Varnish to cache static copies of the pages.

share|improve this answer
    
I already installed php-apc, yes this speed up app but not in 5-10x. I will reed more info about Varnish, thanks –  Dmitro Feb 8 '12 at 13:32
    
or use proxy_cache module from nginx - wiki.nginx.org/HttpProxyModule#proxy_cache –  mt_ Feb 10 '13 at 20:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.