1

My website playsongs.pk got 499, 502 and php-fpm.sock failed (11: Resource temporarily unavailable) errors when I received 300-400 active users on site.

Sample/example error logs:

2017-02-27 14:56:23 Error   162.158.78.50   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   502 GET / HTTP/1.1          166     nginx access
2017-02-27 14:56:23 Error   162.158.78.170  502 GET / HTTP/1.1          166     nginx access
2017-02-27 14:56:23 Error   162.158.79.63   502 GET / HTTP/1.1          166     nginx access
2017-02-27 14:56:23 Error   162.158.78.50   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   499 GET / HTTP/1.1          0   nginx access

and After few time I got

2017-02-27 14:56:23 Error   172.68.65.4     24026#0: *2784527 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24023#0: *2784529 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.100       24028#0: *2784539 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24021#0: *2784541 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24021#0: *2784545 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24021#0: *2784547 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24028#0: *2784549 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   162.158.78.50       24028#0: *2784551 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24028#0: *2784552 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error

My Nginx configuration /etc/nginx/nginx.conf:

user  nginx;
worker_processes auto; #some last versions calculate it automatically

worker_rlimit_nofile 99999;

error_log /var/log/nginx/error.log crit;

include /etc/nginx/modules.conf.d/*.conf;

events {

    worker_connections 66536;
    use epoll;
    multi_accept on;
}

http {
    include   mime.types;
    default_type  application/octet-stream;

    open_file_cache max=2048 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 5;
    open_file_cache_errors off;


    access_log off;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    keepalive_timeout  2;
    keepalive_requests 100000;
    reset_timedout_connection on;
    client_body_timeout 30;
    send_timeout 15;

client_header_timeout 12;
proxy_connect_timeout  60;
proxy_send_timeout  60;
proxy_read_timeout  60;


fastcgi_buffers 8 128k;
fastcgi_buffer_size 256k;

# reduce the data that needs to be sent over network
    gzip on;
    gzip_min_length 1000;
    gzip_proxied expired no-cache no-store private auth;
    gzip_types application/x-javascript text/css application/javascript text/javascript text/plain text/xml application/json application/v$
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";


    server_tokens off;

    include /etc/nginx/conf.d/*.conf;
}

# override global parameters e.g. worker_rlimit_nofile
include /etc/nginx/*global_params;

My php-fpm pool setting are following:

;[php-fpm-pool-settings]
pm = dynamic
pm.max_children = 5000
pm.start_servers = 50
pm.min_spare_servers = 20
pm.max_spare_servers = 70
pm.max_requests = 2000

Please note

Server have 64GB of RAM with 1 GBit dedicate INTERNET PORT and average load on CPU also does not exceed from 1.somthing % or 2.somthing %

Server OS: Centos 7.2

Control Panel: Plesk Onyx

nginx Version: nginx/1.11.4

Output of #cat /proc/$(cat /run/nginx.pid)/limits is following. May it will be helpful to solve the issue.

Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             515124               515124               processes
Max open files            1024                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       515124               515124               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                   
Max realtime priority     0                    0                   
Max realtime timeout      unlimited            unlimited            us

Output of tail /var/log/nginx/error.log

2017/03/01 15:16:15 [notice] 31867#0: getrlimit(RLIMIT_NOFILE): 1024:4096
2017/03/01 15:16:15 [notice] 31868#0: start worker processes
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31869
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31870
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31871
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31872
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31873
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31874
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31875
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31876
  • I suspect it's PHP playing up. Read your PHP error logs, or if you don't understand them edit your question to include them. 499 is an Nginx specific error code stackoverflow.com/questions/12973304/nginx-499-error-codes – Tim Mar 1 '17 at 8:18
  • @Tim there is nothing in php_error logs. I have update my question. Added output of #cat /proc/$(cat /run/nginx.pid)/limits may it helps – Hassaan Mar 1 '17 at 9:55
  • You include nginx access.log in your question. That shows only a single line summary of the request. The nginx error.log shows the errors from PHP if there is nothing in PHP error log. Please include entries from nginx error.log to the question. – Tero Kilkanen Mar 1 '17 at 10:34
  • @TeroKilkanen I have update my question – Hassaan Mar 1 '17 at 10:37
  • OK, then it seems that your Plesk control panel is hiding the PHP error messages from you. Anyway, the problem is in your application, it cannot keep up with the requests. You need to find out your PHP error log, or how to output more stuff into it. There you will have some clues on what is not working in your application. Then you need to fix it. – Tero Kilkanen Mar 1 '17 at 10:42
2

Try this:

sysctl net.core.somaxconn=65535

php/fpm/pool.d/www.conf:

listen.backlog 65535;


Or try using local IP connection instead of unix socket:

php/fpm/pool.d/www.conf:

listen 127.0.0.1:9000;

nginx/sites-enabled/sitename:

fastcgi_pass 127.0.0.1:9000;

|improve this answer|||||
  • Thank you for response. Can you tell me were I can see current value of listen.backlog – Hassaan Mar 1 '17 at 10:55
  • @Hassaan In fpm pool config, if not mentioned - php defaults are used – Selivanov Pavel Mar 1 '17 at 10:58
  • Its not mentioned in fpm pool config, how can I see php's default number for listen.backlog? – Hassaan Mar 1 '17 at 11:01
  • @Hassaan Somewhere in PHP documentation I suppose. It may be reasonable, try tuning sysctl and see if error is gone – Selivanov Pavel Mar 1 '17 at 11:06
  • net.core.somaxconn = 65535 is exits in /etc/sysctl.conf. Should I increase this one? – Hassaan Mar 1 '17 at 11:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.