1

We currently have some servers hosting a e-commerce platform in Digital Ocean, we tried to optimize how much we could, but was not enough, site is going down when too much traffic, like 800~1000 users. We currently think its the API that is going down. The error is:

2017/11/23 14:20:51 [error] 26784#26784: *126599 connect() to unix:/var/run/php/php7.0-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: {ip...}, server: www.....com.br, request: "GET /....", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "www......com.br"

We currently have:

Front-end server:

4 cores, 8gb ram - running nginx 1.10.3 and php fpm 7.0

API server:

8 cores, 16gb ram - running nginx 1.10.3 and php fpm 5.5.9

MYSQL server

MASTER + 8 SLAVES (We don't think thats the problem), We had some problem before with connection numbers tcp/ip ports (I dont know exactly), but we could't solve and created slaves for load balance. Servers are running linux ubuntu 14.04

If more information is needed, please ask. We are struggling with this.

Nginx conf:

worker_processes auto;
worker_rlimit_nofile 65536;
worker_connections 2048;
use epoll;
multi_accept on;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 15;
types_hash_max_size 2048;
server_names_hash_max_size 4112;
access_log off;
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 2;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;]
fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;
fastcgi_connect_timeout 360;
fastcgi_send_timeout 360;
fastcgi_read_timeout 360;
fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
client_body_buffer_size    128k;
client_max_body_size       10m;
client_header_buffer_size    1k;
large_client_header_buffers  4 4k;
output_buffers   1 32k;
postpone_output  1460;
client_header_timeout  3m;
client_body_timeout    3m;
send_timeout           3m;

PHP FPM conf:

pm = on demand;
pm.max_children = 200;
pm.process_idle_timeout = 10s;
pm.max_requests = 500;

/etc/sysctl.conf configs:

net.ipv4.ip_local_port_range = 2000 65535;
net.ipv4.tcp_rfc1337 = 1;
net.ipv4.tcp_fin_timeout = 15;
net.ipv4.tcp_keepalive_time = 300;
net.ipv4.tcp_keepalive_probes = 5;
net.ipv4.tcp_keepalive_intvl = 15;
net.core.rmem_default = 31457280;
net.core.rmem_max = 12582912;
net.core.wmem_default = 31457280;
net.core.wmem_max = 12582912;
net.core.somaxconn = 65535;
net.core.netdev_max_backlog = 65535;
net.core.optmem_max = 25165824;
net.ipv4.tcp_mem = 65535 131072 262144;
net.ipv4.udp_mem = 65535 131072 262144;
net.ipv4.tcp_rmem = 8192 87380 16777216;
net.ipv4.udp_rmem_min = 16384;
net.ipv4.tcp_wmem = 8192 65535 16777216;
net.ipv4.udp_wmem_min = 16384;
net.ipv4.tcp_max_tw_buckets = 1440000;
net.ipv4.tcp_tw_recycle = 1;
net.ipv4.tcp_tw_reuse = 1;
  • Could you post my.cnf/ini from MASTER and 1 SLAVE? – Wilson Hauck Nov 28 '17 at 19:33
  • MASTER (4 CORES, 8GB RAM): gist.github.com/arthurlauck/1462e7adc67a268593cf1bda68b235a1 SLAVE (2 cores 4gb): gist.github.com/arthurlauck/6f5cc19f0740d648418d027b12a5ca14 I used show variables to get them. What would be ini? There is no php installed. – Gabriel de Souza Dec 1 '17 at 14:03
  • Thanks for the GLOBAL VARIABLES lists from MASTER and 1 SLAVE. I see that both have expire_logs_days at 10 and one of them has log_warnings at 2 (for additional details). Please ensure the server you think is causing the problem (that has been named at the evert) has days at 10 and log_warnings at 2 and error_log active. Share the error_logs thru gist and someone will be able to assist. When I compare the MASTER and SLAVE global lists provided 3 hours ago, using fc /L /N master.txt slave.txt >diff.txt I see some differences I did not expect. I expected max concat to be the same size. – Wilson Hauck Dec 1 '17 at 17:09
  • We don't think its the mysql server causing the issue, but the API with php/nginx is. The error in the log was "2017/11/23 14:20:51 [error] 26784#26784: *126599 connect() to unix:/var/run/php/php7.0-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: {ip...}, server: www.....com.br, request: "GET /....", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "www......com.br"" – Gabriel de Souza Dec 1 '17 at 20:09
  • Please share the upstream error_log thru gist and someone may be able to assist. Browser search for - nginx error 26784 - will find potential guidance. – Wilson Hauck Dec 3 '17 at 11:12
0

Resource temporarily unavailable clearly indicates that you have the bottleneck on front-end servers (on the servers that are logging this error), not even the API ones. This error usually indicates the lack of memory of some sort or some limit hit (the TCP connections for instance), so you need to investigate further - lack of information doesn't leave much space for assumptions.

Also note that 200 workers on 4 or even 8 cores is generally an overkill (unless you have a realkly slow and cumbersome application, whic is not the case in 95% of setups, because it's a problem by itself), and php 5.5.9 is unsupported and insecure. This has nothing to dop with the problem you're facing, but these issues are minor (and in case of PHP - major) problems also.

  • Hey, thanks for the anwser, I will look into it. So, we can set 200 workers in all machines that use nginx? Front-end and back-end? How can i check what is causing the bottleneck? Do you recommend some software or website to stress test the machines? – Gabriel de Souza Dec 1 '17 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.