My VPS keeps hanging up every 5 days. I can see that the nginx worker processes are eating up my memory over time. I think this is due the massive amount of CLOSE_WAIT's open. When I type LSOF, I get tons and tons. A sample can be found below.
nginx 1934 www-data 170u IPv4 476876 0t0 TCP something.com:www->141.101.98.116:10086 (CLOSE_WAIT)
nginx 1934 www-data 171u IPv4 1162471 0t0 TCP something.com:www->cf-199-27-128-89.cloudflare.com:23774 (CLOSE_WAIT)
nginx 1934 www-data 172u IPv4 567606 0t0 TCP something.com:www->108.162.221.20:37267 (CLOSE_WAIT)
nginx 1934 www-data 173u IPv4 2323097 0t0 TCP something.com:www->108.162.245.14:23335 (CLOSE_WAIT)
nginx 1934 www-data 174u IPv4 1344669 0t0 TCP something.com:www->108.162.225.231:60467 (CLOSE_WAIT)
nginx 1934 www-data 175u IPv4 743417 0t0 TCP something.com:www->197.234.243.4:59188 (CLOSE_WAIT)
nginx 1934 www-data 176u IPv4 1798404 0t0 TCP something.com:www->108.162.222.215:57796 (CLOSE_WAIT)
nginx 1934 www-data 177u IPv4 1180153 0t0 TCP something.com:www->108.162.238.45:17968 (CLOSE_WAIT)
nginx 1934 www-data 178u IPv4 497718 0t0 TCP something.com:www->141.101.104.47:11815 (CLOSE_WAIT)
nginx 1934 www-data 179u IPv4 567607 0t0 TCP something.com:www->108.162.221.20:11240 (CLOSE_WAIT)
nginx 1934 www-data 180u IPv4 432698 0t0 TCP something.com:www->108.162.222.219:52570 (CLOSE_WAIT)
nginx 1934 www-data 181u IPv4 3330260 0t0 TCP something.com:www->cf-173-245-50-213.cloudflare.com:15185 (CLOSE_WAIT)
nginx 1934 www-data 182u IPv4 404397 0t0 TCP something.com:www->108.162.229.221:17852 (CLOSE_WAIT)
nginx 1934 www-data 183u IPv4 567609 0t0 TCP something.com:www->108.162.221.20:31861 (CLOSE_WAIT)
nginx 1934 www-data 184u IPv4 677137 0t0 TCP something.com:www->108.162.225.154:22771 (CLOSE_WAIT)
nginx 1934 www-data 185u IPv4 320850 0t0 TCP something.com:www->141.101.98.228:57247 (CLOSE_WAIT)
nginx 1934 www-data 186u IPv4 566451 0t0 TCP something.com:www->108.162.221.20:50783 (CLOSE_WAIT)
nginx 1934 www-data 187u IPv4 1756167 0t0 TCP something.com:www->108.162.237.32:18998 (CLOSE_WAIT)
nginx 1934 www-data 188u IPv4 751612 0t0 TCP something.com:www->108.162.219.73:26342 (CLOSE_WAIT)
nginx 1934 www-data 189u IPv4 566453 0t0 TCP something.com:www->108.162.221.20:32939 (CLOSE_WAIT)
nginx 1934 www-data 190u IPv4 1973567 0t0 TCP something.com:www->108.162.250.47:17339 (CLOSE_WAIT)
nginx 1934 www-data 191u IPv4 1798441 0t0 TCP something.com:www->103.22.201.109:26477 (CLOSE_WAIT)
nginx 1934 www-data 192u IPv4 829030 0t0 TCP something.com:www->cf-173-245-52-65.cloudflare.com:34624 (CLOSE_WAIT)
nginx 1934 www-data 193u IPv4 168016 0t0 TCP something.com:www->108.162.219.87:53679 (CLOSE_WAIT)
nginx 1934 www-data 194u IPv4 751613 0t0 TCP something.com:www->108.162.219.73:48552 (CLOSE_WAIT)
nginx 1934 www-data 195u IPv4 744531 0t0 TCP something.com:www->197.234.243.4:55992 (CLOSE_WAIT)
nginx 1934 www-data 196u IPv4 858578 0t0 TCP something.com:www->141.101.104.8:47353 (CLOSE_WAIT)
nginx 1934 www-data 197u IPv4 744534 0t0 TCP something.com:www->197.234.243.4:23059 (CLOSE_WAIT)
nginx 1934 www-data 198u IPv4 751615 0t0 TCP something.com:www->108.162.219.73:43779 (CLOSE_WAIT)
nginx 1934 www-data 199u IPv4 751616 0t0 TCP something.com:www->cf-173-245-56-246.cloudflare.com:30916 (CLOSE_WAIT)
nginx 1934 www-data 200u IPv4 1189674 0t0 TCP something.com:www->173.245.54.17:27328 (CLOSE_WAIT)
nginx 1934 www-data 201u IPv4 3184758 0t0 TCP something.com:www->108.162.210.38:26233 (CLOSE_WAIT)
nginx 1934 www-data 202u IPv4 1344668 0t0 TCP something.com:www->108.162.225.231:39050 (CLOSE_WAIT)
nginx 1934 www-data 203u IPv4 1181119 0t0 TCP something.com:www->108.162.238.45:28510 (CLOSE_WAIT)
nginx 1934 www-data 204u IPv4 285484 0t0 TCP something.com:www->cf-173-245-48-166.cloudflare.com:64434 (CLOSE_WAIT)
nginx 1934 www-data 205u IPv4 1867742 0t0 TCP something.com:www->108.162.215.134:64332 (CLOSE_WAIT)
nginx 1934 www-data 206u IPv4 1946314 0t0 TCP something.com:www->108.162.210.179:39163 (CLOSE_WAIT)
nginx 1934 www-data 207u IPv4 1632128 0t0 TCP something.com:www->108.162.222.18:14036 (CLOSE_WAIT)
nginx 1934 www-data 208u IPv4 1457453 0t0 TCP something.com:www->198.41.226.251:57972 (CLOSE_WAIT)
nginx 1934 www-data 209u IPv4 2123937 0t0 TCP something.com:www->141.101.98.114:11156 (CLOSE_WAIT)
Is there someone that can help me out with debugging and fixing this problem ? I think something is misconfigured and nginx keeps on waiting on certain connections. Is there a way to find what url's are called for these connections ?
EDIT: Nginx config
user www-data;
worker_processes 6;
worker_rlimit_nofile 300000;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {
worker_connections 14000;
# multi_accept on;
}
http {
include /etc/nginx/mime.types;
server_names_hash_max_size 812000;
geoip_country /etc/nginx/geoip/GeoIP.dat; # the country IP database
geoip_city /etc/nginx/geoip/GeoLiteCity.dat; # the city IP database
log_format withhost '$host - $remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
# access_log /var/log/nginx/access.log withhost;
# set_real_ip_from 141.101.64.0/18;
# set_real_ip_from 108.162.192.0/18;
# set_real_ip_from 190.93.240.0/20;
# set_real_ip_from 2400:cb00::/32;
# set_real_ip_from 2606:4700::/32;
set_real_ip_from 69.164.223.55;
set_real_ip_from 212.123.14.6;
set_real_ip_from 192.168.255.17;
real_ip_header X-Forwarded-For;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 30;
client_body_timeout 20;
reset_timedout_connection on;
send_timeout 5;
server_tokens off;
gzip on;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";
# include /etc/nginx/conf.d/*.conf;
# include /nfs/vhosts/*;
include /etc/nginx/sites-enabled/*;
server {
server_name www.x.com;
access_log off;
error_log /srv/www/www.x.com/logs/error.log;
root /srv/www/www.x.com/public_html/src/public;
gzip on;
gzip_http_version 1.0;
gzip_comp_level 2;
gzip_proxied any;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_types text/plain text/html text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;
# Some version of IE 6 don't handle compression well on some mime-types, so just disable for them
gzip_disable "MSIE [1-6].(?!.*SV1)";
# Set a vary header so downstream proxies don't send cached gzipped content to IE6
gzip_vary on;
## /Compression
location ^~ /ads-management {
auth_basic "Restricted";
auth_basic_user_file /srv/www/www.x.com/public_html/.htpasswd;
if (!-e $request_filename) {
rewrite ^.*$ /index.php last;
}
index index.html index.htm index.php;
}
location ~ ^/thumbnail/(.*)$ {
resolver 8.8.8.8;
proxy_pass http://$arg_code;
}
location / {
#Don't use slash at end
rewrite ^/(.*)/$ /$1 permanent;
if ($args ~* "/?q=(.*)") {
rewrite ^ /search/q/$arg_q/? permanent;
}
if (!-e $request_filename) {
rewrite ^.*$ /index.php last;
}
index index.html index.htm index.php;
}
location ~ .(php|phtml)$ {
include fastcgi_params;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /srv/www/www.x.com/public_html/src/public$fastcgi_script_name;
}
}
EDIT 2 TCP_MEM values:
cat /proc/sys/net/ipv4/tcp_mem
47706 63610 95412
Thanks!
2015/04/15 06:46:44 [alert] 26304#0: 14000 worker_connections are not enoughin my logsFINflagged segment to tell the server that it is actively closing the connection while the server didn't answer with aFIN+ACKsegment (if it did it would have switched to theLAST_ACKstate). You can usestrace -fp <master_process_pid>to check the syscalls related to what's happening. Maybe the server is hanging on asend()/write()call or something.