2

We use nginx for load balancing to a pair of websocket servers, and have hit an issue.

It will not gracefully exit or shut down once it has actually connected traffic to a web socket server. E.g. service nginx stop, or nginx -s quit or nginx -s reload causes one or more worker processes to report "worker process is shutting down" forever.

The flow is:

  1. Start nginx with config below.
  2. Pass traffic to the nginx endpoint (even using web browser to hit the 443 port and get 404 error is enough)
  3. Use service control or send quit command
  4. nginx is now hung.

We run nginx on centos v6

Details of our compile options and high level config:

    [root@nginx1 nginx]# nginx -V
    nginx version: nginx/1.7.9
    built by gcc 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC)
    TLS SNI support enabled
    configure arguments: --user=nginx --group=nginx --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-http_gzip_static_module 
--with-http_ssl_module --add-module=/opt/nginx_upstream_check_module-master/

Our config is below. How do we go about chasing this down? Right now we are forced to do hard kill/restart of nginx to update the config.

worker_processes  2;

error_log  logs/error.log;

events {
    worker_connections  20000;
}

worker_rlimit_nofile    40000;

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;

    upstream websocketserver {
        server 192.168.2.16:3842 max_fails=1 fail_timeout=60s;
        server 192.168.2.19:3842 max_fails=1 fail_timeout=60s;
    }

    server {
        listen 192.168.2.28:80;

    location / {

        proxy_pass http://websocketserver;

        proxy_next_upstream    error timeout invalid_header http_500;
        proxy_connect_timeout  2;
        proxy_read_timeout      86400;

        # WebSocket support (nginx 1.4)
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        }

        location / {
            deny all;
            return 404;
        }       

    }
}
2
+50

Possibly netstat and tcpdump are useful for debugging as well as lsof - is the worker process still connected and exchanging data? I notice your proxy_read_timeout is a day rather than the default 60s, and wonder if that's significant. It sounds like an nginx bug, and just possibly this post about ZLIB compression and keepalive is related: http://forum.nginx.org/read.php?2,170139,209671

  • "proxy_read_timeout is a day rather than the default 60s" You win the prize. That is the issue. Thank you! – samsmith Feb 22 '15 at 21:30
0

What i would do is attach to the nginx process using strace , then try and shut it down and check your strace to see what file descriptor it is hung on. With this information use lsof to track down which file descriptor it is waiting for and go from there. My guess is it may be one of your upstream servers causing this.

  • Thank you! I agree with your suspicion. But how do I modify the nginx.conf so that it times out and moves on, instead of hanging for 5-10 or more minutes? – samsmith Feb 10 '15 at 15:57
0

If it is a websocket server like the browser websocket protocol it needs to send the websocket shutdown to the connected browsers and close the socket. The application at 192.168.2.*:3842 would be capable of doing so. So you would need to send a signal to that application to tell it to send shutdown to its connected websockets.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.