Nginx shows 15k active connections, few are reading/writing/waiting - what are they doing?

Question

Currently using NGINX as a load balancer for an upstream to a sharded mongo cluster.

.conf reads:

stream {
    upstream prodapp_mongo {
        least_conn;

        server xxx.xxx.xxx.xxx:27017;
        server xxx.xxx.xxx.xxx:27017;
        server xxx.xxx.xxx.xxx:27017;
    }   

    server {
        listen 27017;
        proxy_connect_timeout 25s;
        proxy_timeout 60s;
        proxy_pass prodapp_mongo;
        error_log /var/log/nginx/mongo.error.log;
    }
}

stub_status output:

Active connections: 15751
server accepts handled requests
 15070690 15070506 129824
Reading: 0 Writing: 2 Waiting: 0

The number of active connections seems to be pegged around 15k for about 14 hours and I am concerned as to why they are not in a state of reading/writing/waiting.

Our data-dog reports show a steady increase of connections over this time period and I'm trying to further dig into "why" there are so many active connections, and what are these doing?

Answer 1

"why" there are so many active connections

nginx decreases Active connections count on client's connection close, so it's certain that you have a lot of idle connections there. It's established, but it's currently not sending/receiving any data.

Check your DB connections from mongodb by connecting to the db and run

db.serverStatus().connections

and what are these doing?

As you don't describe your application/database client, it's probably coming from application side of things, such as connection pools from DB drivers, application's persistent connections, or even non-close()d connections.