Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a simple node.js application that makes a single call connection call to my ReplicaSet. Things were actually working fine earlier today and nothing was changed in the application, now my connection is failing (silently) and I'm seeing the following in my mongod logs.

Thu Apr 18 17:16:56 [conn1782026] end connection 10.x.x.x:50720 (40 connections now open)
Thu Apr 18 17:16:57 [initandlisten] connection accepted from 10.x.x.x:50721 #1782027 (41 connections now open)
Thu Apr 18 17:16:57 [initandlisten] connection accepted from 10.x.x.x:50722 #1782028 (42 connections now open)
Thu Apr 18 17:16:57 [initandlisten] connection accepted from 10.x.x.x:50723 #1782029 (43 connections now open)
Thu Apr 18 17:16:57 [initandlisten] connection accepted from 10.x.x.x:50724 #1782030 (44 connections now open)
Thu Apr 18 17:16:57 [initandlisten] connection accepted from 10.x.x.x:50725 #1782031 (45 connections now open)
Thu Apr 18 17:16:57 [initandlisten] connection accepted from 10.x.x.x:50736 #1782032 (46 connections now open)
[conn1782028] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.x.x.x:50722]

I haven't been able to track down what causes this or how to fix it.

Here's my connection code:

var _ = require("underscore");
var db_options = {
    db:{
            w:1,
            native_parser: false
        },
    server:{
        auto_reconnect:true,
        socketOptions: {}
    },
    replSet:{
        rs_name: "my_replset_name",
        readPreference: "secondary"
    },
    mongos: {}
};
db.client.connect("mongodb://server1:27017,server2:27017,server3:27017/my_database", db_options, function(err, _db){
    if(err){
        console.log(err);
    }else{
        _db.collection("my_collection", function(err, collection){
            collection.find({q:1}).toArray(function(err,item){
                if(err){
                    console.log(err);
                }else{
                    _.each(item, function(r){
                        if(!_.isEmpty(r)){
                            // do something with this item!
                            console.log(r._id);
                        }
                    });
                    console.log("Finished.");
                }
            });
        });
    }
});

I'm running my ReplicaSet on EC2 and everything can successfully connect to each otherhave a simple node.js application that makes a single call connection call to my ReplicaSet - like I said, this worked and then stopped working without notice.

Does anyone have any insights into this? What causes it? How to prevent it? How to fix it?

Thanks in advance.

share|improve this question

1 Answer 1

It's tough to diagnose with the information provided, but a possible clue is the number of connections you have opened (the numbers start at 1 - conn1 - and increment for each new connection). conn1782026 indicates that you have opened nearly 1.8 million connections since this mongod was started. With concurrent connections of ~40, I would hope that the mongod has been up a very long time to see that number of connections opened and closed.

If not, then that might be a hint in terms of what is going on. Are you creating new connections every time you run a query or similar? If so, you may need to tweak your code to pool connections instead (see the various options here).

Now, in theory, you should be able to do this anyway - in and of itself having a lot of connections is not going to cause problems, but the amount of churn here may be having unintended consequences. Try restarting the mongod, test, if still there restart the instance and test again. If either one clears up the issue, then it's not likely to be network related (a reset of the system will reset the network interfaces, so there is still a small chance), even though it is manifesting as a socket exception.

By the way, if you want to get an idea of how many connections you are creating, take a look here:

> db.serverStatus().connections
{ "current" : 1, "available" : 2047, "totalCreated" : NumberLong(1) }

totalCreated is a counter for the lifetime of the particular mongod or mongos process. If you divide it by the uptime you can also get an average created per second etc.

If the resets do not have any effect and you are still seeing exceptions, then it is time to look at your network. Check the output of netstat -s (multiple samples) and look for incrementing retransmits and fast retransmits, run ifconfig to look for errors on your interfaces, ping, traceroute etc. - all the usual things you would do to try to determine if there is a network issue.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.