1

I am using mongoDB for my application hosted on 3 amazon instances. I have a got a Primary Node with highest priority (100) and 2 Secondaries with priority 99 and 98. A few days ago I started getting MongoCursorExceptions while accessing primary node. I thought it had something to do with my primary server running out of resources and slow to respond, so I decided to switch one of the secondary nodes to primary.

I changed priorities from 100, 99, and 98 to 1, 100, 98 but this did not switch my secondary server to primary. On checking the status of my replica set, I found out that the 3 nodes were not syncing properly as optime was more than 17 hours behind. The real concern is that optime of both secondary nodes is constant while it keeps on increasing for the primary node. I also verified that my secondary nodes did not have upto date data by using slaveOk query. Following is the status that I got on my primary node:

PRIMARY> rs.status()
{
    "set" : "rs_a",
    "date" : ISODate("2012-02-28T09:34:50Z"),
    "myState" : 1,
    "members" : [
            {
                    "_id" : 0,
                    "name" : "server1.com:27018",
                    "health" : 1,
                    "state" : 1,
                    "stateStr" : "PRIMARY",
                    "optime" : {
                            "t" : 1330421648000,
                            "i" : 2
                    },
                    "optimeDate" : ISODate("2012-02-28T09:34:08Z"),
                    "self" : true
            },
            {
                    "_id" : 1,
                    "name" : "server2.com:27018",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 3510,
                    "optime" : {
                            "t" : 1330358351000,
                            "i" : 1
                    },
                    "optimeDate" : ISODate("2012-02-27T15:59:11Z"),
                    "lastHeartbeat" : ISODate("2012-02-28T09:34:50Z"),
                    "pingMs" : 0
            },
            {
                    "_id" : 2,
                    "name" : "server3.com:27018",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 3510,
                    "optime" : {
                            "t" : 1330358351000,
                            "i" : 1
                    },
                    "optimeDate" : ISODate("2012-02-27T15:59:11Z"),
                    "lastHeartbeat" : ISODate("2012-02-28T09:34:50Z"),
                    "pingMs" : 0
            }
    ],
    "ok" : 1
}

I couldn't find much in logs. On few occasions the logs on secondary server reported that there was problem connecting to primary server. But the interesting thing is that all writebacks are done properly on secondary nodes. I would like all servers to be up to date at all times so I can read from any node as this is why I am using mongoDB in the first place.

2

1 Answer 1

2

I managed to resolve this issue by restarting mongod on secondary nodes. I think the nodes had gone into a stale state.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .