Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am using mongoDB for my application hosted on 3 amazon instances. I have a got a Primary Node with highest priority (100) and 2 Secondaries with priority 99 and 98. A few days ago I started getting MongoCursorExceptions while accessing primary node. I thought it had something to do with my primary server running out of resources and slow to respond, so I decided to switch one of the secondary nodes to primary.

I changed priorities from 100, 99, and 98 to 1, 100, 98 but this did not switch my secondary server to primary. On checking the status of my replica set, I found out that the 3 nodes were not syncing properly as optime was more than 17 hours behind. The real concern is that optime of both secondary nodes is constant while it keeps on increasing for the primary node. I also verified that my secondary nodes did not have upto date data by using slaveOk query. Following is the status that I got on my primary node:

PRIMARY> rs.status()
{
    "set" : "rs_a",
    "date" : ISODate("2012-02-28T09:34:50Z"),
    "myState" : 1,
    "members" : [
            {
                    "_id" : 0,
                    "name" : "server1.com:27018",
                    "health" : 1,
                    "state" : 1,
                    "stateStr" : "PRIMARY",
                    "optime" : {
                            "t" : 1330421648000,
                            "i" : 2
                    },
                    "optimeDate" : ISODate("2012-02-28T09:34:08Z"),
                    "self" : true
            },
            {
                    "_id" : 1,
                    "name" : "server2.com:27018",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 3510,
                    "optime" : {
                            "t" : 1330358351000,
                            "i" : 1
                    },
                    "optimeDate" : ISODate("2012-02-27T15:59:11Z"),
                    "lastHeartbeat" : ISODate("2012-02-28T09:34:50Z"),
                    "pingMs" : 0
            },
            {
                    "_id" : 2,
                    "name" : "server3.com:27018",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 3510,
                    "optime" : {
                            "t" : 1330358351000,
                            "i" : 1
                    },
                    "optimeDate" : ISODate("2012-02-27T15:59:11Z"),
                    "lastHeartbeat" : ISODate("2012-02-28T09:34:50Z"),
                    "pingMs" : 0
            }
    ],
    "ok" : 1
}

I couldn't find much in logs. On few occasions the logs on secondary server reported that there was problem connecting to primary server. But the interesting thing is that all writebacks are done properly on secondary nodes. I would like all servers to be up to date at all times so I can read from any node as this is why I am using mongoDB in the first place.

share|improve this question
    
Did you manually promote the secondary to primary first as documented here? : mongodb.org/display/DOCS/Forcing+a+Member+to+be+Primary –  Remon van Vliet Feb 28 '12 at 12:17
    
I tried it and now I have no Primary node :( –  umair Feb 28 '12 at 12:24
add comment

1 Answer 1

up vote 1 down vote accepted

I managed to resolve this issue by restarting mongod on secondary nodes. I think the nodes had gone into a stale state.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.