Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a replica set that I am trying to upgrade the primary to one with more memory and upgraded disk space. So I raided a couple disks together on the new primary, rsync'd the data from a secondary and added it to the replica set. After checking out the rs.status(), I noticed that all the secondaries are at about 12 hours behind the primary. So when I try to force the new server to the primary spot it won't work, because it is not up to date.

This seems like a big issue, because in case the primary fails, we are at least 12 hours and some almost 48 hours behind.

The oplogs all overlap and the oplogsize is fairly large. The only thing that I can figure is I am performing a lot of writes/reads on the primary, which could keep the server in lock, not allowing for proper catch up.

Is there a way to possibly force a secondary to catch up to the primary?

There are currently 5 Servers with last 2 are to replace 2 of the other nodes. The node with _id as 6, is to be the one to replace the primary. The node that is the furthest from the primary optime is a little over 48 hours behind.

{
"set" : "gryffindor",
"date" : ISODate("2011-05-12T19:34:57Z"),
"myState" : 2,
"members" : [
    {
        "_id" : 1,
        "name" : "10******:27018",
        "health" : 1,
        "state" : 2,
        "stateStr" : "SECONDARY",
        "uptime" : 20231,
        "optime" : {
            "t" : 1305057514000,
            "i" : 31
        },
        "optimeDate" : ISODate("2011-05-10T19:58:34Z"),
        "lastHeartbeat" : ISODate("2011-05-12T19:34:56Z")
    },
    {
        "_id" : 2,
        "name" : "10******:27018",
        "health" : 1,
        "state" : 2,
        "stateStr" : "SECONDARY",
        "uptime" : 20231,
        "optime" : {
            "t" : 1305056009000,
            "i" : 400
        },
        "optimeDate" : ISODate("2011-05-10T19:33:29Z"),
        "lastHeartbeat" : ISODate("2011-05-12T19:34:56Z")
    },
    {
        "_id" : 3,
        "name" : "10******:27018",
        "health" : 1,
        "state" : 1,
        "stateStr" : "PRIMARY",
        "uptime" : 20229,
        "optime" : {
            "t" : 1305228858000,
            "i" : 422
        },
        "optimeDate" : ISODate("2011-05-12T19:34:18Z"),
        "lastHeartbeat" : ISODate("2011-05-12T19:34:56Z")
    },
    {
        "_id" : 5,
        "name" : "10*******:27018",
        "health" : 1,
        "state" : 2,
        "stateStr" : "SECONDARY",
        "uptime" : 20231,
        "optime" : {
            "t" : 1305058009000,
            "i" : 226
        },
        "optimeDate" : ISODate("2011-05-10T20:06:49Z"),
        "lastHeartbeat" : ISODate("2011-05-12T19:34:56Z")
    },
    {
        "_id" : 6,
        "name" : "10*******:27018",
        "health" : 1,
        "state" : 2,
        "stateStr" : "SECONDARY",
        "optime" : {
            "t" : 1305050495000,
            "i" : 384
        },
        "optimeDate" : ISODate("2011-05-10T18:01:35Z"),
        "self" : true
    }
],
"ok" : 1
}
share|improve this question
add comment

2 Answers

To answer the original question (which wouldn't have solved the OP's problem), I believe that the best way to force a secondary to "catch up" would be to remove it from the set and re-add it, but chances are (such as in this case), there are other problems. Check your logs.

share|improve this answer
add comment
up vote 2 down vote accepted

After looking through everything I saw a single error, which led me back to a mapreduce that was run on the primary, which had this issue: https://jira.mongodb.org/browse/SERVER-2861 . So when replication was attempted it failed to sync because of a faulty/corrupt operation in the oplog.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.