Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Question: How do I fix my servers and get them back to the normal ReplicaSet with no error messages?

I'd offer a serious bounty, but all my points are on SO :(

Details of the Problem

I'm running a 3-node MongoDB ReplicaSet (1 master, 1 slave, 1 arbiter) on separate servers. I was toying around with automatic failover, ensuring that my slave would be promoted to master properly. After I took master down and the slave took over as master, I followed the instructions to re-elect my old master back to primary (Forcing a Member to be Primary). From my new PRIMARY, I ran:

> use admin
> db.auth("admin", "mypassword")
> config = rs.conf()
{
    "_id" : "foo",
    "version" : 1,
    "members" : [
        {
            "_id" : 0,
            "host" : "A",
        },
        {
            "_id" : 1,
            "host" : "B",
        },
        {
            "_id" : 2,
            "host" : "C",
        }
    ]
}
> config.version++
> // the default priority is 1
> config.members[1].priority = 2
> rs.reconfig(config)

This threw an error, so I tried method #2:

db.adminCommand({replSetStepDown:1000000, force:1})

This seemed to botch my servers, causing no server to be promoted to primary and my old master and arbiter to start receiving the following errors:

Mon Aug 29 19:16:22 [rs Manager] replset msgReceivedNewConfig version: version: 2
Mon Aug 29 19:16:22 [rs Manager] replSet info saving a newer config version to local.system.replset
Mon Aug 29 19:16:22 [rs Manager] Server::doWork task:rs Manager exception:unauthorized db:local lock type:2 client:(NONE)
Mon Aug 29 19:16:22 [initandlisten] connection accepted from 192.168.139.50:28130 #476
Mon Aug 29 19:16:22 [initandlisten] connection accepted from 192.168.132.14:32014 #477
Mon Aug 29 19:16:23 [initandlisten] connection accepted from 192.168.139.50:28133 #478
Mon Aug 29 19:16:23 [initandlisten] connection accepted from 192.168.139.50:28136 #479
Mon Aug 29 19:16:23 [initandlisten] connection accepted from 192.168.132.14:32017 #480
Mon Aug 29 19:16:24 [initandlisten] connection accepted from 192.168.139.50:28139 #481
Mon Aug 29 19:16:24 [rs Manager] replset msgReceivedNewConfig version: version: 2
Mon Aug 29 19:16:24 [rs Manager] replSet info saving a newer config version to local.system.replset
Mon Aug 29 19:16:24 [rs Manager] Server::doWork task:rs Manager exception:unauthorized db:local lock type:2 client:(NONE)
Mon Aug 29 19:16:24 [rs Manager] replSet info not electing self, we are not freshest
Mon Aug 29 19:16:24 [initandlisten] connection accepted from 192.168.139.50:28142 #482
Mon Aug 29 19:16:24 [initandlisten] connection accepted from 192.168.132.14:32020 #483
Mon Aug 29 19:16:25 [initandlisten] connection accepted from 192.168.139.50:28145 #484
Mon Aug 29 19:16:25 [initandlisten] connection accepted from 192.168.139.50:28148 #485
Mon Aug 29 19:16:25 [initandlisten] connection accepted from 192.168.132.14:32023 #486
Mon Aug 29 19:16:26 [initandlisten] connection accepted from 192.168.139.50:28151 #487

After some googling, I found this to be a bug with ReplicaSets:

A few bits of information on my setup:

  • I'm running on Mongo 1.8.1
  • I have journal = true
  • I have keyFile = /path/to/file
  • I have user authentication enabled (which appears to be causing the bug)
  • I have a firewall enabled with ports 27017-27019 opened for mongod
  • Everything was working fine prior to the switch
share|improve this question
    
As you have mentioned, this was a bug you found reported in the Jira ticket. It has been reported fixed in version 1.8.2 and later, as well as the more recent versions. Is this still a problem? –  Mike Fiedler Feb 20 '12 at 6:19
add comment

1 Answer

As the reporter updated in the question, this was reported as a bug, and has since been fixed in a newer release of the software.

Bug reported/fixed here: https://jira.mongodb.org/browse/SERVER-2919

This question should be marked answered/closed.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.