Question: How do I fix my servers and get them back to the normal ReplicaSet with no error messages?
I'd offer a serious bounty, but all my points are on SO :(
Details of the Problem
I'm running a 3-node MongoDB ReplicaSet (1 master, 1 slave, 1 arbiter) on separate servers. I was toying around with automatic failover, ensuring that my slave would be promoted to master properly. After I took master down and the slave took over as master, I followed the instructions to re-elect my old master back to primary (Forcing a Member to be Primary). From my new PRIMARY, I ran:
> use admin
> db.auth("admin", "mypassword")
> config = rs.conf()
{
"_id" : "foo",
"version" : 1,
"members" : [
{
"_id" : 0,
"host" : "A",
},
{
"_id" : 1,
"host" : "B",
},
{
"_id" : 2,
"host" : "C",
}
]
}
> config.version++
> // the default priority is 1
> config.members[1].priority = 2
> rs.reconfig(config)
This threw an error, so I tried method #2:
db.adminCommand({replSetStepDown:1000000, force:1})
This seemed to botch my servers, causing no server to be promoted to primary and my old master and arbiter to start receiving the following errors:
Mon Aug 29 19:16:22 [rs Manager] replset msgReceivedNewConfig version: version: 2
Mon Aug 29 19:16:22 [rs Manager] replSet info saving a newer config version to local.system.replset
Mon Aug 29 19:16:22 [rs Manager] Server::doWork task:rs Manager exception:unauthorized db:local lock type:2 client:(NONE)
Mon Aug 29 19:16:22 [initandlisten] connection accepted from 192.168.139.50:28130 #476
Mon Aug 29 19:16:22 [initandlisten] connection accepted from 192.168.132.14:32014 #477
Mon Aug 29 19:16:23 [initandlisten] connection accepted from 192.168.139.50:28133 #478
Mon Aug 29 19:16:23 [initandlisten] connection accepted from 192.168.139.50:28136 #479
Mon Aug 29 19:16:23 [initandlisten] connection accepted from 192.168.132.14:32017 #480
Mon Aug 29 19:16:24 [initandlisten] connection accepted from 192.168.139.50:28139 #481
Mon Aug 29 19:16:24 [rs Manager] replset msgReceivedNewConfig version: version: 2
Mon Aug 29 19:16:24 [rs Manager] replSet info saving a newer config version to local.system.replset
Mon Aug 29 19:16:24 [rs Manager] Server::doWork task:rs Manager exception:unauthorized db:local lock type:2 client:(NONE)
Mon Aug 29 19:16:24 [rs Manager] replSet info not electing self, we are not freshest
Mon Aug 29 19:16:24 [initandlisten] connection accepted from 192.168.139.50:28142 #482
Mon Aug 29 19:16:24 [initandlisten] connection accepted from 192.168.132.14:32020 #483
Mon Aug 29 19:16:25 [initandlisten] connection accepted from 192.168.139.50:28145 #484
Mon Aug 29 19:16:25 [initandlisten] connection accepted from 192.168.139.50:28148 #485
Mon Aug 29 19:16:25 [initandlisten] connection accepted from 192.168.132.14:32023 #486
Mon Aug 29 19:16:26 [initandlisten] connection accepted from 192.168.139.50:28151 #487
After some googling, I found this to be a bug with ReplicaSets:
- https://groups.google.com/group/mongodb-user/browse_thread/thread/8aad09f39d34370e?pli=1
- http://www.google.com/url?sa=D&q=https://jira.mongodb.org/browse/SERVER-2919
A few bits of information on my setup:
- I'm running on Mongo 1.8.1
- I have
journal = true
- I have
keyFile = /path/to/file
- I have user authentication enabled (which appears to be causing the bug)
- I have a firewall enabled with ports 27017-27019 opened for mongod
- Everything was working fine prior to the switch