Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have three servers with mongod installed on it running as a replication set. Suddenly the two secondories became unavailable (the mongod process died) - I think because they were too stale.

The problem is that the original PRIMARY is now the SECONDARY and my application doesn't work because it can't connect to a PRIMARY.

I mean, in which way does that help me? If the replica set can't do failover?! Am I missing something?

Furhtermore I am asking myself why did the SECONDARIES die / why are they too stale? What can I do about it?

FYI: My database is quite big (40GB on disk).

share|improve this question
add comment

1 Answer

up vote 3 down vote accepted

The reason why your primary is now no longer primary is because you have to have enough votes to form a majority in a replica set - in your case that means 2/3 minimum. With your specific failure that doesn't seem to make sense, but imagine instead that the primary in question had instead become isolated from the 2 secondaries (a network event for example). In that case you would not want two primaries elected, so instead it would step down until it could see the other set members.

In order to get the primary back to working you need to do one of two things:

  1. Get the other set members back up
  2. Reconfigure the set so that a majority can be formed (add more members, go to a single)

See here for more information on the reconfiguration option:

http://www.mongodb.org/display/DOCS/Reconfiguring+a+replica+set+when+members+are+down

Basically, your replica set will failover, but you have to have enough members to actually do an election.

In terms of why the secondaries died, I can't say based on the information you have provided here, but I can tell you that it's not because they were too stale. A secondary that falls too far behind will report as being stale but will keep running (and voting for the primary) even in that state, until you fix it and bring it back up to date. See here for how to remedy a stale set member:

http://www.mongodb.org/display/DOCS/Resyncing+a+Very+Stale+Replica+Set+Member

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.