mongodb: can client suggest re-election of replica set?

Question

A Mongodb replica set provides automatic fail over. If a primary goes down (ie there is a network problem between the primary and any of the secondary), then a re-election occurs.

However, consider the following case:

All servers are up, the network between the primary and secondaries is fine and thus replica set is happy as can be. However, due to another network issue (unkown to the replica set), the client can't reach the primary. Thus HA is lost.

In this case, can the client ping a secondary and inform the secondary that the primary can't be reached and that a new election should be run. Is this possible?

Answer 1

You could probably do this in a disaster recovery situation, where getting the database up is important, and you don't mind a rollback.

Something like:

cfg = rs.conf()
cfg.members[0].priority = 2
rs.reconfig(cfg, { force: true })

Where members[0] is a known 'good' server.

A safer way to do this would probably to install a small management service on each mongo node that you could hit via rest api. If at all possible, admin commands should be relayed to the primary.

Automating this sort of thing would be quite difficult, as there is likely no easy way for your applications and servers to agree on what servers are 'good', and which servers are bad.

One common model for mongo clusters is to run nodes in multiple locations, some of which may have less reliable routing. In these cases I would recommend modifying your node priorities. If you have servers you would never want as a primary, set their priorities to 0, and set nodes in your preferred location as priority 2 (or higher).

For more information force reconfigure, read: https://docs.mongodb.com/v3.2/tutorial/reconfigure-replica-set-with-unavailable-members/

Edit: I think I'd be tempted to play about with mongo arbiters. By running mongo arbiters in the locations where you run the clients, they would be able to vote in the election process. Servers that can't be seen by the majority of voting servers cannot be a primary.