- Considering that each EBS volume already has triple redundancy built in and that it's fairly straight forward to restore from a backup, how do I measure the added fault tolerance of 2 vs. 3 replicas.
As far as MongoDB goes, key considerations with only two data-bearing members in a three node replica set are that if one of those data-bearing members is unavailable for any reason (planned maintenance or unplanned failure):
- you no longer have active replication (there is only one data-bearing member remaining)
- your deployment can no longer acknowledge write concerns higher than
w:1 (for example: w:majority or w:2)
This configuration has high availability in terms of maintaining/electing a primary in the event of a single member failure, but the arbiter compromises data redundancy if one of your data-bearing members is unavailable. Assuming you have reasonable time-to-restore from your EBS backups (and trust in EBS redundancy), this may be an acceptable compromise for your use case.
- Are there additional considerations besides redundancy when considering the trade offs?
If your code is using MongoDB write concerns higher than the default (w:1) you will want to add a wtimeout value. If you do not specify the wtimeout option and the level of write concern is unachievable, write operations will block indefinitely.
AWS guarantees on redundant infrastructure generally only extend to failures across multiple availability zones, so to maximise availability you should also deploy your replica set members into different availability zones.
- Does anyone have experience (good or bad) with running only 2 replicas + an Arbiter
I've definitely seen bad outcomes where users failed to consider the above points (particularly with consideration of write concerns & timeouts). If you plan (and test) with those caveats in mind, you should be able to land on the side of good experience.
On top of this, we have both a prod and staging environment (with a third "dev" environment coming soon)
There's definitely an argument for having prod-like staging and dev environments, but a typical cost savings would be deploying lower spec environments for dev with less failover than production. For staging you may want to deploy lower spec environments but have similar configuration so you can test realistic failover scenarios. If you are doing performance or load testing in staging environments, they should be provisioned with the same specs as production.
