2

We have a fairly large Mongo DB running in AWS. Currently, we are running with a replica set with 3 instances. Each instance has 5TB of attached EBS storage. This comes out to over $1000/month per instance. On top of this, we have both a prod and staging environment (with a third "dev" environment coming soon). Further, these costs stand to explode in the future when/if we move to a sharded environment.

The question is, how necessary are 3 replicas in an AWS environment?

Ok ok ok, I already know the answer is "it depends". What I'm looking for is some advice on how to best weigh the trade offs. For example...

  1. Considering that each EBS volume already has triple redundancy built in and that it's fairly straight forward to restore from a backup, how do I measure the added fault tolerance of 2 vs. 3 replicas.

  2. Are there additional considerations besides redundancy when considering the trade offs?

  3. Does anyone have experience (good or bad) with running only 2 replicas + an Arbiter?

1
  • Just for some added context... removing 1 replica from each environment could reduce our month infrastructure costs by ~20% (so nothing to sneeze at) Commented Mar 15, 2018 at 21:08

1 Answer 1

2
  1. Considering that each EBS volume already has triple redundancy built in and that it's fairly straight forward to restore from a backup, how do I measure the added fault tolerance of 2 vs. 3 replicas.

As far as MongoDB goes, key considerations with only two data-bearing members in a three node replica set are that if one of those data-bearing members is unavailable for any reason (planned maintenance or unplanned failure):

  • you no longer have active replication (there is only one data-bearing member remaining)
  • your deployment can no longer acknowledge write concerns higher than w:1 (for example: w:majority or w:2)

This configuration has high availability in terms of maintaining/electing a primary in the event of a single member failure, but the arbiter compromises data redundancy if one of your data-bearing members is unavailable. Assuming you have reasonable time-to-restore from your EBS backups (and trust in EBS redundancy), this may be an acceptable compromise for your use case.

  1. Are there additional considerations besides redundancy when considering the trade offs?

If your code is using MongoDB write concerns higher than the default (w:1) you will want to add a wtimeout value. If you do not specify the wtimeout option and the level of write concern is unachievable, write operations will block indefinitely.

AWS guarantees on redundant infrastructure generally only extend to failures across multiple availability zones, so to maximise availability you should also deploy your replica set members into different availability zones.

  1. Does anyone have experience (good or bad) with running only 2 replicas + an Arbiter

I've definitely seen bad outcomes where users failed to consider the above points (particularly with consideration of write concerns & timeouts). If you plan (and test) with those caveats in mind, you should be able to land on the side of good experience.

On top of this, we have both a prod and staging environment (with a third "dev" environment coming soon)

There's definitely an argument for having prod-like staging and dev environments, but a typical cost savings would be deploying lower spec environments for dev with less failover than production. For staging you may want to deploy lower spec environments but have similar configuration so you can test realistic failover scenarios. If you are doing performance or load testing in staging environments, they should be provisioned with the same specs as production.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .