0

I have two databases with the same data (~1TB): one for write with no shards and indexes I need for updating, and the second one for reading sharded and indexes only for reading. How to setup automatic replication between two databases? I need to have these databases syncronized. Now I'm using message queue and scripts for moving data, but is there some common solution for this problem?

1

I have two databases with the same data (~1TB): one for write with no shards and indexes I need for updating, and the second one for reading sharded and indexes only for reading. How to setup automatic replication between two databases? I need to have these databases syncronized.

What you are describing is sync between two deployments. MongoDB's built-in replication only supports replication between members of the same deployment, where all members share a common data history via the replication oplog. With independent deployments that each allow write access, there is no guarantee the deployments will be (or remain) in sync.

If you are working with the same data set, it would be more typical to have a single sharded deployment with appropriate zone sharding and read preferences to achieve the desired workload isolation.

Sharding is also more typically used for horizontal write scaling, with each shard backed by a replica set for redundancy and read scaling.

Now I'm using message queue and scripts for moving data, but is there some common solution for this problem?

If you do want to maintain separate deployments you could consider writing a monitoring script using change streams to watch individual collections (MongoDB 3.6+) or all non-system collections (MongoDB 4.0+). There are also third party tools like Mongo Connector, or you could roll your own solution with message queues (as you have done).

Unless there are strong reasons not to use the built-in replication, I would consider doing so in order to avoid the administrative overhead of syncing and maintaining two deployments.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.