I am going to implement Kafka in the infrastructure. I have a use case where we have below components in the Kafka architecture.
- Number of brokers: 5
- Replication Factors: 3
- Number of topics: 3
- Number of partition: 3
I have to implement scaling of the Kafka cluster, which means I want the kafka infrastructure to be capable enough to add/remove/replace brokers. Adding/Removing the instance needs to sync the whole data of other brokers in order to be the ISR(in-sync replicas). We wanted to AWS autoscaling to do that and we don't actually want the 100s of GB of data across the availability zone if eventually Kafka rebalances the leader of some partition and assign to this new broker. Hence, we wanted to use the same old volume of one of the broker.
The questions are:
- Which broker i have to opt for if I want to take a snapshot of its
ephermal storage -> create volume out of it and mount it to another (new) broker.
- Does Kafka replicate the same data across all brokers regardless of the leaders of a particular partition?
- What would be the best way to achieve this if we want to save the data transfer cost which incurs while syncing the data from other brokers to the new broker and also want to scale the cluster.