Does Kafka replicate the same data across all brokers?

Question

I am going to implement Kafka in the infrastructure. I have a use case where we have below components in the Kafka architecture.

• Number of brokers: 5
• Replication Factors: 3
• Number of topics: 3
• Number of partition: 3

I have to implement scaling of the Kafka cluster, which means I want the kafka infrastructure to be capable enough to add/remove/replace brokers. Adding/Removing the instance needs to sync the whole data of other brokers in order to be the ISR(in-sync replicas). We wanted to AWS autoscaling to do that and we don't actually want the 100s of GB of data across the availability zone if eventually Kafka rebalances the leader of some partition and assign to this new broker. Hence, we wanted to use the same old volume of one of the broker.

The questions are:

1. Which broker i have to opt for if I want to take a snapshot of its
   ephermal storage -> create volume out of it and mount it to another (new) broker.
2. Does Kafka replicate the same data across all brokers regardless of the leaders of a particular partition?
3. What would be the best way to achieve this if we want to save the data transfer cost which incurs while syncing the data from other brokers to the new broker and also want to scale the cluster.

Answer 1

1. I doubt you would able to just make a copy of existing broker to scale up.... as no too brokers are same, as node-1 may be a leader for topic-1>partition-1 while node-2 may be a leader for topic-1>partition-2.

However I'll be delighted to find if there is any such solution.

2. Kafka replicate it to the replication factor, since your replication factor is 3, so it will be replicated to 3 machines. However replication is gonna happen by partition. So continuing above example... topic-1>partition-1 leader is node-1, but copy may be stored on node-2 and node-3

3. Ideally one should clear the topics, and don't retain data too long ... that way you don't have to replicate lot of data across new nodes. But if your use-case demands so then I'm not sure if there is any efficient solution