I am aware that as long as there is quorum my zookeeper ensemble will keep working. But does the missing server have any notable impact on the cluster performance?
Let's suppose 1 of 3 servers crashes (and gets hdd destroyed). I guess I can join a new (clean) server without downtime as long as it has the same server-id as the old one and the other two servers can connect to it (it has same IP / hostname as old one in their configs)?
What is the impact of the third server "resyncing" (i.e., will it affect the speed on which consensus for new stuff is reached)? How long does that operation usually take (in relation to the amount of data in zookeeper)?
Can (or should) I just copy data and datalog from one of the existing servers? Snapshots are probably safe to be copied as-is, but transaction logs might need a "point-in-time copy" (I have btrfs CoW, so this is no problem)?
Or to be more specific I also wonder whether data on all nodes is equivalent (beside the latest writes) and interchangable. Or is somehow server-id specific stuff stored inside?