I have a zookeeper ensemble running in Kubernetes consisting of 3 instances. The ensemble operates mostly correctly and successfully services the Solr cluster it is backing. One thing does seem incorrect however, and it's breaking some health checks in another part of my application (Sitecore) as a result.
For some reason, the Solr -> Cloud -> ZK Status page -AND- the mntr command on the leader itself report sporadically changing and incorrect numbers of followers:
root@zk-1:/zookeeper-3.4.14# echo mntr | nc localhost 2181 zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT zk_avg_latency 2 zk_max_latency 4 zk_min_latency 1 zk_packets_received 1132 zk_packets_sent 1131 zk_num_alive_connections 5 zk_outstanding_requests 0 zk_server_state leader zk_znode_count 1405 zk_watch_count 0 zk_ephemerals_count 57 zk_approximate_data_size 4703048 zk_open_file_descriptor_count 42 zk_max_file_descriptor_count 1048576 zk_fsync_threshold_exceed_count 0 zk_followers 5 zk_synced_followers 2 zk_pending_syncs 0 zk_last_proposal_size 92 zk_max_proposal_size 55791 zk_min_proposal_size 32
Note the number of reported followers (zk_followers) is not consistent - in this sample the GUI reports 4 but the command line reports 5. This number jumps up and down every time I refresh the GUI or re-run the mntr command, usually anywhere from 3-6 (the correct result obviously should be 2)
I can't work out what these extra followers being reported could be, as no output other than the zk_followers var shows anything other than the two real ones and logs don't seem to report anything untoward.
root@zk-1:/zookeeper-3.4.14# cat /conf/zoo.cfg clientPort=2181 dataDir=/store/data dataLogDir=/store/datalog tickTime=2000 initLimit=5 syncLimit=2 autopurge.snapRetainCount=3 autopurge.purgeInterval=0 maxClientCnxns=60 server.1=zk-0.zk:2888:3888 server.2=0.0.0.0:2888:3888 server.3=zk-2.zk:2888:3888
root@zk-1:/zookeeper-3.4.14# cat /conf/zooservers.txt server.1=zk-0.zk:2888:3888 server.2=0.0.0.0:2888:3888 server.3=zk-2.zk:2888:3888
As noted this doesn't seem to cause any real problems with the ensemble or the solr cluster except that it's failing some container start health checks which cause inconsistency in the application, and could potentially bring down parts of it.