In order to make our Kafka cluster available both from Internet and from our private network, we thought fine to configure Kafka this way :
Private VIP:9000 => All brokers:9092 (topology query only) Private VIP:9001 => Broker #1:9092 Private VIP:9002 => Broker #2:9092 ... Public VIP:9000 => All brokers:9092 (topology query only) Public VIP:9001 => Broker #1:9092 Public VIP:9002 => Broker #2:9092 ...
We configured our Load balancer this way, and then our brokers :
listeners=PLAINTEXT://<server_priv_ip>:9092 advertised.listeners=INTERNAL://<private_VIP>:9001,EXTERNAL://<public_vip>:9001 listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT inter.broker.listener.name=INTERNAL
Now, as you would expect, Kafka brokers don't start :
ERROR Exiting Kafka due to fatal exception (kafka.Kafka$) java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port, listeners: INTERNAL://<private_VIP>:9001,EXTERNAL://<public_vip>:9001
From my Administrator point of view, my approach was totally logical, although I expected a problem where producer/consumers would get both addresses wherever they contacted the broker from. Using the same port on 2 different addresses seems logical and promotes clarity...
First question : Why is this wrong ?
Second question : How can I achieve my goal if not this way? (most options are on the table)