Scenario: Kafka cluster, secured with SSL encryption and SASL/PLAIN authentication, resides in an AWS private subnet within a dedicated VPC. Within the private subnet, all is fine. I use self-generated CA and keys to secure the communication, host identities are based on the AWS internal DNS.

What I want to achieve: to have a possibilty to access the producer API (not REST API) from the outside.

I am struggeling with the combination of keys, DNS, kafka listeners and the fact that the permanent connection from a producer to a broker is possibly not the one that I used to start the connection.

Several attempts with reverse proxies failed - even a ssh tunnel does not work since the keys cannot be resolved.

Does anybody have a kind of reference architecture for such a case? I spare the config details here since it is too much spread in the different configs, keys, etc. but if needed I can provide my settings.


Ok I found a solution - maybe not most efficient but fine for me. I implemented three level 4 load balancers in front of the three brokers and configured each broker's advanced_listener address with the one of the load balancer. works like a charm - even if it's comparatively expensive and throttles traffic rates somewhat.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.