Scenario: Kafka cluster, secured with SSL encryption and SASL/PLAIN authentication, resides in an AWS private subnet within a dedicated VPC. Within the private subnet, all is fine. I use self-generated CA and keys to secure the communication, host identities are based on the AWS internal DNS.
What I want to achieve: to have a possibilty to access the producer API (not REST API) from the outside.
I am struggeling with the combination of keys, DNS, kafka listeners and the fact that the permanent connection from a producer to a broker is possibly not the one that I used to start the connection.
Several attempts with reverse proxies failed - even a ssh tunnel does not work since the keys cannot be resolved.
Does anybody have a kind of reference architecture for such a case? I spare the config details here since it is too much spread in the different configs, keys, etc. but if needed I can provide my settings.