Questions tagged [kafka]

Kafka is a distributed messaging queue application written in java.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

How to remove escape character when using kafka console consumer?

For compliance purposes, I have to save all kafka raw documents and keep them one year. To do this, I use the following : kafka-console-consumer.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:...
0
votes
0answers
19 views

Kafka + Query the detailed configuration of all Topics

we have Kafka servers cluster with Kafka version 2.6 on Linux RHEL machines machine in the cluster are - kafka1,kafka2,kafka3 usually we are use the following cli in order to get the detailed ...
1
vote
1answer
44 views

zookeeper ssl alert "alert bad certificate"

I am using Kafka (version 2.3.0) and Zookeeper (version 3.5.5-3) - the stable version is 3.6.3. When I test the SSL of my Zookeeper using this command: openssl s_client -showcerts -connect 127.0.0.1:...
1
vote
1answer
20 views

apache, kafka, docker, on domain/URL

I need run Kafka on docker with web UI on some specific domain AND LINK, for example: http://somesite.com/kafka I am using this docker-compose.yml. version: '3.3' services: zookeeper: image: '...
2
votes
1answer
61 views

kafka confluent vs Apache kafka [closed]

for now we have HDP 2.6.5 version , with Kafka cluster with Kafka version 0.1 all Kafka machines are installed on RHEL version 7.6 all machines with RHEL 7.6 are on DELL physical machines we want to ...
0
votes
0answers
39 views

Kafka logs get reset when all brokers restart at the same time

I have a kafka cluster running on an integration Kubernetes cluster with 3 Kafka brokers and 3 Zookeepers, each component running in a statefulset. The helm chart I use to deploy the cluster is a ...
0
votes
1answer
177 views

Kafka Broker Runs locally via docker compose but fails in kubernetes with near identical config

Kafka broker runs beautifully when running locally via docker-compose. version: "3" services: zookeeper: image: 'bitnami/zookeeper:latest' ports: - '2181:2181' ...
0
votes
1answer
118 views

How to manage storage / persistence overflow in Bitnami Kafka Helm chart?

I deploy Bitanmi Kafka Helm chart on AWS provisioned with Terraform. I find the documentation storage and persistence allocation, very confusing. From what I understood from the documentation; logs ...
0
votes
0answers
30 views

Unable to update kafka cluster version in AWS MSK

We have written python code to upgrade kafka version in AWS MSK and its giving error .................................................. Traceback (most recent call last): File "./...
0
votes
0answers
107 views

Debezium blocks replication between two Aurora's

I'm running Debezium 1.4 using docker and Kafka-Connect image. Debezium connector has this configuration: { "name": "connector-name", "config": { ...
0
votes
1answer
109 views

How to rotate compressed log file on centos 7?

I have customized kafka server and the app is creating log files and configures log rotation. My problem is that I need to compress the rotation logs that were created. I tried to create new log ...
0
votes
0answers
37 views

What is required or need to implement inside microservice which has kafka consumer( B microservice) code to avoid 401 error

I have Microservices called A, B, C, D. A contains Kafka producer code. B contains Kafka consumer code. B consume messages from Kafka consumer and process further and interact with other microservice ...
0
votes
0answers
288 views

How to connect clickhouse to eventhub?

We have eventhub in azure and can connect to it using kafkacat tool. Config: security.protocol=SASL_SSL sasl.mechanisms=PLAIN sasl.username=$ConnectionString sasl.password=Endpoint=xxx In clickhouse(...
0
votes
0answers
81 views

Prometheus scraping issue with kafka-exporter

I am using kafka-exporter to be as a scraping endpoint for Kafka metrics. After installation of kafka exporter Prometheus does not collect and display any Kafka metric. Some extra information: i am ...
0
votes
0answers
17 views

Production Kafka cluster on Windows Server?

Anyone successfully running or deployed Kafka cluster on Windows servers in production? What's even worse to co-locate it with Zookeeper, NiFi and one more other application.. Am I only one to think ...
0
votes
1answer
228 views

The Debezium Kafka based KAFKA_ADVERTISED_LISTENERS property setting is ignored

I have a Docker Compose based Swarm with a Zookeeper service that starts fine, but the Kafka service cannot start. It keeps using the default KAFKA_ADVERTISED_LISTENERS property: debezium_kafka.1....
1
vote
1answer
108 views

PUB/SUB Message Queue with server side filtering?

I am looking for a message queue to realize a chat application. The most important requirement is: when a subscriber specifies a filter criteria, all the messages need to be filtered on the server ...
0
votes
1answer
118 views

How to implement a secured kafka environment with a public interface?

Scenario: Kafka cluster, secured with SSL encryption and SASL/PLAIN authentication, resides in an AWS private subnet within a dedicated VPC. Within the private subnet, all is fine. I use self-...
1
vote
0answers
131 views

How can I further analyze ram usage in a possible leak? [debian@HyperV + Kafka + Zookeeper]

We have a 3-box cluster Kafka+Zookeeper, in a virtualized enviroment (in Windows 2019 HyperV). In the machines' config dynamic memory is disabled, and they have 4G RAM each (I know it's a very few ...
1
vote
0answers
456 views

Confluent Kafka Connect S3 re-processing data due to offset out of range

I'm experiencing a strange situation. My Confluent Kafka Connect S3 is processing around 8K messages/sec. What started happening randomly every few days is: The Fetcher class would try to fetch an ...
0
votes
1answer
43 views

How set a fixed private IP for ECS containers?

I have a Task Definition called kafka-zookeeper, it has an Apache Kafka broker and a Zookeeper node in it. Those containers need to communicate with each other (that's easy) and with the kafka and ...
0
votes
1answer
781 views

Filebeat kafka input with SASL?

I'm trying to get filebeat to consume messages from kafka using the kafka input. I'm unable to authenticate with SASL for some reason and I'm not sure why that is. The documentation for both Kafka and ...
13
votes
9answers
19k views

Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)

Below the steps I did to get this issue : Launch ZooKeeper Launch Kafka : .\bin\windows\kafka-server-start.bat .\config\server.properties Also note that my OS is Windows And at the second step ...
0
votes
1answer
195 views

kafka in kubernetes pod

I am new at working with kubernetes setup, and are trying to run kafka pod with a persistence volume, so in case a pod goes down, the memory is not lost, and I can spin up a new cluster, using the ...
0
votes
1answer
263 views

mkfs + xfs + what is the right mkfs cli in order to create xfs file-system on huge disk

We need to create xfs file-system on kafka disk The special thing about kafka disk is the disk size kafka disk have 20TB size in our case I not sure about the following mkfs , but I need advice to ...
0
votes
0answers
949 views

zookeeper server due unable to load database on disk

We have kafka cluster with 3 nodes , each kafka include zookeeper server and schema registry We get the following error on one of the zookeeper server [2019-11-12 07:44:20,719] ERROR Unable to load ...
2
votes
1answer
337 views

Linux + docker container folder that mounted to OS volume + how to know the real limited size

inside the docker-compose.yml we configured the following volume image: confluentinc/cp-kafka:latest volumes: - /grid/kafka-data:/var/lib/kafka/data . docker-compose ps Name ...
1
vote
0answers
919 views

Getting error INFO [SocketServer brokerId=3] Failed authentication with /10.2.***.* (SSL handshake failed) (org.apache.kafka.common.network.Selector)

I have kafka multibroker deployed on kubernetes, with interbroker SSL communication, I have set the folowing properties: advertised.listeners=SSL://kafka-1.**.***.com:9093,PLAINTEXT://kafka-1.**.*****...
0
votes
1answer
87 views

Can logstash be used as a kafka forwarder?

I am looking at the plugins for Logstash and I noticed it had both an input and output plugin for kafka. For a single topic, can Logstash be used to forward kafka topic messages to another kafka ...
0
votes
1answer
72 views

Way to forward remote internal network ip and port to same ip and port on local mac

For development reasons I want to connect my local machine (macos) to a remote private network of a kafka/zookeeper cluster. When I do port forwarding with ssh root@000.000.000.000 -N -L 10.0.0.8:...
3
votes
2answers
1k views

Enable logical replication on Google Cloud Postgres

Is there anyway to enable logical replication wal_level = logical on Google Cloud Postgres instance? I want to do CDC (change data capture) to stream WAL Record to Apache Kafka. Thanks for your help!
0
votes
1answer
256 views

Creating RabbitMQ cluster (by Bitnami) and Kafka cluster (by Bitnami) in Azure and connecting to them from Kubernetes in another v-net

I'm trying to create an architecture with: 1. Kafka cluster (by Bitnami) 2. Rabbitmq cluster (by Bitnami) 3. Kubernetes cluster (AKS) that can use both Kafka and RMQ When I created both clusters (...
0
votes
1answer
52 views

Can I control which terraform instances are discarded when reducing count?

When having a multi-instance provisioning that needs to be reduced - can I control which of the existing resource will be killed? I have a Kafka cluster with variable load, so I would like to provide ...
0
votes
0answers
16 views

Message Forwarder to MapR Streams/Kafka

We have an application that forwards data to MapR Streams (Kafka). Problem is if the network link goes down then data is lost as the application itself doesn't keep a history. We are looking at ...
0
votes
1answer
116 views

Expose Confluent Kafka outside of the cluster but inside of the GCP

I've installed Confluent Kafka OSS using their official helm chart: https://github.com/confluentinc/cp-helm-charts However, I would like to consume Kafka topics in a GCP Dataflow so this mean, I ...
0
votes
1answer
119 views

hadoop + can we install zookeeper servers on kafka hosts

we want to dedicated the zookeeper servers only for kafka machines so each kafka machine include the zookeeper server and zookeeper server will serve only the kafka host and not other application in ...
1
vote
1answer
277 views

When to use Apache Kafka log roll jitter?

There are two log roll jitter related parameters in config doc, i.e., log.roll.jitter.ms/log.roll.jitter.hours. Both of them are marked as high importance, but I have no idea when to use it. What's ...
0
votes
1answer
78 views

Kafka Producer & Consumer hosts

Would Kafka Producer & Consumer hosts can be same. Suppose if I've producers like host1,host2 & host3. Can I use same host1, host2 & host3 as consumers? What are the consequences/...
1
vote
0answers
302 views

Replace value of key in Apache Kafka

I want kafka to only store the latest value of a given key in a topic. This seems so be possible, as shown here. However, the only think I archived so far, are that ALL Key/Value pairs get deleted or ...
0
votes
0answers
44 views

reducing tps in kafka disks on linux

we're using kafka brokers with: 24 HT cores 256GB RAM 16 X 7.2K RPM NLSAS disks configured in RAID-10 and replication factor 3. the sar -q command the runq-sz reached ~40 on average, load average ...
2
votes
1answer
533 views

Kafka log retention not wokring as expected

Note: I am aware that there are many similar questions, already looked at them, didn't help. I am running a kafka benchmark test in a closed environment, I gave each broker a filesystem of 40GB for ...
1
vote
3answers
1k views

kafka scripts in "bin" directory fail with "Address already in use" (running in kubernetes)

I have a kafka set up running in minikube. It's up and running, and I'm able to produce messages into it and consume them back out. However, when I'm logging in to one the brokers (using kubectl exec ...
0
votes
1answer
119 views

Dose zookeeper servers must use SSD disks?

we want to install kafka cluster and 3 zookeeper servers kafka should use the zookeeper servers in order to save the metadata on the zookeeper servers ZK Data and Log files should be on disks, which ...
4
votes
1answer
9k views

kafka failed authentication due to: SSL handshake failed

I have to add encryption and authentication with SSL in kafka. This is what I have done: - 1) Generate certificate for each broker kafka: COMANDO: keytool -keystore server.keystore.jks -alias ...
1
vote
0answers
2k views

Kafka broker listing empty nodes list *for a while*

For integration testing purposes I'm creating a very simple single-node Kafka deployment: 1x Zookeeper 1x Kafka 1x Kafka client (e.g. AdminClient, creating topics) (All cleanly deployed in fresh ...
0
votes
1answer
2k views

Kafka multiple listeners

Initial apologies for the long post (this is also on superuser as i wasn't sure the best place for this https://superuser.com/questions/1404421/kafka-multiple-listeners so let me know if one needs ...
0
votes
1answer
48 views

kafka machines in the cluster and kafka comunications

We have kafka cluster with 3 kafka brokers nodes and 3 zookperes servers kafka version - 10.1 ( hortonworks ) from my understanding since all meta data is located on the zookeeper servers , and ...
0
votes
1answer
167 views

How to monitor Kafka Consumer lag in Stackdriver?

I can view all the Kafka broker metrics by using the JMX exporter -> stackdriver monitoring agent and is displayed in the Stackdriver but i cant see any consumer metrics, i am particularly interested ...
1
vote
3answers
4k views

ZooKeeper not starting because acceptedEpoch is less than the currentEpoch

I have zookeeper.version=3.4.10, and I see: https://issues.apache.org/jira/browse/ZOOKEEPER-2307 that is however in unresolved status. The problem appeared because of filling up the disk space. Now ...
2
votes
0answers
693 views

Can connect to kafka, but cannot consume

I have a 3 broker kafka cluster set up in an AWS VPC with kubernetes, I am using the https://github.com/Yolean/kubernetes-kafka repo for this. I'm having the strange issue of being able to connect ...