Questions tagged [kafka]

Kafka is a distributed messaging queue application written in java.

Filter by
Sorted by
Tagged with
0
votes
0answers
24 views

dial tcp: lookup *** no such host. Kafka connect issue

I'm struggling to understand what might be the problem with my system here. I have a service with a pretty basic workflow: connect to the kafka cluster, generate some messages, send them in a few ...
1
vote
0answers
25 views

How can I further analyze ram usage in a possible leak? [debian@HyperV + Kafka + Zookeeper]

We have a 3-box cluster Kafka+Zookeeper, in a virtualized enviroment (in Windows 2019 HyperV). In the machines' config dynamic memory is disabled, and they have 4G RAM each (I know it's a very few ...
0
votes
0answers
37 views

I would like to find all the consumers for a kafka topic

Normally I used to check list of consumer groups by "kafka-consumer-groups.sh" command on Linux. Now I want to get the result from a Windows machine. I noticed that .bat binaries missing. ...
0
votes
0answers
109 views

Confluent Kafka Connect S3 re-processing data due to offset out of range

I'm experiencing a strange situation. My Confluent Kafka Connect S3 is processing around 8K messages/sec. What started happening randomly every few days is: The Fetcher class would try to fetch an ...
0
votes
1answer
29 views

How set a fixed private IP for ECS containers?

I have a Task Definition called kafka-zookeeper, it has an Apache Kafka broker and a Zookeeper node in it. Those containers need to communicate with each other (that's easy) and with the kafka and ...
0
votes
0answers
24 views

Push mongodb collections to Amazon redshift

I have got a mongodb database. I need to stream or push data from mongodb to Amazon redshift as and when data arrives in mongodb. Is there any solution for this..? Can this be implemented using Apache ...
0
votes
0answers
61 views

kafka + using RAM disk insted of physical disk

we are using kafka cluster , when each kafka broker have physical disk is it possible that kafka broker will used RAM disk instead of physical disk ? , What is a memory based file system (RAM disk)...
0
votes
1answer
260 views

Filebeat kafka input with SASL?

I'm trying to get filebeat to consume messages from kafka using the kafka input. I'm unable to authenticate with SASL for some reason and I'm not sure why that is. The documentation for both Kafka and ...
0
votes
0answers
41 views

Dose even number of Kafka brokers affect ensemble operation and leader election

we are little confuse about if is valid or not to use kafka cluster with even numbers the story begin when we build kafka cluster with 6 machines and trying to use the kafka Reassignment partitions ...
0
votes
0answers
48 views

kafka + Enabling High Performance Data Transfers

when we look on kafka best practice we see the following details: Max socket buffer size , can be increased to enable high-performance data transfer. More details are here http://www.psc.edu/index....
1
vote
6answers
8k views

Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)

Below the steps I did to get this issue : Launch ZooKeeper Launch Kafka : .\bin\windows\kafka-server-start.bat .\config\server.properties Also note that my OS is Windows And at the second step ...
0
votes
0answers
18 views

kafka migration failure when attempting to upgrade data stream

I have inherited a legacy kafka system, version 0.8.2. I am now trying to upgrade the system to something more modern. I know I cannot go straight from 0.8.2 to something past 1.0, but I was hoping ...
0
votes
1answer
108 views

kafka in kubernetes pod

I am new at working with kubernetes setup, and are trying to run kafka pod with a persistence volume, so in case a pod goes down, the memory is not lost, and I can spin up a new cluster, using the ...
0
votes
0answers
38 views

is it right to mix diff file system between kafka machines

we have kafka cluster with 3 machines each kafka machine use disk for DATA , and each disk is 20TB size the file system on the disk is ext4 now we want to add additional 5 kafka machines but to ...
0
votes
0answers
46 views

Connect Kafka to Keycloak Gatekeeper

I have Kafka and zookeeper running on my Kubernetes cluster. I am also using KeyCloak for authentication. How can I connect Kafka to Keycloak Gatekeeper? How can I set up the generic json gatekeeper ...
0
votes
1answer
93 views

mkfs + xfs + what is the right mkfs cli in order to create xfs file-system on huge disk

We need to create xfs file-system on kafka disk The special thing about kafka disk is the disk size kafka disk have 20TB size in our case I not sure about the following mkfs , but I need advice to ...
0
votes
0answers
217 views

zookeeper server due unable to load database on disk

We have kafka cluster with 3 nodes , each kafka include zookeeper server and schema registry We get the following error on one of the zookeeper server [2019-11-12 07:44:20,719] ERROR Unable to load ...
0
votes
0answers
19 views

how to recover HDP cluster by installing HDP from scratch and save data-node disks and kafka disks data

I want to ask this important question lets say we have the following: HDP cluster with : 3 masters machine ( active/standby name-node ) , ( active/standby resource manager ) 3 datanode machines ...
2
votes
1answer
75 views

Linux + docker container folder that mounted to OS volume + how to know the real limited size

inside the docker-compose.yml we configured the following volume image: confluentinc/cp-kafka:latest volumes: - /grid/kafka-data:/var/lib/kafka/data . docker-compose ps Name ...
1
vote
0answers
502 views

Getting error INFO [SocketServer brokerId=3] Failed authentication with /10.2.***.* (SSL handshake failed) (org.apache.kafka.common.network.Selector)

I have kafka multibroker deployed on kubernetes, with interbroker SSL communication, I have set the folowing properties: advertised.listeners=SSL://kafka-1.**.***.com:9093,PLAINTEXT://kafka-1.**.*****...
0
votes
0answers
40 views

kafka using docker VS not using docker ( ordinary kafka cluster )

for now we have production HDP cluster include kafka as the following: 3 masters machines 10 data node machines 3 kafka machines ( in cluster ) HDP version - 2.6.4 Ambari - 2.6.2.2 We are ...
0
votes
0answers
138 views

kafka topic consumer offset reset using --to-datetime always results in latest offset values

I am attempting to rewind the consumer offsets on a topic back to 1 day ago. I am finding that when I use kafka-consumer-groups.sh --reset-offsets with either --to-datetime or --by-duration, the ...
0
votes
1answer
52 views

Can logstash be used as a kafka forwarder?

I am looking at the plugins for Logstash and I noticed it had both an input and output plugin for kafka. For a single topic, can Logstash be used to forward kafka topic messages to another kafka ...
0
votes
1answer
36 views

Way to forward remote internal network ip and port to same ip and port on local mac

For development reasons I want to connect my local machine (macos) to a remote private network of a kafka/zookeeper cluster. When I do port forwarding with ssh root@000.000.000.000 -N -L 10.0.0.8:...
0
votes
0answers
28 views

Can I change kafka retention period dynamically server wide on all topics?

If I want to increase the retention period server wide on all topics, I'll update "log.retention.hours= " parameter in server.properties file. Is it mandatory to restart all the brokers and zookeepers ...
0
votes
0answers
38 views

Modify MSK Configuration

I created an MSK cluster with a cluster configuration. I now want to modify the configuration. MSK clusters have a "Revision" field, so intuitively I want to create a new revision. However, I can't ...
1
vote
1answer
382 views

Enable logical replication on Google Cloud Postgres

Is there anyway to enable logical replication wal_level = logical on Google Cloud Postgres instance? I want to do CDC (change data capture) to stream WAL Record to Apache Kafka. Thanks for your help!
0
votes
0answers
14 views

update kafka broker from 2.11-1.0.0 to 1.01

We have 3 zookeeper & 3 kafka brokers. I just wants to update kafka broker from 2.11-1.0.0 to 1.0.1. What is the procedure to update the version and keep all my topics as is?
0
votes
0answers
47 views

Return MSK Kafka endpoint in CloudFormation script

I'm using a CloudFormation script to create a MSK cluster in AWS. I want to add a Route53 entry for the cluster in the same CloudFormation script, but for this I need an endpoint for the cluster (...
0
votes
1answer
195 views

Creating RabbitMQ cluster (by Bitnami) and Kafka cluster (by Bitnami) in Azure and connecting to them from Kubernetes in another v-net

I'm trying to create an architecture with: 1. Kafka cluster (by Bitnami) 2. Rabbitmq cluster (by Bitnami) 3. Kubernetes cluster (AKS) that can use both Kafka and RMQ When I created both clusters (...
0
votes
1answer
34 views

Can I control which terraform instances are discarded when reducing count?

When having a multi-instance provisioning that needs to be reduced - can I control which of the existing resource will be killed? I have a Kafka cluster with variable load, so I would like to provide ...
0
votes
0answers
15 views

Message Forwarder to MapR Streams/Kafka

We have an application that forwards data to MapR Streams (Kafka). Problem is if the network link goes down then data is lost as the application itself doesn't keep a history. We are looking at ...
0
votes
1answer
61 views

Expose Confluent Kafka outside of the cluster but inside of the GCP

I've installed Confluent Kafka OSS using their official helm chart: https://github.com/confluentinc/cp-helm-charts However, I would like to consume Kafka topics in a GCP Dataflow so this mean, I ...
0
votes
1answer
52 views

hadoop + can we install zookeeper servers on kafka hosts

we want to dedicated the zookeeper servers only for kafka machines so each kafka machine include the zookeeper server and zookeeper server will serve only the kafka host and not other application in ...
1
vote
1answer
108 views

When to use Apache Kafka log roll jitter?

There are two log roll jitter related parameters in config doc, i.e., log.roll.jitter.ms/log.roll.jitter.hours. Both of them are marked as high importance, but I have no idea when to use it. What's ...
0
votes
1answer
70 views

Kafka Producer & Consumer hosts

Would Kafka Producer & Consumer hosts can be same. Suppose if I've producers like host1,host2 & host3. Can I use same host1, host2 & host3 as consumers? What are the consequences/...
1
vote
0answers
129 views

Replace value of key in Apache Kafka

I want kafka to only store the latest value of a given key in a topic. This seems so be possible, as shown here. However, the only think I archived so far, are that ALL Key/Value pairs get deleted or ...
0
votes
0answers
35 views

reducing tps in kafka disks on linux

we're using kafka brokers with: 24 HT cores 256GB RAM 16 X 7.2K RPM NLSAS disks configured in RAID-10 and replication factor 3. the sar -q command the runq-sz reached ~40 on average, load average ...
2
votes
1answer
187 views

Kafka log retention not wokring as expected

Note: I am aware that there are many similar questions, already looked at them, didn't help. I am running a kafka benchmark test in a closed environment, I gave each broker a filesystem of 40GB for ...
0
votes
3answers
440 views

kafka scripts in “bin” directory fail with “Address already in use” (running in kubernetes)

I have a kafka set up running in minikube. It's up and running, and I'm able to produce messages into it and consume them back out. However, when I'm logging in to one the brokers (using kubectl exec ...
0
votes
1answer
54 views

Dose zookeeper servers must use SSD disks?

we want to install kafka cluster and 3 zookeeper servers kafka should use the zookeeper servers in order to save the metadata on the zookeeper servers ZK Data and Log files should be on disks, which ...
4
votes
1answer
4k views

kafka failed authentication due to: SSL handshake failed

I have to add encryption and authentication with SSL in kafka. This is what I have done: - 1) Generate certificate for each broker kafka: COMANDO: keytool -keystore server.keystore.jks -alias ...
1
vote
0answers
1k views

Kafka broker listing empty nodes list *for a while*

For integration testing purposes I'm creating a very simple single-node Kafka deployment: 1x Zookeeper 1x Kafka 1x Kafka client (e.g. AdminClient, creating topics) (All cleanly deployed in fresh ...
0
votes
1answer
1k views

Kafka multiple listeners

Initial apologies for the long post (this is also on superuser as i wasn't sure the best place for this https://superuser.com/questions/1404421/kafka-multiple-listeners so let me know if one needs ...
0
votes
1answer
40 views

kafka machines in the cluster and kafka comunications

We have kafka cluster with 3 kafka brokers nodes and 3 zookperes servers kafka version - 10.1 ( hortonworks ) from my understanding since all meta data is located on the zookeeper servers , and ...
0
votes
1answer
134 views

How to monitor Kafka Consumer lag in Stackdriver?

I can view all the Kafka broker metrics by using the JMX exporter -> stackdriver monitoring agent and is displayed in the Stackdriver but i cant see any consumer metrics, i am particularly interested ...
1
vote
2answers
2k views

ZooKeeper not starting because acceptedEpoch is less than the currentEpoch

I have zookeeper.version=3.4.10, and I see: https://issues.apache.org/jira/browse/ZOOKEEPER-2307 that is however in unresolved status. The problem appeared because of filling up the disk space. Now ...
2
votes
0answers
401 views

Can connect to kafka, but cannot consume

I have a 3 broker kafka cluster set up in an AWS VPC with kubernetes, I am using the https://github.com/Yolean/kubernetes-kafka repo for this. I'm having the strange issue of being able to connect ...
0
votes
1answer
260 views

stackdriver not monitoring kafka metrics

I am trying to setup kafka metrics on stackdriver. I followed all steps as mentioned in https://cloud.google.com/monitoring/agent/plugins/kafka , but still cant see "kafka" metrics. Although I can see ...
3
votes
1answer
151 views

Does Kafka replicate the same data across all brokers?

I am going to implement Kafka in the infrastructure. I have a use case where we have below components in the Kafka architecture. Number of brokers: 5 Replication Factors: 3 Number of topics: 3 ...