Questions tagged [kafka]

Kafka is a distributed messaging queue application written in java.

Filter by
Sorted by
Tagged with
0
votes
0answers
24 views

Return MSK Kafka endpoint in CloudFormation script

I'm using a CloudFormation script to create a MSK cluster in AWS. I want to add a Route53 entry for the cluster in the same CloudFormation script, but for this I need an endpoint for the cluster (...
0
votes
0answers
27 views

Creating RabbitMQ cluster (by Bitnami) and Kafka cluster (by Bitnami) in Azure and connecting to them from Kubernetes in another v-net

I'm trying to create an architecture with: 1. Kafka cluster (by Bitnami) 2. Rabbitmq cluster (by Bitnami) 3. Kubernetes cluster (AKS) that can use both Kafka and RMQ When I created both clusters (...
0
votes
0answers
8 views

Kafka topic creater details

How to find who created a kafka topic. From the zookeeper I can only see the timestamp of the topic. But I want to know who and how topic got created.
0
votes
1answer
25 views

Can I control which terraform instances are discarded when reducing count?

When having a multi-instance provisioning that needs to be reduced - can I control which of the existing resource will be killed? I have a Kafka cluster with variable load, so I would like to provide ...
0
votes
0answers
9 views

Message Forwarder to MapR Streams/Kafka

We have an application that forwards data to MapR Streams (Kafka). Problem is if the network link goes down then data is lost as the application itself doesn't keep a history. We are looking at ...
0
votes
0answers
11 views

Kafka is flaky, timing out on requests

We have a kafka (version 2.2.0) cluster running in k8s. We noticed about a month ago, the CPU suddenly jumped to maximum utilisation, and hasn't dropped since. Since then, kafka is flaky and timing ...
0
votes
1answer
38 views

Expose Confluent Kafka outside of the cluster but inside of the GCP

I've installed Confluent Kafka OSS using their official helm chart: https://github.com/confluentinc/cp-helm-charts However, I would like to consume Kafka topics in a GCP Dataflow so this mean, I ...
0
votes
1answer
31 views

hadoop + can we install zookeeper servers on kafka hosts

we want to dedicated the zookeeper servers only for kafka machines so each kafka machine include the zookeeper server and zookeeper server will serve only the kafka host and not other application in ...
0
votes
0answers
7 views

Jboss spawning multiple kafka processes

I have two jboss instances. One has a problem, I think it's about configuration. Simply put, when I go see the active processes made by my user jbossuser I also see this: jbossusr 20168 1.7 8.7 ...
0
votes
0answers
9 views

When to use Apache Kafka log roll jitter?

There are two log roll jitter related parameters in config doc, i.e., log.roll.jitter.ms/log.roll.jitter.hours. Both of them are marked as high importance, but I have no idea when to use it. What's ...
0
votes
1answer
59 views

Kafka Producer & Consumer hosts

Would Kafka Producer & Consumer hosts can be same. Suppose if I've producers like host1,host2 & host3. Can I use same host1, host2 & host3 as consumers? What are the consequences/...
1
vote
0answers
30 views

Replace value of key in Apache Kafka

I want kafka to only store the latest value of a given key in a topic. This seems so be possible, as shown here. However, the only think I archived so far, are that ALL Key/Value pairs get deleted or ...
0
votes
0answers
23 views

reducing tps in kafka disks on linux

we're using kafka brokers with: 24 HT cores 256GB RAM 16 X 7.2K RPM NLSAS disks configured in RAID-10 and replication factor 3. the sar -q command the runq-sz reached ~40 on average, load average ...
0
votes
0answers
28 views

Kafka log retention not wokring as expected

Note: I am aware that there are many similar questions, already looked at them, didn't help. I am running a kafka benchmark test in a closed environment, I gave each broker a filesystem of 40GB for ...
0
votes
2answers
107 views

kafka scripts in “bin” directory fail with “Address already in use” (running in kubernetes)

I have a kafka set up running in minikube. It's up and running, and I'm able to produce messages into it and consume them back out. However, when I'm logging in to one the brokers (using kubectl exec ...
0
votes
1answer
37 views

Dose zookeeper servers must use SSD disks?

we want to install kafka cluster and 3 zookeeper servers kafka should use the zookeeper servers in order to save the metadata on the zookeeper servers ZK Data and Log files should be on disks, which ...
0
votes
0answers
132 views

How to enable Server host name verification in Kafka 2?

This is my server.properties: listeners=PLAINTEXT://localhost:9092,SSl://localhost:9093 ssl.client.auth=required ssl.keystore.location=/home/xrobot/kafka_2.12-2.1.0/certificate/server.keystore.jks ...
1
vote
0answers
1k views

kafka failed authentication due to: SSL handshake failed

I have to add encryption and authentication with SSL in kafka. This is what I have done: - 1) Generate certificate for each broker kafka: COMANDO: keytool -keystore server.keystore.jks -alias ...
0
votes
0answers
275 views

Kafka broker listing empty nodes list *for a while*

For integration testing purposes I'm creating a very simple single-node Kafka deployment: 1x Zookeeper 1x Kafka 1x Kafka client (e.g. AdminClient, creating topics) (All cleanly deployed in fresh ...
0
votes
1answer
329 views

Kafka multiple listeners

Initial apologies for the long post (this is also on superuser as i wasn't sure the best place for this https://superuser.com/questions/1404421/kafka-multiple-listeners so let me know if one needs ...
0
votes
0answers
27 views

Kafka broker in cluster start/stop sequence

I have a 3 Broker node kafka cluster. What is the right stop and start sequence of brokers in the kafka cluster?
0
votes
0answers
79 views

kafka + what could be the reasons for kafka broker isn't the leader for topic partition

we have HDP cluster - 2.6.4 with ambari 2.6.1 version we have 3 kafka brokers with version 10.1 we saw in the /var/log/kafka/server.log many errors messages as : in this example we have 6601 errors ...
0
votes
1answer
19 views

kafka machines in the cluster and kafka comunications

We have kafka cluster with 3 kafka brokers nodes and 3 zookperes servers kafka version - 10.1 ( hortonworks ) from my understanding since all meta data is located on the zookeeper servers , and ...
0
votes
0answers
152 views

Kafka Metrics monitoring using Prometheus

currently I am monitoring Kafka with Prometheus to which I am new. I have successfully setup Prometheus with nade_exporter, jmx_exporter and kafka_exporter. I want to monitor other metrics given at ...
0
votes
1answer
84 views

How to monitor Kafka Consumer lag in Stackdriver?

I can view all the Kafka broker metrics by using the JMX exporter -> stackdriver monitoring agent and is displayed in the Stackdriver but i cant see any consumer metrics, i am particularly interested ...
0
votes
0answers
12 views

How to configure Apache Kafka so every message in a topic is handled sequentially?

I'm writing a quick test app in Go to wrap my head kafka producers, consumers, topics, and everything kafka. I have to use it at work but I get a lot of control over how it's configured. I have many ...
1
vote
2answers
608 views

ZooKeeper not starting because acceptedEpoch is less than the currentEpoch

I have zookeeper.version=3.4.10, and I see: https://issues.apache.org/jira/browse/ZOOKEEPER-2307 that is however in unresolved status. The problem appeared because of filling up the disk space. Now ...
0
votes
0answers
116 views

Can connect to kafka, but cannot consume

I have a 3 broker kafka cluster set up in an AWS VPC with kubernetes, I am using the https://github.com/Yolean/kubernetes-kafka repo for this. I'm having the strange issue of being able to connect ...
0
votes
1answer
146 views

stackdriver not monitoring kafka metrics

I am trying to setup kafka metrics on stackdriver. I followed all steps as mentioned in https://cloud.google.com/monitoring/agent/plugins/kafka , but still cant see "kafka" metrics. Although I can see ...
3
votes
1answer
51 views

Does Kafka replicate the same data across all brokers?

I am going to implement Kafka in the infrastructure. I have a use case where we have below components in the Kafka architecture. Number of brokers: 5 Replication Factors: 3 Number of topics: 3 ...
0
votes
0answers
40 views

Avoid kafka disk to became 100% used by Cron job

We want to suggest the following based on our issues on kafka disks We have many HDP clusters ( based on ambari , and all machines are redhat version 7.2 ) Each cluster include 3 kafka machines , ...
1
vote
1answer
2k views

Apache Kafka configuration behind NAT/LB

In order to make our Kafka cluster available both from Internet and from our private network, we thought fine to configure Kafka this way : Private VIP:9000 => All brokers:9092 (topology query ...
1
vote
1answer
175 views

How to do data rebalance on kafka if data is stored persistently

I'm new to kafka and preparing use it for production. What strategies can be used for rebalancing data storage if brokers for a topic's current partitions are running out of disk space, if more ...
0
votes
1answer
9 views

What are the minimum permissions needed by Kafka brokers for them to operate?

I'm securing a Kafka cluster. I read and followed the Confluent Kafka Security Tutorial, and it works like a charm. Now, I would like to add ACLs to the brokers in order to remove them from the super ...
1
vote
1answer
112 views

Kafka long coordinator load time and small ISRs

I'm using Kafka 0.8.2.1, running a topic with 200 partitions and RF=3, with log retention set to about 1GB. An unknown event caused the cluster to enter the "coordinator load" or "group load" state. ...
1
vote
2answers
495 views

What is ideal directory for kafka messages

The default value of logs.dir=/tmp/kafka-logsin server.properties. Usually /tmp is avoided from keeping any important files and we are storing messages and offsets! Any particular reason why one may ...
0
votes
1answer
959 views

kafka with ACL fails to connect zk and stops

I have a kafka installation (with ssl listener and ssl client authentication). It works fine until I try to add ACL. this is the error (full error in the bottom): [edit-1] the first error on kafka ...
2
votes
1answer
117 views

Determine a Kafka broker's ID from JMX

Given a broker host/ip is there a way to interrogate it to learn the broker ID other than by querying ZK? I'm trying to map the IP to the broker ID for monitoring. ZK is authenticated and I don't have ...
0
votes
2answers
348 views

Kafka cluster: Withstanding a network outage

We have a kafka cluster. And a network. Yay. The network will be unavailable across all racks in our data center for 5-10 minutes (!) because maintenance requires it. I'm concerned that is too long an ...
0
votes
1answer
295 views

Is it possible to get Apache Kafka to present an SSL intermediate certificate?

After reading this documentation: http://kafka.apache.org/documentation.html#security_ssl I was able to set up Kafka using SSL with a self-signed certificate. I tried setting it up with a Digicert ...
0
votes
2answers
530 views

Mongodb primary node not Resolved in Debezium Kafka Connect — Docker Compose

i have this docker compose file, that contains bitnami/mongodb containers for creating a replica set. and I've used zookeeper, kafka & debezium-connector for monitoring my mongodb replica set. ...
1
vote
1answer
147 views

How to perform bulk topic deletion in kafka

We have 3 node Kafka cluster with replication factor 3 and default log retention period of 4 days. We have enabled the auto topic creation and It will create topics for each of our tasks. Once the ...
0
votes
1answer
133 views

How to return the original log retention after purge kafka topics

We have 3 kafka machines version - 0.10.0.2.6 We want to purge all data files from all kafka's topics ( purge completely all data ) The CLI for this action should be ( should be run from the zoo ...
1
vote
0answers
259 views

What is the attribute you need to specify for kafka Log start offset metric?

I have an environment where logstash (v6.2.3) is connecting kafka_2.11 cluster (v.1.1.0) using kafka input. What I want to achieve is fetch consumer offset information using kafka-consumer-group.sh . ...
1
vote
1answer
164 views

kafka + how to revert topic deletion

just in case we delete the wrong topic as /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper hdpmaster01:2181 --delete --topic gtom.poli.pri.proc Topic gtom.poli.pri.procis already ...
0
votes
1answer
802 views

what is the safe and best way to delete the kafka topic folders

on all our kafka machines ( production machines ) , we see that: ( no free space ) df -h /var/kafka Filesystem Size Used Avail Use% Mounted on /dev/sdb 11T 11T 2.3M 100% /var/kafka ...
3
votes
1answer
383 views

Running Kafka in Kubernetes on AWS

I've the following scenario: 3 nodes instances spread across 3 Availability Zones 6 Brokers running in StatefulSets with AWS external volume for Kafka data. Instance size: m4.2xlarge EBS Volume: st1 ...
4
votes
1answer
646 views

Kafka compatible with Zookeeper 3.5 feature 'Rebalancing Client Connections'

In this document https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html dynamic configuration functionality is described for Zookeeper 3.5. There are 2 important points in this document: ...
2
votes
0answers
1k views

kafka docker - ERROR when sending a message

I would like to implement kafka in a docker container. I use the official zookeper instance for kafka as docker container, too. When I want to write a message about the terminal, on my linux mint 18....
1
vote
0answers
61 views

Operation APPEND failed with HTTP500?

package org.apache.spark.examples.kafkaToflink; import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.OutputStream; import java.io.PrintStream; import java.nio.charset....