Questions tagged [kafka]

Kafka is a distributed messaging queue application written in java.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

Kafka Connect cluster showing erratic behaviour

We have a running KafkaConnect cluster (Strimzi distribution), deployed in an Openshift (Kubernetes for the matter) cluster that is showing an erratic behaviour. The REST API of Kafka Connect is ...
  • 101
0 votes
1 answer
39 views

Rsyslog server - conditionally forward logs

The idea is to receive the logs on the Rsyslog server and then send them to a specific Kafka server depending on the log contents. For example, Logs are being received on a single port, 514 Forwarding ...
  • 13
0 votes
0 answers
7 views

Where to run Kafka Streams?

I'm playing around with Apache Kafka a bit and have a functional multi-node cluster configured. I want to now introduce a Kafka Stream Processor. I'll just do something simple, but here's my ...
0 votes
0 answers
30 views

Separate Apache Kafka clusters unreachable at the same time - kafka_network_socketserver_networkprocessoravgidlepercent goes to zero

We have 4 Kafka clusters: ENV1: 6 brokers and 3 zookeepers ENV2: 6 brokers and 3 zookeepers ENV3: 8 brokers (on 2 DCs, 4-4 brokers) and 9 zookeepers (on 3 DCs, 3-3-3 nodes) ENV4: 16 brokers (on 2 DCs, ...
  • 1
0 votes
1 answer
51 views

kafka cluster RAID vs JBOD with new hardware and kafka versions

we want to design production kafka cluster with 7 kafka machines , on RHEL OS kafka version should be - 2.7 usually we are installing the kafka machine/s disks as RAID 10 ( as confluent recommended ) ...
0 votes
1 answer
33 views

Handle 150 req/s and 2TB of logs for 7 days of retention available by API

I would like to get advise from experienced people to build a HA infrastructure to log 2To of data in JSON format every week. I need to have a retention time of 7 days and need to be able to requests ...
0 votes
0 answers
425 views

How to get Kafka consumer metrics with JMX exporter for Prometheus?

Kafka version: 3.1 I didn't find a way to get Kafka consumer metrics as described in Kafka documentation from my Kafka servers. Even if I leave the "rules" section empty in the config of the ...
0 votes
0 answers
207 views

How do I optimize Kafka Consumer and should I use Apache Spark?

I have written a Kafka consumer to consume encrypted data(~1MB) stream and decrypt them before adding them to the S3 bucket. It takes ~20mins to process 1000 records, and if I remove the encryption ...
1 vote
1 answer
221 views

Page cache in Kafka

We are receiving alerts for kafka memory utilization in our kubernetes cluster and analysing it seems that most of the memory is page cache. I am aware that kafka uses loads of page cache for IO ...
0 votes
0 answers
123 views

kafka + File handle leak due a large number of pipe/eventpoll leak

we have kafka cluster on linux rhel version - 7.6 we notice too many open files on kafka machines but when we did lsof >/tmp/res_file form file /tmp/res_file , we can see many lines as java ...
0 votes
0 answers
21 views

Kafka brokers and applications on same server

I have 3 servers available with me on which I am running a 3 node kafka cluster and also the kafka streams applications that consume from kafka and rewrite to kafka. Can this kind of setup where ...
  • 101
0 votes
0 answers
299 views

Ansible playbook to post message into kafka topic

Playbook 1: --- - name: Message into topic hosts: web1 become: yes tasks: - name: post message expect: shell: "/usr/local/kafka/bin/kafka-console-producer.sh --broker-list ...
1 vote
1 answer
622 views

No connection to Kafka from Faust client

I am having a hard time connection to a machine running Kafka from a client running a Faust script.The script looks like this: import faust import logging from asyncio import sleep class Test(faust....
0 votes
0 answers
356 views

Kafka: monitoring length of queues / topics

We have several producers and several consumers connected via Kafka - basically, it's batch processing jobs that are created on demand, placed on Kafka in several queues, and the batch processors pick ...
3 votes
2 answers
677 views

Kafka stores log4j logs in directory literally called ${kafka.logs.dir}

I installed a preview version of Kafka 3.0 with log4j2 support (http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/) on a RHEL 8 server. Kafka and Zookeeper are running successfully as ...
0 votes
0 answers
156 views

kafka + Leader none + and kafka broker id not signed in zookeeper

we have 3 Kafka brokers on Linux RHEL 7.6 ( 3 linux machines ) kafka version is 2.7.X brokers ID's are - 1010,1011,1012 from kafka described we can see the following Topic: __consumer_offsets ...
0 votes
1 answer
801 views

Kafka - Get Ip address or hostnames connected to the cluster

I have a test Kafka Cluster in AWS MSK with three brokers. I'd like to know how to get information about who is connecting to the cluster either to produce or consume messages. For instance, MSSQL ...
0 votes
0 answers
59 views

Issue with syncing data from Amazon RDS to Amazon S3

I am trying to sync binlogs in RDS mysql DB with Amazon S3 in real time to query using Amazon Athena. I have tried using debezium as source connector for Apache Kafka connect- kafka stores streaming ...
0 votes
0 answers
405 views

How to remove escape character when using kafka console consumer?

For compliance purposes, I have to save all kafka raw documents and keep them one year. To do this, I use the following : kafka-console-consumer.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:...
0 votes
1 answer
122 views

Kafka + Query the detailed configuration of all Topics

we have Kafka servers cluster with Kafka version 2.6 on Linux RHEL machines machine in the cluster are - kafka1,kafka2,kafka3 usually we are use the following cli in order to get the detailed ...
1 vote
1 answer
452 views

zookeeper ssl alert "alert bad certificate"

I am using Kafka (version 2.3.0) and Zookeeper (version 3.5.5-3) - the stable version is 3.6.3. When I test the SSL of my Zookeeper using this command: openssl s_client -showcerts -connect 127.0.0.1:...
1 vote
0 answers
100 views

Kafka installation with Confluent vs Apache Kafka [closed]

for now we have HDP 2.6.5 version , with Kafka cluster with Kafka version 0.1 all Kafka machines are installed on RHEL version 7.6 all machines with RHEL 7.6 are on DELL physical machines we want to ...
0 votes
1 answer
932 views

Kafka Broker Runs locally via docker compose but fails in kubernetes with near identical config

Kafka broker runs beautifully when running locally via docker-compose. version: "3" services: zookeeper: image: 'bitnami/zookeeper:latest' ports: - '2181:2181' ...
  • 11
0 votes
1 answer
576 views

How to manage storage / persistence overflow in Bitnami Kafka Helm chart?

I deploy Bitanmi Kafka Helm chart on AWS provisioned with Terraform. I find the documentation storage and persistence allocation, very confusing. From what I understood from the documentation; logs ...
0 votes
1 answer
259 views

How to rotate compressed log file on centos 7?

I have customized kafka server and the app is creating log files and configures log rotation. My problem is that I need to compress the rotation logs that were created. I tried to create new log ...
  • 1
1 vote
2 answers
505 views

The Debezium Kafka based KAFKA_ADVERTISED_LISTENERS property setting is ignored

I have a Docker Compose based Swarm with a Zookeeper service that starts fine, but the Kafka service cannot start. It keeps using the default KAFKA_ADVERTISED_LISTENERS property: debezium_kafka.1....
  • 227
1 vote
1 answer
211 views

PUB/SUB Message Queue with server side filtering?

I am looking for a message queue to realize a chat application. The most important requirement is: when a subscriber specifies a filter criteria, all the messages need to be filtered on the server ...
0 votes
1 answer
170 views

How to implement a secured kafka environment with a public interface?

Scenario: Kafka cluster, secured with SSL encryption and SASL/PLAIN authentication, resides in an AWS private subnet within a dedicated VPC. Within the private subnet, all is fine. I use self-...
  • 3
1 vote
0 answers
263 views

How can I further analyze ram usage in a possible leak? [debian@HyperV + Kafka + Zookeeper]

We have a 3-box cluster Kafka+Zookeeper, in a virtualized enviroment (in Windows 2019 HyperV). In the machines' config dynamic memory is disabled, and they have 4G RAM each (I know it's a very few ...
  • 11
1 vote
0 answers
695 views

Confluent Kafka Connect S3 re-processing data due to offset out of range

I'm experiencing a strange situation. My Confluent Kafka Connect S3 is processing around 8K messages/sec. What started happening randomly every few days is: The Fetcher class would try to fetch an ...
0 votes
1 answer
255 views

How set a fixed private IP for ECS containers?

I have a Task Definition called kafka-zookeeper, it has an Apache Kafka broker and a Zookeeper node in it. Those containers need to communicate with each other (that's easy) and with the kafka and ...
0 votes
1 answer
1k views

Filebeat kafka input with SASL?

I'm trying to get filebeat to consume messages from kafka using the kafka input. I'm unable to authenticate with SASL for some reason and I'm not sure why that is. The documentation for both Kafka and ...
  • 155
16 votes
9 answers
26k views

Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)

Below the steps I did to get this issue : Launch ZooKeeper Launch Kafka : .\bin\windows\kafka-server-start.bat .\config\server.properties Also note that my OS is Windows And at the second step ...
0 votes
1 answer
302 views

kafka in kubernetes pod

I am new at working with kubernetes setup, and are trying to run kafka pod with a persistence volume, so in case a pod goes down, the memory is not lost, and I can spin up a new cluster, using the ...
  • 1
0 votes
1 answer
499 views

mkfs + xfs + what is the right mkfs cli in order to create xfs file-system on huge disk

We need to create xfs file-system on kafka disk The special thing about kafka disk is the disk size kafka disk have 20TB size in our case I not sure about the following mkfs , but I need advice to ...
  • 451
0 votes
0 answers
2k views

zookeeper server due unable to load database on disk

We have kafka cluster with 3 nodes , each kafka include zookeeper server and schema registry We get the following error on one of the zookeeper server [2019-11-12 07:44:20,719] ERROR Unable to load ...
  • 451
2 votes
1 answer
415 views

Linux + docker container folder that mounted to OS volume + how to know the real limited size

inside the docker-compose.yml we configured the following volume image: confluentinc/cp-kafka:latest volumes: - /grid/kafka-data:/var/lib/kafka/data . docker-compose ps Name ...
  • 451
1 vote
0 answers
1k views

Getting error INFO [SocketServer brokerId=3] Failed authentication with /10.2.***.* (SSL handshake failed) (org.apache.kafka.common.network.Selector)

I have kafka multibroker deployed on kubernetes, with interbroker SSL communication, I have set the folowing properties: advertised.listeners=SSL://kafka-1.**.***.com:9093,PLAINTEXT://kafka-1.**.*****...
  • 113
0 votes
1 answer
120 views

Can logstash be used as a kafka forwarder?

I am looking at the plugins for Logstash and I noticed it had both an input and output plugin for kafka. For a single topic, can Logstash be used to forward kafka topic messages to another kafka ...
0 votes
1 answer
139 views

Way to forward remote internal network ip and port to same ip and port on local mac

For development reasons I want to connect my local machine (macos) to a remote private network of a kafka/zookeeper cluster. When I do port forwarding with ssh root@000.000.000.000 -N -L 10.0.0.8:...
  • 299
3 votes
2 answers
2k views

Enable logical replication on Google Cloud Postgres

Is there anyway to enable logical replication wal_level = logical on Google Cloud Postgres instance? I want to do CDC (change data capture) to stream WAL Record to Apache Kafka. Thanks for your help!
  • 133
0 votes
1 answer
291 views

Creating RabbitMQ cluster (by Bitnami) and Kafka cluster (by Bitnami) in Azure and connecting to them from Kubernetes in another v-net

I'm trying to create an architecture with: 1. Kafka cluster (by Bitnami) 2. Rabbitmq cluster (by Bitnami) 3. Kubernetes cluster (AKS) that can use both Kafka and RMQ When I created both clusters (...
0 votes
1 answer
94 views

Can I control which terraform instances are discarded when reducing count?

When having a multi-instance provisioning that needs to be reduced - can I control which of the existing resource will be killed? I have a Kafka cluster with variable load, so I would like to provide ...
  • 3
0 votes
0 answers
17 views

Message Forwarder to MapR Streams/Kafka

We have an application that forwards data to MapR Streams (Kafka). Problem is if the network link goes down then data is lost as the application itself doesn't keep a history. We are looking at ...
  • 285
0 votes
1 answer
159 views

Expose Confluent Kafka outside of the cluster but inside of the GCP

I've installed Confluent Kafka OSS using their official helm chart: https://github.com/confluentinc/cp-helm-charts However, I would like to consume Kafka topics in a GCP Dataflow so this mean, I ...
0 votes
1 answer
302 views

hadoop + can we install zookeeper servers on kafka hosts

we want to dedicated the zookeeper servers only for kafka machines so each kafka machine include the zookeeper server and zookeeper server will serve only the kafka host and not other application in ...
  • 451
1 vote
1 answer
430 views

When to use Apache Kafka log roll jitter?

There are two log roll jitter related parameters in config doc, i.e., log.roll.jitter.ms/log.roll.jitter.hours. Both of them are marked as high importance, but I have no idea when to use it. What's ...
  • 2,200
0 votes
1 answer
104 views

Kafka Producer & Consumer hosts

Would Kafka Producer & Consumer hosts can be same. Suppose if I've producers like host1,host2 & host3. Can I use same host1, host2 & host3 as consumers? What are the consequences/...
  • 135
1 vote
0 answers
434 views

Replace value of key in Apache Kafka

I want kafka to only store the latest value of a given key in a topic. This seems so be possible, as shown here. However, the only think I archived so far, are that ALL Key/Value pairs get deleted or ...
  • 11
0 votes
0 answers
62 views

reducing tps in kafka disks on linux

we're using kafka brokers with: 24 HT cores 256GB RAM 16 X 7.2K RPM NLSAS disks configured in RAID-10 and replication factor 3. the sar -q command the runq-sz reached ~40 on average, load average ...