Questions tagged [kafka]
Kafka is a distributed messaging queue application written in java.
107
questions
0
votes
0
answers
26
views
nginx + php-fpm + rdkafka = SIGSEGV (mac m1)
I’m facing a very strange issue that I can’t seem to understand or resolve.
Here’s the setup:
MacOS Sonoma (M1)
Tried different versions and configurations of Nginx
PHP 8.2/8.3/8.4 (the issue occurs ...
- 101
0
votes
0
answers
29
views
kafka + undertsanding better the kafka controller.log logs
we have cluster with 12 Kafka machines and 3 zookeeper servers on Linux servers
Kafka version is 2.7 version , ( broker and controller are co-hosted on the same PID )
as known Kafka have 2 important ...
- 697
3
votes
0
answers
76
views
why is my kafka cluster under such a high load for only 1.2K clients, 3K message/s, max bandwidth 25Mb/s
I have a single cloud server, the spec:
Intel Xeon 4 core 3GHz Cpu, 16G memory
I've run a kafka cluster(3 kafka instances + zookeeper) via docker compose on it(docker images were pulled at early ...
- 141
1
vote
1
answer
98
views
does my kafka cluster need expansion and how?
I've run a kafka cluster(3 kafka instances + zookeeper) via docker compose on my server recently(July 2024) with all latest tag for docker images.
I've setup jmx_exporter and node-exporter for ...
- 141
0
votes
0
answers
55
views
What is the proper way to remove a brokers from a kraft cluster?
before couple month's we installed in production environment 34 Kafka Kraft machines with version - apache 3.5 , all machines are working well in the cluster , when topics are created with replica 3 ...
- 697
1
vote
1
answer
400
views
dmesg + many messages as error count since last fsck
we have kafka cluster with 3 VM machines. , when each kafka machines use the sdb disk ( VMDK disk ) in order to store the data
on all machines we seen the following kernel messages
[1123783.849575] ...
- 697
1
vote
1
answer
590
views
Apache Kafka Single Node Docker- how to set listener to docker hostname
I am using the Apache Kafka 3.7.0 docker image as outlined here, with some minor tweaks:
docker run --network=internal --name my-kafka apache/kafka:3.7.0
My intention is to only access this within the ...
- 121
0
votes
0
answers
62
views
Can Kafka handle cloud-scale load without horizontal sharding?
Per this kadek.com article,
Kafka partition limits
There are no hard limits on the number of partitions in Kafka
clusters. But here are a few general rules:
maximum 4000 partitions per broker (in ...
- 463
1
vote
0
answers
126
views
why apache Kafka documentation not gives the ability to use Kafka over NFS as Netapp
We read the latest apache documentation as https://kafka.apache.org/35/documentation.html
And what is interesting is that , documentation not mentioned the option to use storage over NFS As Netapp or ...
- 697
0
votes
0
answers
154
views
Apache Kafka Kraft best practices and what is the preferred disk storage for the meta-data topic
My question concerns the recommended disk storage for the topic __cluster_metadata-0 under controllers in Kraft mode.
we have two options:
Locate the topic __cluster_metadata-0 , on OS volume as /...
- 697
0
votes
0
answers
656
views
How to solve strange error in Kafka
I try to run Kafka with SASL_SSL. When I run Kafka server, it log strange error in server log:
[2023-12-19 01:57:09,297] INFO [Controller id=0, targetBrokerId=0] Client requested connection close from ...
- 101
0
votes
1
answer
128
views
Error in starting Kafka server
I tried to make kafka with sasl_ssl. I make some configuration, but Kafka server fail on some issue, which I dont know how to solve. I use Kafka 2.13-3.6.0.
My zookeeper.properties
quorum.auth....
- 101
0
votes
0
answers
42
views
Kafka Scripts in bin Directory throws an error is "java.net.BindException: Address in use (Bind failed)"
I have Deployed Kafka as a Stateful Set on k8's and the Service type is ClusterIP. It's Up & running.
However, when I'm logging into Kafka brokers (using kubectl exec -it pod_name -- bash) and go ...
- 1
0
votes
0
answers
128
views
kafka + what is the meaning of Log directory is offline
our Kafka production cluster include 23 broker machines , and each broker include 35 JBOD disks
broker version is - apache 2.7 version , and cluster include 5 zookeeper servers
from /var/log/kafka/...
- 697
0
votes
0
answers
287
views
Rsyslog output to Kafka Timing Out
I'm having trouble setting up Rsyslog output to Kafka using the omkafka module. I have a robust Kafka cluster running in a production environment. In the current workflow, Rsyslog receives events on ...
0
votes
1
answer
585
views
Kafka REST Proxy: connect only one consumer
I have Kafka topic "purchase2" with 2 partion and Kafka REST Proxy for consumers (from this toturial ). When I run the first consumer for this topic, it all right:
curl -X POST -H &...
- 1
0
votes
1
answer
345
views
Kafka producer looses queued messages when broker restarts
I have no much experience on the subject and maybe doing something wrong.
The problem occurs when I stop Kafka broker and after a pause start it again.
Some seconds after broker starts my producer ...
- 1
1
vote
0
answers
55
views
Kafka not stopping consumers when deleting group
I am sending logs from Logstash as the producer to Kafka and then consuming them from Graylog. This works as expected.
However, when stopping the Graylog input, thus sending a LeaveGroup request, the ...
- 143
0
votes
0
answers
1k
views
Kafka Issue : Request joining group due to: group is already rebalancing
The pod is not getting restarted, No ongoing rebalance to finish consumer group experiences no frequent changes, but still rebalancing issue occurs. is there anything wrong with the configuration?
...
- 101
1
vote
1
answer
75
views
Display metric for multiple servers on a single grafana panel, using a Snowflake datasource populated from Kafka
I have a collection of servers which submit metrics to a single kafka topic. This data is stored as a row in a snowflake database in this format:
{
"disk_util": 47.7,
"location"...
- 121
1
vote
1
answer
830
views
How to forward port to connect multiple MSK Kafka brokers through bastion?
I have MSK Kafka cluster on AWS with 3 brokers: b1.amazonaws.com:9092, b2.amazonaws.com:9092, b3.amazonaws.com:9092. Brokers are not public, so I have opened 3 tunnels through bastion:
ssh -L 9092:b1....
- 111
0
votes
1
answer
592
views
Rsyslog server - conditionally forward logs
The idea is to receive the logs on the Rsyslog server and then send them to a specific Kafka server depending on the log contents.
For example,
Logs are being received on a single port, 514
Forwarding ...
- 15
1
vote
1
answer
533
views
kafka cluster RAID vs JBOD with new hardware and kafka versions
we want to design production kafka cluster with 7 kafka machines , on RHEL OS
kafka version should be - 2.7
usually we are installing the kafka machine/s disks as RAID 10 ( as confluent recommended )
...
- 697
1
vote
1
answer
109
views
Handle 150 req/s and 2TB of logs for 7 days of retention available by API
I would like to get advise from experienced people to build a HA infrastructure to log 2To of data in JSON format every week.
I need to have a retention time of 7 days and need to be able to requests ...
- 11
3
votes
1
answer
913
views
Page cache in Kafka
We are receiving alerts for kafka memory utilization in our kubernetes cluster and analysing it seems that most of the memory is page cache. I am aware that kafka uses loads of page cache for IO ...
- 41
0
votes
0
answers
1k
views
Ansible playbook to post message into kafka topic
Playbook 1:
---
- name: Message into topic
hosts: web1
become: yes
tasks:
- name: post message
expect:
shell: "/usr/local/kafka/bin/kafka-console-producer.sh --broker-list ...
- 1
2
votes
1
answer
2k
views
No connection to Kafka from Faust client
I am having a hard time connection to a machine running Kafka from a client running a Faust script.The script looks like this:
import faust
import logging
from asyncio import sleep
class Test(faust....
- 202
0
votes
0
answers
1k
views
Kafka: monitoring length of queues / topics
We have several producers and several consumers connected via Kafka - basically, it's batch processing jobs that are created on demand, placed on Kafka in several queues, and the batch processors pick ...
- 1,277
5
votes
1
answer
3k
views
Kafka stores log4j logs in directory literally called ${kafka.logs.dir}
I installed a preview version of Kafka 3.0 with log4j2 support (http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/) on a RHEL 8 server. Kafka and Zookeeper are running successfully as ...
1
vote
1
answer
3k
views
Kafka - Get Ip address or hostnames connected to the cluster
I have a test Kafka Cluster in AWS MSK with three brokers. I'd like to know how to get information about who is connecting to the cluster either to produce or consume messages.
For instance, MSSQL ...
- 133
0
votes
0
answers
1k
views
How to remove escape character when using kafka console consumer?
For compliance purposes, I have to save all kafka raw documents and keep them one year. To do this, I use the following :
kafka-console-consumer.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:...
- 1
0
votes
1
answer
267
views
Kafka + Query the detailed configuration of all Topics
we have Kafka servers cluster with Kafka version 2.6 on Linux RHEL machines
machine in the cluster are - kafka1,kafka2,kafka3
usually we are use the following cli in order to get the detailed ...
- 697
1
vote
1
answer
1k
views
zookeeper ssl alert "alert bad certificate"
I am using Kafka (version 2.3.0) and Zookeeper (version 3.5.5-3) - the stable version is 3.6.3.
When I test the SSL of my Zookeeper using this command:
openssl s_client -showcerts -connect 127.0.0.1:...
- 13
1
vote
0
answers
113
views
Kafka installation with Confluent vs Apache Kafka [closed]
for now we have HDP 2.6.5 version , with Kafka cluster with Kafka version 0.1
all Kafka machines are installed on RHEL version 7.6
all machines with RHEL 7.6 are on DELL physical machines
we want to ...
- 697
0
votes
1
answer
2k
views
Kafka Broker Runs locally via docker compose but fails in kubernetes with near identical config
Kafka broker runs beautifully when running locally via docker-compose.
version: "3"
services:
zookeeper:
image: 'bitnami/zookeeper:latest'
ports:
- '2181:2181'
...
- 21
0
votes
1
answer
1k
views
How to manage storage / persistence overflow in Bitnami Kafka Helm chart?
I deploy Bitanmi Kafka Helm chart on AWS provisioned with Terraform. I find the documentation storage and persistence allocation, very confusing. From what I understood from the documentation; logs ...
- 101
0
votes
1
answer
592
views
How to rotate compressed log file on centos 7?
I have customized kafka server and the app is creating log files and configures log rotation. My problem is that I need to compress the rotation logs that were created.
I tried to create new log ...
- 1
1
vote
2
answers
980
views
The Debezium Kafka based KAFKA_ADVERTISED_LISTENERS property setting is ignored
I have a Docker Compose based Swarm with a Zookeeper service that starts fine, but the Kafka service cannot start.
It keeps using the default KAFKA_ADVERTISED_LISTENERS property:
debezium_kafka.1....
- 227
1
vote
1
answer
400
views
PUB/SUB Message Queue with server side filtering?
I am looking for a message queue to realize a chat application.
The most important requirement is: when a subscriber specifies a filter criteria, all the messages need to be filtered on the server ...
- 11
0
votes
1
answer
211
views
How to implement a secured kafka environment with a public interface?
Scenario: Kafka cluster, secured with SSL encryption and SASL/PLAIN authentication, resides in an AWS private subnet within a dedicated VPC. Within the private subnet, all is fine. I use self-...
- 3
1
vote
0
answers
479
views
How can I further analyze ram usage in a possible leak? [debian@HyperV + Kafka + Zookeeper]
We have a 3-box cluster Kafka+Zookeeper, in a virtualized enviroment (in Windows 2019 HyperV).
In the machines' config dynamic memory is disabled, and they have 4G RAM each (I know it's a very few ...
- 11
1
vote
0
answers
954
views
Confluent Kafka Connect S3 re-processing data due to offset out of range
I'm experiencing a strange situation.
My Confluent Kafka Connect S3 is processing around 8K messages/sec. What started happening randomly every few days is:
The Fetcher class would try to fetch an ...
- 483
0
votes
1
answer
2k
views
How set a fixed private IP for ECS containers?
I have a Task Definition called kafka-zookeeper, it has an Apache Kafka broker and a Zookeeper node in it. Those containers need to communicate with each other (that's easy) and with the kafka and ...
- 171
0
votes
1
answer
2k
views
Filebeat kafka input with SASL?
I'm trying to get filebeat to consume messages from kafka using the kafka input. I'm unable to authenticate with SASL for some reason and I'm not sure why that is. The documentation for both Kafka and ...
- 155
16
votes
9
answers
39k
views
Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)
Below the steps I did to get this issue :
Launch ZooKeeper
Launch Kafka : .\bin\windows\kafka-server-start.bat .\config\server.properties
Also note that my OS is Windows
And at the second step ...
- 261
0
votes
1
answer
427
views
kafka in kubernetes pod
I am new at working with kubernetes setup, and are trying to run kafka pod with a persistence volume,
so in case a pod goes down, the memory is not lost, and I can spin up a new cluster, using the ...
- 1
0
votes
1
answer
1k
views
mkfs + xfs + what is the right mkfs cli in order to create xfs file-system on huge disk
We need to create xfs file-system on kafka disk
The special thing about kafka disk is the disk size
kafka disk have 20TB size in our case
I not sure about the following mkfs , but I need advice to ...
- 511
0
votes
0
answers
3k
views
zookeeper server due unable to load database on disk
We have kafka cluster with 3 nodes , each kafka include zookeeper server and schema registry
We get the following error on one of the zookeeper server
[2019-11-12 07:44:20,719] ERROR Unable to load ...
- 511
2
votes
1
answer
620
views
Linux + docker container folder that mounted to OS volume + how to know the real limited size
inside the docker-compose.yml we configured the following volume
image: confluentinc/cp-kafka:latest
volumes:
- /grid/kafka-data:/var/lib/kafka/data
.
docker-compose ps
Name ...
- 511
1
vote
0
answers
1k
views
Getting error INFO [SocketServer brokerId=3] Failed authentication with /10.2.***.* (SSL handshake failed) (org.apache.kafka.common.network.Selector)
I have kafka multibroker deployed on kubernetes, with interbroker SSL communication, I have set the folowing properties:
advertised.listeners=SSL://kafka-1.**.***.com:9093,PLAINTEXT://kafka-1.**.*****...
- 113
Related Tags
linux × 19
zookeeper × 14
kubernetes × 10
hard-drive × 5
hadoop × 5
monitoring × 5
docker × 4
nginx × 4
rsyslog × 4
ssl × 4
java × 3
google-cloud-platform
centos × 3
collectd × 3
bitnami × 3
logging × 3
networking × 3
redhat × 3
scalability × 3
azure × 2
graylog × 2
hostname × 2
helm × 2
more related tags
Hot Network Questions
- 3D Chip Design using TikZ
- Some of them "have no hair"
- What are some limitations of this learning method?
- Trinitarian Christianity says Jesus was fully God and Fully man. Did Jesus (the man) know this to be the case?
- How to assign a definition locally?
- Why the simulator should be a PPT in simulation-based security?
- How to add Z-axis to analog oscilloscope?
- The graph of a continuous function is a topological manifold
- mp4 not opening in Ubuntu 24.04.1 LTS
- How do exchange mail enabled security groups handle emails to multiple groups with overlapping members?
- How to avoid repeated input while using newcommand?
- Is it possible to make sand from bones ? Would it have the same properties as regular sand?
- Determining Entropy in PHP
- What early 60s puppet show similar to fireball XL5 used the phrase "Meson Power?"
- How to achieve 24t-38t front chainrings
- Writing in first person for fiction novel, how to portray her inner dialogue and drag it out to make a chapter long enough?
- Fundamental Sampling Theorem
- Analytic continuation gives a covering space (and not just a local homeomorphism)
- Cheapest / Most efficient way for a human Wizard to not age?
- My one-liner 'delete old files' command finds the right files but will not delete them
- Why did mire/bog skis fall out of use?
- Cutting a curve through a thick timber without waste
- Would a material that could absorb 99.5% of light be able to protect someone from Night Vision?
- Apple IIgs to VGA adapter