Questions tagged [kafka]
Kafka is a distributed messaging queue application written in java.
99
questions
0
votes
1
answer
24
views
Kafka REST Proxy: connect only one consumer
I have Kafka topic "purchase2" with 2 partion and Kafka REST Proxy for consumers (from this toturial ). When I run the first consumer for this topic, it all right:
curl -X POST -H &...
- 1
0
votes
0
answers
26
views
kafka kraft configuration + dedicated controllers machines
We are preparing Huge Kafka Kraft cluster
Installation is on 35 physical Linux machines and 3 dedicated controllers on 3 VM Linux machines
each broker machine include:
512G RAM memory
64 CORE's
...
- 559
0
votes
0
answers
54
views
Kafka producer looses queued messages when broker restarts
I have no much experience on the subject and maybe doing something wrong.
The problem occurs when I stop Kafka broker and after a pause start it again.
Some seconds after broker starts my producer ...
- 1
1
vote
0
answers
26
views
Kafka not stopping consumers when deleting group
I am sending logs from Logstash as the producer to Kafka and then consuming them from Graylog. This works as expected.
However, when stopping the Graylog input, thus sending a LeaveGroup request, the ...
- 143
0
votes
0
answers
51
views
production kafka clusters + How to optimize Page Cache
one of the importance things in Kafka production cluster is the page cache
here is a good explains what is page cache
PageCache is a typical read/write cache. The operating system uses the free ...
- 559
0
votes
0
answers
424
views
Kafka Issue : Request joining group due to: group is already rebalancing
The pod is not getting restarted, No ongoing rebalance to finish consumer group experiences no frequent changes, but still rebalancing issue occurs. is there anything wrong with the configuration?
...
- 101
0
votes
0
answers
20
views
Expose MSK serverless cluster to internet
I followed the https://docs.aws.amazon.com/msk/latest/developerguide/public-access.html guide, but the Properties → Network settings → Edit public access option doesn't appear at all. Could it be that ...
- 101
1
vote
1
answer
57
views
Display metric for multiple servers on a single grafana panel, using a Snowflake datasource populated from Kafka
I have a collection of servers which submit metrics to a single kafka topic. This data is stored as a row in a snowflake database in this format:
{
"disk_util": 47.7,
"location"...
- 121
0
votes
0
answers
39
views
Very old segments in Kafka
I have set my segment retention via (log.retention.ms) for topic to 2h. However I still have a lot of old messages in the message queue and some of the segment data files (data/topic-0/...
- 870
0
votes
0
answers
36
views
Vertical Scaling resources (CPU, RAM) for Kafka Production cluster
Currently we are using three node kafka Cluster in Prod with hardware specifications : Zookeeper : CPU 2 Core Ram 4GB Kafka : CPU 4 Core Ram 8GB
We want to vertically scale our Kafka cluster to double ...
0
votes
0
answers
23
views
Can Apache MirrorMaker2 run on only one server?
I have a couple of test clusters (3 nodes in each) and I’m trying to configure them such that one runs in one country and another in another country (hypothetically).
I have read that remote consume ...
1
vote
1
answer
452
views
How to forward port to connect multiple MSK Kafka brokers through bastion?
I have MSK Kafka cluster on AWS with 3 brokers: b1.amazonaws.com:9092, b2.amazonaws.com:9092, b3.amazonaws.com:9092. Brokers are not public, so I have opened 3 tunnels through bastion:
ssh -L 9092:b1....
- 111
0
votes
0
answers
123
views
Kafka Connect cluster showing erratic behaviour
We have a running KafkaConnect cluster (Strimzi distribution), deployed in an Openshift (Kubernetes for the matter) cluster that
is showing an erratic behaviour.
The REST API of Kafka Connect is ...
- 101
0
votes
1
answer
389
views
Rsyslog server - conditionally forward logs
The idea is to receive the logs on the Rsyslog server and then send them to a specific Kafka server depending on the log contents.
For example,
Logs are being received on a single port, 514
Forwarding ...
- 15
1
vote
1
answer
276
views
kafka cluster RAID vs JBOD with new hardware and kafka versions
we want to design production kafka cluster with 7 kafka machines , on RHEL OS
kafka version should be - 2.7
usually we are installing the kafka machine/s disks as RAID 10 ( as confluent recommended )
...
- 559
1
vote
1
answer
85
views
Handle 150 req/s and 2TB of logs for 7 days of retention available by API
I would like to get advise from experienced people to build a HA infrastructure to log 2To of data in JSON format every week.
I need to have a retention time of 7 days and need to be able to requests ...
- 11
2
votes
1
answer
592
views
Page cache in Kafka
We are receiving alerts for kafka memory utilization in our kubernetes cluster and analysing it seems that most of the memory is page cache. I am aware that kafka uses loads of page cache for IO ...
- 31
0
votes
0
answers
760
views
Ansible playbook to post message into kafka topic
Playbook 1:
---
- name: Message into topic
hosts: web1
become: yes
tasks:
- name: post message
expect:
shell: "/usr/local/kafka/bin/kafka-console-producer.sh --broker-list ...
- 1
2
votes
1
answer
2k
views
No connection to Kafka from Faust client
I am having a hard time connection to a machine running Kafka from a client running a Faust script.The script looks like this:
import faust
import logging
from asyncio import sleep
class Test(faust....
- 177
0
votes
0
answers
896
views
Kafka: monitoring length of queues / topics
We have several producers and several consumers connected via Kafka - basically, it's batch processing jobs that are created on demand, placed on Kafka in several queues, and the batch processors pick ...
- 1,208
3
votes
2
answers
2k
views
Kafka stores log4j logs in directory literally called ${kafka.logs.dir}
I installed a preview version of Kafka 3.0 with log4j2 support (http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/) on a RHEL 8 server. Kafka and Zookeeper are running successfully as ...
1
vote
1
answer
2k
views
Kafka - Get Ip address or hostnames connected to the cluster
I have a test Kafka Cluster in AWS MSK with three brokers. I'd like to know how to get information about who is connecting to the cluster either to produce or consume messages.
For instance, MSSQL ...
- 133
0
votes
0
answers
970
views
How to remove escape character when using kafka console consumer?
For compliance purposes, I have to save all kafka raw documents and keep them one year. To do this, I use the following :
kafka-console-consumer.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:...
- 1
0
votes
1
answer
194
views
Kafka + Query the detailed configuration of all Topics
we have Kafka servers cluster with Kafka version 2.6 on Linux RHEL machines
machine in the cluster are - kafka1,kafka2,kafka3
usually we are use the following cli in order to get the detailed ...
- 559
1
vote
1
answer
918
views
zookeeper ssl alert "alert bad certificate"
I am using Kafka (version 2.3.0) and Zookeeper (version 3.5.5-3) - the stable version is 3.6.3.
When I test the SSL of my Zookeeper using this command:
openssl s_client -showcerts -connect 127.0.0.1:...
- 13
1
vote
0
answers
111
views
Kafka installation with Confluent vs Apache Kafka [closed]
for now we have HDP 2.6.5 version , with Kafka cluster with Kafka version 0.1
all Kafka machines are installed on RHEL version 7.6
all machines with RHEL 7.6 are on DELL physical machines
we want to ...
- 559
0
votes
1
answer
2k
views
Kafka Broker Runs locally via docker compose but fails in kubernetes with near identical config
Kafka broker runs beautifully when running locally via docker-compose.
version: "3"
services:
zookeeper:
image: 'bitnami/zookeeper:latest'
ports:
- '2181:2181'
...
- 11
0
votes
1
answer
924
views
How to manage storage / persistence overflow in Bitnami Kafka Helm chart?
I deploy Bitanmi Kafka Helm chart on AWS provisioned with Terraform. I find the documentation storage and persistence allocation, very confusing. From what I understood from the documentation; logs ...
- 101
0
votes
1
answer
398
views
How to rotate compressed log file on centos 7?
I have customized kafka server and the app is creating log files and configures log rotation. My problem is that I need to compress the rotation logs that were created.
I tried to create new log ...
- 1
1
vote
2
answers
784
views
The Debezium Kafka based KAFKA_ADVERTISED_LISTENERS property setting is ignored
I have a Docker Compose based Swarm with a Zookeeper service that starts fine, but the Kafka service cannot start.
It keeps using the default KAFKA_ADVERTISED_LISTENERS property:
debezium_kafka.1....
- 227
1
vote
1
answer
309
views
PUB/SUB Message Queue with server side filtering?
I am looking for a message queue to realize a chat application.
The most important requirement is: when a subscriber specifies a filter criteria, all the messages need to be filtered on the server ...
- 11
0
votes
1
answer
194
views
How to implement a secured kafka environment with a public interface?
Scenario: Kafka cluster, secured with SSL encryption and SASL/PLAIN authentication, resides in an AWS private subnet within a dedicated VPC. Within the private subnet, all is fine. I use self-...
- 3
1
vote
0
answers
401
views
How can I further analyze ram usage in a possible leak? [debian@HyperV + Kafka + Zookeeper]
We have a 3-box cluster Kafka+Zookeeper, in a virtualized enviroment (in Windows 2019 HyperV).
In the machines' config dynamic memory is disabled, and they have 4G RAM each (I know it's a very few ...
- 11
1
vote
0
answers
864
views
Confluent Kafka Connect S3 re-processing data due to offset out of range
I'm experiencing a strange situation.
My Confluent Kafka Connect S3 is processing around 8K messages/sec. What started happening randomly every few days is:
The Fetcher class would try to fetch an ...
- 433
0
votes
1
answer
866
views
How set a fixed private IP for ECS containers?
I have a Task Definition called kafka-zookeeper, it has an Apache Kafka broker and a Zookeeper node in it. Those containers need to communicate with each other (that's easy) and with the kafka and ...
- 161
0
votes
1
answer
2k
views
Filebeat kafka input with SASL?
I'm trying to get filebeat to consume messages from kafka using the kafka input. I'm unable to authenticate with SASL for some reason and I'm not sure why that is. The documentation for both Kafka and ...
- 155
16
votes
9
answers
33k
views
Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)
Below the steps I did to get this issue :
Launch ZooKeeper
Launch Kafka : .\bin\windows\kafka-server-start.bat .\config\server.properties
Also note that my OS is Windows
And at the second step ...
- 261
0
votes
1
answer
378
views
kafka in kubernetes pod
I am new at working with kubernetes setup, and are trying to run kafka pod with a persistence volume,
so in case a pod goes down, the memory is not lost, and I can spin up a new cluster, using the ...
- 1
0
votes
1
answer
786
views
mkfs + xfs + what is the right mkfs cli in order to create xfs file-system on huge disk
We need to create xfs file-system on kafka disk
The special thing about kafka disk is the disk size
kafka disk have 20TB size in our case
I not sure about the following mkfs , but I need advice to ...
- 461
0
votes
0
answers
2k
views
zookeeper server due unable to load database on disk
We have kafka cluster with 3 nodes , each kafka include zookeeper server and schema registry
We get the following error on one of the zookeeper server
[2019-11-12 07:44:20,719] ERROR Unable to load ...
- 461
2
votes
1
answer
561
views
Linux + docker container folder that mounted to OS volume + how to know the real limited size
inside the docker-compose.yml we configured the following volume
image: confluentinc/cp-kafka:latest
volumes:
- /grid/kafka-data:/var/lib/kafka/data
.
docker-compose ps
Name ...
- 461
1
vote
0
answers
1k
views
Getting error INFO [SocketServer brokerId=3] Failed authentication with /10.2.***.* (SSL handshake failed) (org.apache.kafka.common.network.Selector)
I have kafka multibroker deployed on kubernetes, with interbroker SSL communication, I have set the folowing properties:
advertised.listeners=SSL://kafka-1.**.***.com:9093,PLAINTEXT://kafka-1.**.*****...
- 113
0
votes
1
answer
147
views
Can logstash be used as a kafka forwarder?
I am looking at the plugins for Logstash and I noticed it had both an input and output plugin for kafka. For a single topic, can Logstash be used to forward kafka topic messages to another kafka ...
0
votes
1
answer
199
views
Way to forward remote internal network ip and port to same ip and port on local mac
For development reasons I want to connect my local machine (macos) to a remote private network of a kafka/zookeeper cluster.
When I do port forwarding with
ssh [email protected] -N -L 10.0.0.8:...
- 339
3
votes
2
answers
2k
views
Enable logical replication on Google Cloud Postgres
Is there anyway to enable logical replication wal_level = logical on Google Cloud Postgres instance? I want to do CDC (change data capture) to stream WAL Record to Apache Kafka.
Thanks for your help!
google-cloud-platform
0
votes
1
answer
344
views
Creating RabbitMQ cluster (by Bitnami) and Kafka cluster (by Bitnami) in Azure and connecting to them from Kubernetes in another v-net
I'm trying to create an architecture with:
1. Kafka cluster (by Bitnami)
2. Rabbitmq cluster (by Bitnami)
3. Kubernetes cluster (AKS) that can use both Kafka and RMQ
When I created both clusters (...
0
votes
1
answer
137
views
Can I control which terraform instances are discarded when reducing count?
When having a multi-instance provisioning that needs to be reduced - can I control which of the existing resource will be killed?
I have a Kafka cluster with variable load, so I would like to provide ...
- 3
0
votes
0
answers
18
views
Message Forwarder to MapR Streams/Kafka
We have an application that forwards data to MapR Streams (Kafka). Problem is if the network link goes down then data is lost as the application itself doesn't keep a history. We are looking at ...
- 366
0
votes
1
answer
237
views
Expose Confluent Kafka outside of the cluster but inside of the GCP
I've installed Confluent Kafka OSS using their official helm chart: https://github.com/confluentinc/cp-helm-charts
However, I would like to consume Kafka topics in a GCP Dataflow so this mean, I ...
0
votes
1
answer
307
views
hadoop + can we install zookeeper servers on kafka hosts
we want to dedicated the zookeeper servers only for kafka machines
so each kafka machine include the zookeeper server
and zookeeper server will serve only the kafka host and not other application
in ...
- 461
Related Tags
linux × 16
zookeeper × 12
kubernetes × 9
hadoop × 5
monitoring × 5
hard-drive × 4
ssl × 4
logging × 3
docker × 3
collectd × 3
centos × 3
bitnami × 3
nginx × 3
rsyslog × 3
azure × 2
cache × 2
graylog × 2
hostname × 2
helm × 2
ansible × 2
java × 2
jmx × 2
more related tags
Hot Network Questions
- Converting fuse panels to a subpanel in a detached garage
- How do I make my male character take off his bra?
- Will my Hydra sink into the bog?
- Is there a distance so small it can't can't be further divided
- Short story about a prehistoric man discovering either salt, garlic or onions for cooking
- What's the formula for an idealized diode?
- Why is the ECMAScript specification formatted the way it is?
- Why are humans and AI often treated differently in cases where they perform nearly identical processes?
- Should one ever use \bigskip?
- How did Professor Sprout bandage the Whomping Willow?
- How do I tell my work I don't want to do the project that I've been put on?
- Orion stars distance from Earth
- sms send count more than population
- What is the right age to teach your kids about weather-appropriate clothing?
- Is \ProcessKeysPackageOptions obsolete?
- Somebody sent me a check to deposit into my account, but I now think that it's a scam. will I get in trouble?
- How loud are tubular bells?
- How to use systeme to write a system of equations with subscripts in the coefficients
- String containing characters in the same order as other string
- Is it true that common law courts will not resolve a question without a controversy?
- Implement Subleq
- Why AP331A comparator doesn't work without pullup?
- Could world hunger be ended for 40 billion USD per year?
- What’s the deal with צרויה?