15
votes
Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)
I solved this issue on my machine (Windows 10) by deleting the meta.properties file found in the log directory.
The issue itself occured after I had stopped both the zookeeper and server and then ...
- 251
9
votes
Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)
Usually, this problem arises when kafka logs are stored in the persistent folder and zookeeper data in the temporary, or vice-versa.
Then, after system restart, files that are stored in the temporary ...
- 91
7
votes
Kafka logfile location
For other unfortunate lads like my, you need to modify LOG_DIR environment variable (tested for Kafka v0.11).
If you open script kafka-server-start or /usr/bin/zookeeper-server-start, you will see at ...
- 243
7
votes
ZooKeeper not starting because acceptedEpoch is less than the currentEpoch
It worked, I just did:
cp currentEpoch acceptedEpoch
in a zookeeper data directory, and it started without any problem.
- 191
4
votes
Accepted
Enable logical replication on Google Cloud Postgres
Unfortunately, this is currently not possible.
However, this feature was requested some time ago, and you can track its progression with this Public Feature Request
If you click on the star icon on ...
- 126
3
votes
Enable logical replication on Google Cloud Postgres
This is now in preview:
https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication
"You can use logical replication and decoding features in Cloud SQL for PostgreSQL. ...
- 31
3
votes
Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)
I encountered the same issue while running Kafka server on my Windows Machine. You can try the following to resolve this issue:
Open server server.properties file which is located in your kafka ...
- 31
3
votes
What is ideal directory for kafka messages
You'll always find me placing files in standard directories or as close to them as possible.
The reason for this is so that future admins can find them later -- because very often that future admin is ...
- 240k
2
votes
What do MBean parameters mean in collectd?
This looks like a gauge, by virtue of being a 'Per' value -- Kafka is already normalizing this into a particular time frame. If the name was just 'MessagesIn' I'd expect it to be a counter -- a number ...
- 4,955
2
votes
Accepted
Choose number of partitions in Kafka
There is no good default number of partitions, and you should provide more information.
It depends on the size of messages, your platform and the usage pattern. Can a server store all messages with ...
2
votes
Accepted
what is the safe and best way to delete the kafka topic folders
Use the appropriate log.retention.X broker properties, as described here. By default logs are retained for 168 hours (7 days) but you can set it as low as 1 millisecond (which is useful to completely ...
- 36
2
votes
What is ideal directory for kafka messages
The best place is a separate partition mounted to have all of the data in the same place, which holds no other function for the OS and/or other installed packages
/var/log/ is being handled by other ...
- 1,814
2
votes
kafka scripts in "bin" directory fail with "Address already in use" (running in kubernetes)
In my experience, the reason is that JMX exporter will be started under port that already used by Kafka server.
To double-check, you can execute command ps -ef | grep kafka | grep jmx_prometheus, ...
2
votes
Accepted
Apache Kafka configuration behind NAT/LB
We have talked to a Kafka expert and here is what came out of it.
Simply put, while kafka has the Listener Names in its knowledge to differentiate 2 listeners, he needs to know which one the client ...
- 467
2
votes
Accepted
Kafka stores log4j logs in directory literally called ${kafka.logs.dir}
Just add sys: prefix to the kafka.logs.dir system variable in the log4j2.properties file. That is:
AS-IS: ${kafka.logs.dir}
TO-BE: ${sys:kafka.logs.dir}
DISCLAIMER: As you can see in my name, I ...
- 36
1
vote
Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)
Try the following...
Enable following line in ./config/server.properties
listeners=PLAINTEXT://:9092
Modify default ZooKeeper dataDir
Modify default Kafka log dir
- 11
1
vote
Accepted
mkfs + xfs + what is the right mkfs cli in order to create xfs file-system on huge disk
Yes, mkfs.xfs defaults should be sufficient.
XFS has been tested on volumes hundreds of TiB in size for years.
Kafka's XFS notes state that
The XFS filesystem has a significant amount of auto-...
- 30.9k
1
vote
When to use Apache Kafka log roll jitter?
Roll checks are performed on message appends. So if you want to roll out log segments every 10 hours, you might want to set an buffer interval (i.e jitter), let's say 1 hour before in which this ...
- 11
1
vote
Kafka long coordinator load time and small ISRs
I haven't used that exact Kafka version but I will try to respond to the questions:
You might have unclean leader election enabled, it depends on the number of partitions vs the number of consumers
...
- 1,814
1
vote
Dose zookeeper servers must use SSD disks?
It's not mandatory to use SSD for zookeeper, issue raised when actually too many application using same zookeeper and doing lots of transaction on zookeeper, AFAIK I past with production cluster ...
- 2,571
1
vote
kafka failed authentication due to: SSL handshake failed
I used to your shared step to generate the certificates and configured in the Kafka and spring boot producer and consumer level, all are working fine.
I have also got the same error but I did the ...
1
vote
ZooKeeper not starting because acceptedEpoch is less than the currentEpoch
This did solve the problem,
cp ~/zookeeper/version-2/currentEpoch ~/zookeeper/version-2/acceptedEpoch
actual reason for this is issue is time sync b/w the zookeeper nodes
- 11
1
vote
kafka with ACL fails to connect zk and stops
jaas file format is not completely clear to me, anyway what eventually worked for me was to use all user/passoword formats for both Client and Server jaas files
Client {
org.apache.zookeeper....
- 121
1
vote
Kafka logfile location
Maybe not actually the answer to the questions but I've arrived here searching for where kafka stores application logs, not producers/consumers data. IMO calling this data logs is really bad naming.
...
- 111
1
vote
Kafka logfile location
As eddyP23 mentioned in previous response... kata-run-class uses LOG_DIR
However there is logic in it to use Script directly parent folder if LOG_DIR variable is not set.
So if your scripts are in ...
1
vote
Accepted
What do MBean parameters mean in collectd?
I finally manage to get it working. Pretty much all Kafka metrics in JMX have multiple attributes. For instance, this kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec contains the following ...
- 181
1
vote
Accepted
What is the correct way to export environmental values through ansible playbook?
It looks like your STATIC_LIB_... values are prefixed by a space which will make the static lib check to fail.
- 136
1
vote
Accepted
Wrong measurements being reported by JMX and collectd for Apache Kafka
Just a small follow up on this issue: it wasn't neither JMX nor collectd mistakes. I verified JMX, collectd and Librato were with consistent values by logging in directly to each Kafka broker and ...
- 181
1
vote
Accepted
Cannot use supervisor with Apache Kafka
I finally managed to get supervisor working with Kafka with two changes:
Deploy Kafka without -daemon flag, as supervisor requires non-daemozined process to manage
Explicitly define the Java path in ...
- 181
1
vote
Accepted
Rsyslog. Disk-assisted queue for omkafka
You should add this config to your action block:
name="kafkaoutput"
action.resumeretrycount="-1"
And it will write to the DA queue when the queue size > 450000
- 26
Only top scored, non community-wiki answers of a minimum length are eligible
Related Tags
kafka × 96linux × 15
zookeeper × 13
kubernetes × 9
monitoring × 6
hadoop × 5
ssl × 4
nginx × 3
centos × 3
amazon-web-services × 3
docker × 3
logging × 3
google-cloud-platform × 3 hard-drive × 3
rsyslog × 3
bitnami × 3
collectd × 3
jmx × 3
networking × 2
azure × 2
redhat × 2
java × 2
ansible × 2
port-forwarding × 2
log-files × 2