Hot answers tagged

15 votes

Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)

I solved this issue on my machine (Windows 10) by deleting the meta.properties file found in the log directory. The issue itself occured after I had stopped both the zookeeper and server and then ...
  • 251
9 votes

Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)

Usually, this problem arises when kafka logs are stored in the persistent folder and zookeeper data in the temporary, or vice-versa. Then, after system restart, files that are stored in the temporary ...
7 votes

Kafka logfile location

For other unfortunate lads like my, you need to modify LOG_DIR environment variable (tested for Kafka v0.11). If you open script kafka-server-start or /usr/bin/zookeeper-server-start, you will see at ...
  • 243
7 votes

ZooKeeper not starting because acceptedEpoch is less than the currentEpoch

It worked, I just did: cp currentEpoch acceptedEpoch in a zookeeper data directory, and it started without any problem.
  • 191
4 votes
Accepted

Enable logical replication on Google Cloud Postgres

Unfortunately, this is currently not possible. However, this feature was requested some time ago, and you can track its progression with this Public Feature Request If you click on the star icon on ...
  • 126
3 votes

Enable logical replication on Google Cloud Postgres

This is now in preview: https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication "You can use logical replication and decoding features in Cloud SQL for PostgreSQL. ...
  • 31
3 votes

Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)

I encountered the same issue while running Kafka server on my Windows Machine. You can try the following to resolve this issue: Open server server.properties file which is located in your kafka ...
3 votes

What is ideal directory for kafka messages

You'll always find me placing files in standard directories or as close to them as possible. The reason for this is so that future admins can find them later -- because very often that future admin is ...
2 votes

What do MBean parameters mean in collectd?

This looks like a gauge, by virtue of being a 'Per' value -- Kafka is already normalizing this into a particular time frame. If the name was just 'MessagesIn' I'd expect it to be a counter -- a number ...
  • 4,955
2 votes
Accepted

Choose number of partitions in Kafka

There is no good default number of partitions, and you should provide more information. It depends on the size of messages, your platform and the usage pattern. Can a server store all messages with ...
2 votes
Accepted

what is the safe and best way to delete the kafka topic folders

Use the appropriate log.retention.X broker properties, as described here. By default logs are retained for 168 hours (7 days) but you can set it as low as 1 millisecond (which is useful to completely ...
  • 36
2 votes

What is ideal directory for kafka messages

The best place is a separate partition mounted to have all of the data in the same place, which holds no other function for the OS and/or other installed packages /var/log/ is being handled by other ...
  • 1,814
2 votes

kafka scripts in "bin" directory fail with "Address already in use" (running in kubernetes)

In my experience, the reason is that JMX exporter will be started under port that already used by Kafka server. To double-check, you can execute command ps -ef | grep kafka | grep jmx_prometheus, ...
2 votes
Accepted

Apache Kafka configuration behind NAT/LB

We have talked to a Kafka expert and here is what came out of it. Simply put, while kafka has the Listener Names in its knowledge to differentiate 2 listeners, he needs to know which one the client ...
  • 467
2 votes
Accepted

Kafka stores log4j logs in directory literally called ${kafka.logs.dir}

Just add sys: prefix to the kafka.logs.dir system variable in the log4j2.properties file. That is: AS-IS: ${kafka.logs.dir} TO-BE: ${sys:kafka.logs.dir} DISCLAIMER: As you can see in my name, I ...
1 vote

Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)

Try the following... Enable following line in ./config/server.properties listeners=PLAINTEXT://:9092 Modify default ZooKeeper dataDir Modify default Kafka log dir
1 vote
Accepted

mkfs + xfs + what is the right mkfs cli in order to create xfs file-system on huge disk

Yes, mkfs.xfs defaults should be sufficient. XFS has been tested on volumes hundreds of TiB in size for years. Kafka's XFS notes state that The XFS filesystem has a significant amount of auto-...
1 vote

When to use Apache Kafka log roll jitter?

Roll checks are performed on message appends. So if you want to roll out log segments every 10 hours, you might want to set an buffer interval (i.e jitter), let's say 1 hour before in which this ...
1 vote

Kafka long coordinator load time and small ISRs

I haven't used that exact Kafka version but I will try to respond to the questions: You might have unclean leader election enabled, it depends on the number of partitions vs the number of consumers ...
  • 1,814
1 vote

Dose zookeeper servers must use SSD disks?

It's not mandatory to use SSD for zookeeper, issue raised when actually too many application using same zookeeper and doing lots of transaction on zookeeper, AFAIK I past with production cluster ...
  • 2,571
1 vote

kafka failed authentication due to: SSL handshake failed

I used to your shared step to generate the certificates and configured in the Kafka and spring boot producer and consumer level, all are working fine. I have also got the same error but I did the ...
1 vote

ZooKeeper not starting because acceptedEpoch is less than the currentEpoch

This did solve the problem, cp ~/zookeeper/version-2/currentEpoch ~/zookeeper/version-2/acceptedEpoch actual reason for this is issue is time sync b/w the zookeeper nodes
1 vote

kafka with ACL fails to connect zk and stops

jaas file format is not completely clear to me, anyway what eventually worked for me was to use all user/passoword formats for both Client and Server jaas files Client { org.apache.zookeeper....
  • 121
1 vote

Kafka logfile location

Maybe not actually the answer to the questions but I've arrived here searching for where kafka stores application logs, not producers/consumers data. IMO calling this data logs is really bad naming. ...
  • 111
1 vote

Kafka logfile location

As eddyP23 mentioned in previous response... kata-run-class uses LOG_DIR However there is logic in it to use Script directly parent folder if LOG_DIR variable is not set. So if your scripts are in ...
1 vote
Accepted

What do MBean parameters mean in collectd?

I finally manage to get it working. Pretty much all Kafka metrics in JMX have multiple attributes. For instance, this kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec contains the following ...
1 vote
Accepted

What is the correct way to export environmental values through ansible playbook?

It looks like your STATIC_LIB_... values are prefixed by a space which will make the static lib check to fail.
  • 136
1 vote
Accepted

Wrong measurements being reported by JMX and collectd for Apache Kafka

Just a small follow up on this issue: it wasn't neither JMX nor collectd mistakes. I verified JMX, collectd and Librato were with consistent values by logging in directly to each Kafka broker and ...
1 vote
Accepted

Cannot use supervisor with Apache Kafka

I finally managed to get supervisor working with Kafka with two changes: Deploy Kafka without -daemon flag, as supervisor requires non-daemozined process to manage Explicitly define the Java path in ...
1 vote
Accepted

Rsyslog. Disk-assisted queue for omkafka

You should add this config to your action block: name="kafkaoutput" action.resumeretrycount="-1" And it will write to the DA queue when the queue size > 450000
  • 26

Only top scored, non community-wiki answers of a minimum length are eligible