4

Just to make some things clear: I'm talking about the process logfile that contains the stdout and stderr messages.

This is my systemd unit file:

[Unit]
Description=Apache Kafka server
Documentation=http://kafka.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
PIDFile=/var/run/kafka.pid
User=kafka
Group=kafka
Environment="JAVA_HOME=/usr/lib/jvm/jre"
Environment="KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:/var/log/kafka"

ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
SyslogIdentifier=kafka

[Install]
WantedBy=multi-user.target

Note that I added the KAFKA_LOG4J_OPTS environment variable.

However, that doesn't seem to do anything. This is the output, when I try to start my service:

Feb 11 00:55:30 kafka01 kafka[4047]: mkdir: cannot create directory ‘/opt/kafka/bin/../logs’: Permission denied
Feb 11 00:55:30 kafka01 kafka[4047]: OpenJDK 64-Bit Server VM warning: Cannot open file /opt/kafka/bin/../logs/kafkaServer-gc.log due to No such file or directory
Feb 11 00:55:30 kafka01 kafka[4047]: log4j:WARN No appenders could be found for logger (kafka.Kafka$).
Feb 11 00:55:30 kafka01 kafka[4047]: log4j:WARN Please initialize the log4j system properly.

So, can please someone tell me the correct way to change the location of the kafka log file? KAFKA_LOG4J_OPTS doesn't work, KAFKA_LOG_DIR doesn't work either (=I was hoping kafka implemented this similar to zookeeper). The kafka documentation doesn't tell me either.

  • Did you manage to find a solution? – eddyP23 Nov 7 '17 at 14:47
  • Question? What is Kafka version did you install? Kafka 2.11-08.2.2 had this bug and it was fixed in late kafka_2.11-0.11.0.1 version. I had this problem before, but since I reinstalled the kafka_2.11-0.11.0.1 it fixed my issue. – DannyT Nov 9 '17 at 10:46
4

For other unfortunate lads like my, you need to modify LOG_DIR environment variable (tested for Kafka v0.11).

If you open script kafka-server-start or /usr/bin/zookeeper-server-start, you will see at the bottom that it calls kafka-run-class script. And you will see there that it uses LOG_DIR as the folder for the logs of the service (not to be confused with kafka topics data).

0

As eddyP23 mentioned in previous response... kata-run-class uses LOG_DIR

However there is logic in it to use Script directly parent folder if LOG_DIR variable is not set.

So if your scripts are in kafka/bin folders where you are starting kafka... then logs will be present in kafka/log

0

Maybe not actually the answer to the questions but I've arrived here searching for where kafka stores application logs, not producers/consumers data. IMO calling this data logs is really bad naming.

I've deployed a Kafka cluster on GCP Kubernetes engine and I wanted to view the logs of the application and here is how I did it.

1- Get the pods

    > kubectl get pods
    NAME      READY   STATUS    RESTARTS   AGE
    kafka-0   1/1     Running   0          32m
    kafka-1   1/1     Running   0          32m
    kafka-2   1/1     Running   0          32m
    zk-0      1/1     Running   0          23h
    zk-1      1/1     Running   0          23h
    zk-2      1/1     Running   0          23h

2- Connect to one of the Kafkas pods with interactive command

> kubectl exec -it kafka-0 /bin/bash 

3- Echo $, in command line type echo $ and press the tab button several times ... you will get a list of all defined environment variables, search for $KAFKA_HOME or just type

> cd $KAFKA_HOME 

You will get to:

/opt/kafka$ ls -al 
total 68
drwxr-xr-x 1 kafka kafka  4096 Aug  2 09:37 .
drwxr-xr-x 1 root  root   4096 Jun 26  2017 ..
-rw-r--r-- 1 kafka kafka 28824 Apr 21  2017 LICENSE
-rw-r--r-- 1 kafka kafka   336 Apr 21  2017 NOTICE
drwxr-xr-x 1 kafka kafka  4096 Jun 26  2017 bin
drwxr-xr-x 1 kafka kafka  4096 Jun 26  2017 config
drwxr-xr-x 1 kafka kafka  4096 Jun 26  2017 libs
drwxr-xr-x 2 kafka kafka  4096 Aug  2 09:37 logs
drwxr-xr-x 1 kafka kafka  4096 Jun 26  2017 site-docs

The log directory is what I was looking for ...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.