I run pods in kubernetes (1.17.0) that write logs to stdout. This pods are going to be up for a long time. I use promtail to gather logs and send to loki storage. So I can configure to store logs, for example, for last 7 days.

However, I don't clearly understand how kubernetes tracks these logs? I know that kubernetes deletes logs when pod is evicted, but my pods will not be evicted. That means kubernetes will store all logs and eventially node's SSD will be full? How can I configure kubernetes to delete old logs?


Looking at the Logging architecture from the official k8s documentation there are some options:

First one is logging at node level:

Kubernetes currently is not responsible for rotating logs, but rather a deployment tool should set up a solution to address that. For example, in Kubernetes clusters, deployed by the kube-up.sh script, there is a logrotate tool configured to run each hour. You can also set up a container runtime to rotate application’s logs automatically, for example by using Docker’s log-opt. In the kube-up.sh script, the latter approach is used for COS image on GCP, and the former approach is used in any other environment. In both cases, by default rotation is configured to take place when log file exceeds 10MB.

Second one is a sidecar container with logging agent:

Sidecar containers can also be used to rotate log files that cannot be rotated by the application itself. An example of this approach is a small container running logrotate periodically. However, it’s recommended to use stdout and stderr directly and leave rotation and retention policies to the kubelet. Good example of the logrotator can be found here.

Alternatively you can always setup your log retention policy at docker level by specifying appropriate flags. Check out docker docs for that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.