3

kubelet has an option to store log files in a log-dir (https://github.com/kubernetes/kubernetes/issues/21248).

I start kubelet with the options --logtostderr=false --log-dir=/var/log/kubelet (I have already created the directory /var/log/kubelet) and I indeed see log files with the format kubelet.(hostname).root.log.(date) getting generated in /var/log/kubelet.

In my case I see three files that look like this:

kubelet.hostname.root.log.ERROR.20180301-152321.161945
kubelet.hostname.root.log.INFO.20180301-152320.161945
kubelet.hostname.root.log.WARNING.20180301-152320.161945

Moreover, a symbolic link that points to the latest log file is created for each of the ERROR, INFO and WARNING log files:

kubelet.ERROR -> kubelet.hostname.root.log.ERROR.20180301-152321.161945
kubelet.INFO -> kubelet.hostname.root.log.INFO.20180301-152320.161945
kubelet.WARNING -> kubelet.hostname.root.log.WARNING.20180301-152320.161945

My question is how do I rotate and delete the log files that are older than e.g. 5 days old? It seems these files are generated by glog, but it also seems that although glog rotates the log files, it doesn't support deleting old log files (https://github.com/google/glog/issues/36).

I am aware of logrotate, but then the files should have a fixed filename in order to be rotated by logrotate.

Is there a way to enable kubelet logging in files with proper log rotation and deletion of old files?

  • Why not cron + find? By default, with systemd, kubelet will write to journald. If you are taking control of log management, why not use cron and find to do whatever you need to do with old logs? – Jonah Benton Mar 6 '18 at 0:12
  • That's the quick solution I was thinking about (cron + find). However, I was looking if I could somehow use either logrotate or some hidden kubelet configuration parameter that I am not aware of. Then I would have a more streamlined solution that is closer to the way the standard log rotation works. – Vangelis Tasoulas Mar 6 '18 at 16:07
  • Well, distros have moved to systemd. Thats the standard. kubelet does journald by default. Plenty of people dont like systemd, of course. But if you have requirements not met by systemd, would be good to add them to the question. – Jonah Benton Mar 6 '18 at 16:13
  • Yes, this is true about systemd. The problem is that this machine is facing some issues unrelated to this question, but one of the issues is that journald was crashing. That's why kubelet was configured to log into files instead. – Vangelis Tasoulas Mar 6 '18 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.