I am running docker on ubuntu 16.04 and would like to view the logs. However, I am unable to view logs after what I am guessing is some sort of rotation or the logs grow to a certain size.

I have not made any changes to my journald.conf, so I am using defaults there.

Examples of what I am seeing:

systemctl docker status confirms service has been active since Thu 2016-10-13 18:56:28 UTC.

However, when I run something like journalctl -u docker.service --until "2016-10-13 22:00" - the only output I get is -- Logs begin at Fri 2016-10-14 01:18:49 UTC, end at Fri 2016-10-14 16:18:25 UTC. -- I can view the logs in that range as expected.

My question is why can I not view older logs with journalctl, and how can I fix this issue so I can view the logs?

It could be because you are trying to review the journal since the last boot, which seems likely to be the case inside a docker image.

On Ubuntu 16.04, the journal storage defaults to being in-memory. You can change the default to be persistent by opening /etc/systemd/journald.conf and changing the Storage= line from auto to persistent. You may need to restart to the journald service after that.

I think the journal should be persistent-by-default, so I opened a bug about that.

  • I agree with you it should be persistent by default (and will change that setting), but I am certain this machine has not been rebooted, so I am not 100% convinced that alone will resolve my issue. Still glad you filed that issue. – jdf Oct 14 '16 at 17:32
  • Under certain conditions, logs can land in the journal with incomplete metadata, so that they can be found when reviewing the full journalctl output but not the unit logs. This has been known to happen to logs issues just before a unit shuts down. The logs are sent to journald, and then journald issues a query for the metadata. Sometimes in the meantime the process exists. That also doesn't sound like it is a perfect fit for your case, though. – Mark Stosberg Oct 14 '16 at 17:49
up vote 0 down vote accepted

The reason this happens is because of defaults on the size of journald files stored. There is more detail about this in the docs. It's worth reading the whole section I have linked to, but the defaults work like so:

jounald will use 10% of the disk or 4G, whichever is smaller.

journald will leave free 15% of the disk or 4G, whichever is larger.

For viewing logs from the last boot, assuming you have Storage=persistent in your journald.conf, as the other answer notes, you can use the --boot=-1 flag on journalctl commands to get logs from just the previous boot.

In the case of the OP where they were sure the host had not been rebooted, the loss of logs was simply caused by the SystemMaxUse and/or SystemKeepFree defaults.

Note: I'm the OP and this question still has upvotes trickling in, so since I've gained more experience with journald (and rtfm) I am posting this here in the hopes it helps others.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.