Catch logs of containers created by a Docker executor of a GitLab Runner

Question

Given a Docker Engine client (GitLab Runner) which creates and destroys containers dynamically through the Docker Engine API.

Constraints

• The HostConfig.LogConfig.Type is always set to json-file and can not be changed; see also https://gitlab.com/gitlab-org/gitlab-runner/-/commit/aa96ed52c7a8bea6b6679d0bca682404bfed14e4.
• Docker Engine removes all JSON log files whenever the corresponding container is destroyed; the options max-file and max-size do not seem to affect that behavior.
• Log driver configuration of the Docker Daemon will always be overwritten by the client creating a container if given.

Please correct me if I'm wrong in one or more of the constraints above.

How could I catch the logs of the containers created by the GitLab Runner without:

• making a merge request to GitLab Runner or Docker Engine
• changing the image of the created containers

Background is, that GitLab Runner does not publish the logs of so called "service containers"; see also https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2119.

Answer 1

You can use the Docker Engine API. Write a program that:

1. attaches to the event stream
2. attaches to the log stream of already running containers
3. attaches to the log stream of newly started containers
4. detaches from the log stream of destroyed containers

Pipe e.g. the log streams as JSONL to STDOUT. Based on that you should have a lot of further options to proceed.

For 2. and 3. make sure that you use only containers which are using json-file or journald log driver.