I offer one solution which I have now chosen for myself. It is not the answer I was looking for,but one that seems to sail with the flow.
I'm still curious if this could be handled by some common Unix command.
Anyway, here's what I did:
The common way seems to use a tool called Fluentd, which allows you to collect logs form a various sources and
transport them to anywhere you see fit - kind of ETL for logs.
I chose to send the logs to syslog server since I already had one running, but you could choose any of the output plugins from here: Output plugins.
There are also a large set of additional plugins: All plugins
Get a Fluentd setup that has the remote_syslog plugin installed. That does not come with the official docker image, but you can set it up yourself.
RUN fluent-gem install fluent-plugin-remote_syslog
Build the image and push to your registry.
Next setup a Fluentd deployment manifest with read-only volume claims to access pod logs. The actual files reside in /var/log/pods/* and the /var/log/containers contains actually symlinks. We need actual files.
Those logs are owned by root on the host machine and plain fluent user won't have access to read them.
We need to set some security contexts. For the sake of getting things working I have used root group for the fsGroup.
Feel free to dig deeper and find/comment the most optimal solution for this security wise.
- name: varlogpods-pv
- name: fluentd
See my full manifest in this gist: fluentd-deployment
Before you deploy you also need to set up the
fluent.conf and describe some rules there.
Mine is set to match a log line like this:
2022-04-26T20:05:00.847016854+03:00 stderr F time="2022-04-26 17:05:00" level=info msg="processing 3 records ..."
For more look into tail plugin: tail
expression /^(?<logtime>[^ ]*) .* level=(?<level>[^ ]*) msg="(?<message>[^"]*)"$/
One important config attribute here is
read_from_head true which reads log files from top.
It is necessary for this scenario since the pod logs are rotating we want Fluentd to read the full pod log, not just few update lines in the end. For short cron job the log file just appears and tail wont report any initial lines in it.
Fiddle with config and try, try again.
Don't forget to restart your deployment after you have updated the config in configMap.
Few bits from search trail: