0

How would one prefix journald log entries with metadata within the log message?

I'm thinking I may have to view one of the verbose outputs, filter out line breaks and all extra info, find just the items I am looking for (DateTime, Docker CONTAINER_NAME, Message) and ignore the rest all onto a single line.

Could something like awk be used to strip line breaks, grab only X, Y, Z lines, and display them onto a single line? How would args then know to group every X, Y, Z lines? I mean, how would args know about each "grouping"?

More info follows.


Here are two lines from two different docker containers I am logging to journald:

Apr 28 18:09:43 rschool dockerd[1366]: [pid: 9|app: 0|req: 1/1] 68.180.230.53 () {48 vars in 934 bytes} [Fri Apr 28 14:09:42 2017] GET /enrollment/info-sessions/ => generated 17175 bytes in 1072 msecs (HTTP/1.0 200) 3 headers in 112 bytes (1 switches on core 0)
Apr 28 18:09:43 rschool dockerd[1366]: 68.180.230.53 - - [28/Apr/2017:18:09:43 +0000] "GET /enrollment/info-sessions/ HTTP/1.1" 200 3495 "-" "Mozilla/5.0 (Macintosh; IntelMac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36" "-"

As you can see, you can't tell which log entry belongs to which docker container.

There is metadata available with viewing the full log message. Here's one of those messages, when viewed with journald -o verbose:

_UID=0
_GID=0
_SYSTEMD_SLICE=system.slice
_BOOT_ID=f4a6e9569f0349d1817bd92ab779ebe3
_MACHINE_ID=a62f158e48fc45eeb32afaef98d24d5b
_HOSTNAME=rschool
_TRANSPORT=journal
_CAP_EFFECTIVE=3fffffffff
_PID=1366
_COMM=dockerd
_EXE=/usr/bin/dockerd
_CMDLINE=/usr/bin/dockerd -H fd://
_SYSTEMD_CGROUP=/system.slice/docker.service
_SYSTEMD_UNIT=docker.service
CONTAINER_NAME=rschool_web_1
CONTAINER_ID=732e5bf0d0a1
CONTAINER_ID_FULL=732e5bf0d0a1cc110cacce68850143aa3534
CONTAINER_TAG=rschool_web/rschool_web_1/732e5bf0d0a1
MESSAGE=[pid: 9|app: 0|req: 1/1] 68.180.230.53 () {48 vars in 934 bytes} [Fri Apr 28 14:09:42 2017] GET /enrollment/info-sessions/ => generated 17175 bytes in 1072 msecs (HTTP/1.0 200) 3 headers in 112 bytes (1 switches on core 0)
_SOURCE_REALTIME_TIMESTAMP=1493402983899475
Fri 2017-04-28 18:09:43.901030 UTC [s=9d2777df7c3e4658a6d3d2c7896376ce;i=13ca;b=f4a6e9569f0349d1817bd92ab779ebe3;m=c91fe4d38;t=54e3dfa5ca862;x=f8ee400046f7d86f]
PRIORITY=6

But journald doesn't seem to have any deliminating nature with -o verbose.

Part of Docker log driver using journald, it adds things like CONTAINER_NAME - which is exactly what I am looking for.

How would I display the CONTAINER_NAME using a format similar to -o short?

1 Answer 1

3

Ok, I found the answer as I was typing. It's very long-winded though. So a bash alias will help (see the end). Making this into a "Wiki Answer" as it took me a long time to type out, and find the answer. Maybe it can help someone else one day.


There is the JSON output format available with -o json.

We can use jq to pick things from the json, format and concatenate them, and display them on a single.

journalctl -f -n 100 -o json | jq -r '.__REALTIME_TIMESTAMP + " " + .PRIORITY + " " + ._HOSTNAME + " " + ._SYSTEMD_UNIT + " " + .CONTAINER_TAG + " " + .MESSAGE'

I use the docker tag option to specify a custom tag. For example, my docker-compose.production.yml override looks like:

version: '2'
services:
    nginx:
        restart: always
        ports:
            - "80:80"
            - "443:443"
        logging:
            driver: journald
            options:
                tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"

This gives me a line like:

1493405629162557 6 rschool docker.service nginx:1.12.0-alpine/rschool_nginx_1/0f6b8d772957 X.X.X.X - - [28/Apr/2017:18:53:49 +0000] "GET / HTTP/1.1" 200 4399 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36" "-"

Which is exactly what I am after. This is also compatible with all other journald entries:

1493405535267844 4 rschool   [UFW BLOCK] IN=eth0 OUT= MAC=9e:51:e1...:08:00 SRC=X.X.X.X DST=X.X.X.X LEN=439 TOS=0x00 PREC=0x00 TTL=57 ID=41599 DF PROTO=UDP SPT=5273 DPT=5060 LEN=419 

If you aren't using docker log tags, you can just use the CONTAINER_NAME:

journalctl -f -n 100 -o json | jq -r '.__REALTIME_TIMESTAMP + " " + .PRIORITY + " " + ._HOSTNAME + " " + ._SYSTEMD_UNIT + " " + .CONTAINER_NAME + " " + .MESSAGE'

That should work for all Docker containers regardless of linux system logging to journald.

All that is left is to make this into a bash alias to reference quickly. Add this to your ~/.bash_aliases and source ~/.bash_aliases:

alias journalctlf="journalctl -f -n 100 -o json | jq -r '.__REALTIME_TIMESTAMP + \" \" + .PRIORITY + \" \" + ._HOSTNAME + \" \" + ._SYSTEMD_UNIT + \" \" + .CONTAINER_NAME + \" \" + .MESSAGE'"

(the "f" stands for formatted)

Then you can run journalctlf outright, and bask in the glory of jorunald with docker container names.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .