systemd's journalctl: how to filter by message?

Question

journalctl looks like a great tool for looking through logs, but I'm stuck on what feels like a simple ask: I want to see all cron messages that contain the phrase update-ipsets.

Of course I can do this

journalctl -u cron.service | grep update-ipsets

but then you lose all the other benefits of journalctl's output (colour coding, auto paging, live view etc.)

I've tried:

journalctl -u cron.service MESSAGE=update-ipsets
journalctl -u cron.service "MESSAGE=*update-ipsets*"
journalctl -u cron.service "MESSAGE=.*update-ipsets.*"
journalctl -u cron.service "MESSAGE=/.*update-ipsets.*/"

And you don't want to experiment by hitting tab after MESSAGE= - hangs the (zsh/Debian Jessie) shell and Ctrl-C didn't help either!

I sort of can't believe that it doesn't have this basic functionality built in, so I'm sure I must have missed something?

Thanks.

Answer 1

Currently, journalctl does not support patterns or wildcards in field matches. grep is your best option.

I had the same problem, and I think that journalctl only searches for an exact match for VALUE when NAME=VALUE is passed as arguments.

My investigations:

1. man page

   From journalctl(1)

   The pattern is not mentioned in the description of the matches:

    [...] A match is in the format "FIELD=VALUE", e.g.
    "_SYSTEMD_UNIT=httpd.service", referring to the components
    of a structured journal entry. [...]
   

   The man page refers to a pattern when describing -u option only.

      -u, --unit=UNIT|PATTERN
          Show messages for the specified systemd unit UNIT 
          (such as a service unit), or for any of the units
          matched by PATTERN. 
   

2. Source code

   The function fnmatch in src/journal is used when searching for units only.

3. debug journalctl

   Enabling debug output you can see that the pattern is expanded only when using -u.

   $ SYSTEMD_LOG_LEVEL=debug journalctl -n1 -u gdm*
   ...
   Matched gdm.service with pattern _SYSTEMD_UNIT=gdm*
   Matched gdm.service with pattern UNIT=gdm*
   Journal filter: ((OBJECT_SYSTEMD_UNIT=gdm.service AND _UID=0) OR (UNIT=gdm.service AND _PID=1) OR (COREDUMP_UNIT=gdm.service AND _UID=0 AND MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1) OR _SYSTEMD_UNIT=gdm.service)
   ...
   

   All the matches are treated as exact, including UNIT:

   $ SYSTEMD_LOG_LEVEL=debug journalctl -n1 UNIT=gdm.*
   ...
   Journal filter: UNIT=gdm*
   ...
   

Answer 2

Since systemctl --version version 237 there might be grep pattern support with -g/--grep switch, but it has to be compiled with PRCE2 support (it doesn't appear to be included in Debian Buster, >=242 is needed, it's possible to install in from buster-backports)

journalctl -g ipsets*

Without grep support you can still switch to cat output mode and use grep's matching:

journalctl -b -o cat --no-pager | grep "update-ipsets"

if you want a pager it's best to pipe the result to less. You can use invert matching -v / --invert-match to exclude certain messages

journalctl -b -o cat --no-pager | grep -v "ACPI" | less

Yet another option is to use json format:

journalctl -b -o json | jq -C . | less -R

which gives verbose output, single line

{
  "SYSLOG_IDENTIFIER": "kernel",
  "_MACHINE_ID": "d72735cff36a41f0a5326f0bb7eb1778",
  "_SOURCE_MONOTONIC_TIMESTAMP": "0",
  "__REALTIME_TIMESTAMP": "1614516018297106",
  "__CURSOR": "s=2b1deb3dba3e42e4a758cc8f011d19c5;i=1;b=c0f6b0e5143a4d3db9f04f00da1a4ff4;m=670dd9;t=5bc64cdc13912;x=ee44184076fc0590",
  "__MONOTONIC_TIMESTAMP": "6753753",
  "SYSLOG_FACILITY": "0",
  "_HOSTNAME": "w16",
  "PRIORITY": "5",
  "_BOOT_ID": "c0f6b0e5143a4d3db9f04f00da1a4ff4",
  "_TRANSPORT": "kernel",
  "MESSAGE": "Linux version 4.19.0-14-amd64 ([email protected]) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.171-2 (2021-01-30)"
}

using jq you can easily filer messages:

$ journalctl -b -o json | jq '. | select(._COMM=="sensors")' | jq -r .MESSAGE | less
$ journalctl -b -o json | jq '. | select(._TRANSPORT=="kernel")' | jq -r .MESSAGE | head -n 1
Linux version 4.19.0-14-amd64 ([email protected]) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.171-2 (2021-01-30)

Answer 3

With journalctl version 247.3-7 (as systemd), the --grep (or -g) option allows to filter lines in journal where MESSAGE field contain a string or match a regular expression.

man journalctl says :

**-g, --grep=**

           Filter output to entries where the MESSAGE= field matches the specified regular expression. PERL-compatible regular expressions are used, see pcre2pattern(3) for a detailed description of the
           syntax.

           If the pattern is all lowercase, matching is case insensitive. Otherwise, matching is case sensitive. This can be overridden with the --case-sensitive option, see below.

What is not said neither in the manual of journalctl nor in pcre2pattern, is that the regular expression has to be surrounded by either single or dual quotes and no other character !!!

If you run the following command as root or using sudo :

# journalctl --grep "UFW BLOCK"

you will get a list of all recorded block IDs and the request blocked by the firewall...

Hope this help !