3

I have an AWS EC2 instance running a CentOS 7.7.1908 (systemd 219) with a server application. The server logs quite a lot of information to the system logs (using syslog).

I have recently enabled persistent storage of the system logs using this answer. Since then, the memory consumption of systemd-journald is constantly growing.

After a full day, systemd-journald ends up using more than 250M of RAM.

I did a quick test to confirm that the journald RAM usage is actually growing.

Test :

  • Get the systemd-journald memory usage
  • Log a bunch of data to the system journal using logger
  • Get the systemd-journald memory usage
  • Flush the in-memory logs (if any) to /var/log/journal using journalctl --flush
  • Restart the systemd-journald service
  • Get the systemd-journald memory usage

The results :

# ps aux | grep journald | grep -v grep
root     23963  0.0  0.1  61320  2500 ?        Ss   15:03   0:00 /usr/lib/systemd/systemd-journald
# for i in {0..7000}; do logger -t TEST -p err `python -c 'print "A"*1000'`; done;
# ps aux | grep journald | grep -v grep
root     23963  0.1  0.2  69512 11964 ?        Ss   15:08   0:00 /usr/lib/systemd/systemd-journald
# journalctl --flush
# ps aux | grep journald | grep -v grep
root     23963  0.1  0.2  69512 11964 ?        Ss   15:08   0:00 /usr/lib/systemd/systemd-journald
# systemctl restart systemd-journald
# ps aux | grep journald | grep -v grep
root     24237  0.0  0.1  55416  2492 ?        Ss   15:08   0:00 /usr/lib/systemd/systemd-journald

The memory usage goes from 2,5M to almost 12M after logging rougly 7M of data. Restarting the daemon brings the memory usage back to ~2.5M.

My interpretation :

  • systemd-journald memory usage is proportional to the amount of data it has processed.
  • The parameter RuntimeMaxUse does not limit this memory usage.
  • Restarting systemd-journald clears a buffer, that I can not configure the size of.
  • There is no configurable limit to the amount of memory systemd-journald may use.
  • The memory use is not due to logs waiting to be flushed to the disk.

My journald.conf :

[Journal]
#Storage=auto # /var/log/journal is created and used, so this is equivalent to 'Persistent'
SyncIntervalSec=1m
RateLimitInterval=0
RateLimitBurst=0
SystemMaxUse=4G
RuntimeMaxUse=1M

How can I solve this "leak" without restarting the service every day?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.