On a CentOS 7.2 server I am trying to dump the kernel stack trace using echo t> /proc/sysrq-trigger. Some of the stack traces are not getting logged. I see the following error in /var/log/messages.
Jul 5 13:54:51 glusterfs1 journal: Missed 14295 kernel messages
It seems that some of the entries are getting lost. I need to find the applicable stack traces from this server. Please suggest what needs to be done to not miss the kernel messages.
Journald is logging almost everything and it copy the info to rsyslog via socket. This socket have a maximum throughput.
If you like it you can recompile it from source as "explained" here
https://cgit.freedesktop.org/systemd/systemd/commit/?id=5e8b767df6e18444d5aff2987b5e5603361ed528
But you can also limit the amount of data send trough the socket via the journald.conf in /etc/systemd
with the option
MaxLevelSyslog=
the default is debug, witch mean everything is send to rsyslog. But, historicaly, syslog are configurated to output only message with warning and greater importance. So you can make the configuration of journald to send only warning and greater importance log to syslog, this will do the trick on almost all setup. You will still be able to consult notice, info and debug log via the journalctl command, but until you have to dig in a problem those logs are'nt much relevant.
MaxLevelSyslog=warning
dont forget to restart your journald service afther any modification to journald.conf
systemctl restart systemd-journald
Another mentionned solution is to make rsyslog read the journald file instead of being flood by it via socket. But I did'nt find any usefull information on how to achieve this beheavior.
FYI : syslog level (also respected by journald)
0 emerg
1 alert
2 crit
3 err
4 warning
5 notice
6 info
7 debug
There is a problem with journalctl configuration. Try to change /etc/systemd/journald.conf contents to
[Journal]
Compress=yes
ForwardToConsole=no
ForwardToKMsg=no
ForwardToSyslog=no
ForwardToWall=no
MaxFileSec=1h
MaxRetentionSec=1month
RateLimitIntervalSec=0
RateLimitBurst=0
Seal=yes
SplitMode=uid
Storage=persistent
SyncIntervalSec=20
SystemMaxUse=1G
RuntimeMaxUse=1G
And restart systemd-journald.
