Having recently had a network failure that caused a major cascading series of other issues, I've been diving into our rsyslog.conf to try identify opportunities for more stable deployment.
We have all our application servers running rsyslog, forwarding over tcp to a couple of aggregation servers. Yesterday, when one of those servers was nearly unreachable due to high packet loss, the rest of the environment ground to a halt. I think I have discovered a couple of configs that don't make sense.
Here is our client config:
# modules $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) $KLogPath /proc/kmsg # global app configs $FileOwner syslog $FileGroup adm $FileCreateMode 0644 $DirCreateMode 0755 $Umask 0022 $PrivDropToUser syslog $PrivDropToGroup syslog $PreserveFQDN on # queue configs $MainMsgQueueType LinkedList $MainMsgQueueSize 50000 $ActionQueueType Direct $MainMsgQueueDiscardMark 300000 $MainMsgQueueDiscardSeverity 4 $MainMsgQueueFileName mainmsgqueue $MainMsgQueueMaxFileSize 200m $MainMsgQueueLowWaterMark 10000 $MainMsgQueueHighWaterMark 40000 $MainMsgQueueTimeoutEnqueue 1 $ActionQueueTimeoutEnqueue 1 # Log locally. authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog # Send all components to remote log servers *.* @@<server 1 IP> & @@<server 2 IP>
My questions are:
A) On the client servers, what will happen to messages with a "Direct" queue sending TCP to a mostly (but not entirely) unreachable server? Will the MainMsg queue instantly start to back up, if the action queue can't ship logs?
B) What is the maximum total queue size (memory and disk) for a "disk-assisted" queue that has a "QueueSize" of 50000, and no "QueueMaxDiskSpace" set? Is it unlimited, that is to say, the "QueueSize" directive only applies to the in-memory portion?
C) Does "QueueDiscardMark" directive apply to total queued messages (memory and disk) or only to the in-memory portion?