Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Having recently had a network failure that caused a major cascading series of other issues, I've been diving into our rsyslog.conf to try identify opportunities for more stable deployment.

We have all our application servers running rsyslog, forwarding over tcp to a couple of aggregation servers. Yesterday, when one of those servers was nearly unreachable due to high packet loss, the rest of the environment ground to a halt. I think I have discovered a couple of configs that don't make sense.

Here is our client config:

# modules
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)

$KLogPath /proc/kmsg

# global app configs
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0644
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
$PreserveFQDN on

# queue configs
$MainMsgQueueType LinkedList
$MainMsgQueueSize 50000
$ActionQueueType Direct

$MainMsgQueueDiscardMark 300000
$MainMsgQueueDiscardSeverity 4

$MainMsgQueueFileName mainmsgqueue
$MainMsgQueueMaxFileSize 200m
$MainMsgQueueLowWaterMark 10000
$MainMsgQueueHighWaterMark 40000

$MainMsgQueueTimeoutEnqueue 1
$ActionQueueTimeoutEnqueue 1

# Log locally.
authpriv.*                  /var/log/auth.log
*.*;auth,authpriv.none      -/var/log/syslog

# Send all components to remote log servers
*.* @@<server 1 IP>
& @@<server 2 IP>

My questions are:

A) On the client servers, what will happen to messages with a "Direct" queue sending TCP to a mostly (but not entirely) unreachable server? Will the MainMsg queue instantly start to back up, if the action queue can't ship logs?

B) What is the maximum total queue size (memory and disk) for a "disk-assisted" queue that has a "QueueSize" of 50000, and no "QueueMaxDiskSpace" set? Is it unlimited, that is to say, the "QueueSize" directive only applies to the in-memory portion?

C) Does "QueueDiscardMark" directive apply to total queued messages (memory and disk) or only to the in-memory portion?

share|improve this question

1 Answer 1

Got more assistance on the rsyslog forums, and cross-posting here, to help anybody in the future.

Rsyslog forum thread: [http://kb.monitorware.com/post24885.html#p24885]

Answers: 1.) rsyslog targets are processed sequentially, so if "server A" is having packet loss, it will slow down delivery to the rest of the targets. This can be mitigated in current versions of rsyslog by setting additional "action queues", which parallelize the output.

2.) The "QueueSize" for a disk-assisted queue only applies to the in-memory portion. The disk queue will be unlimited unless you specify "QueueMaxDiskSpace"

3.) I didn't get a conclusive answer whether the discard message mark would be applied to memory and disk portions of a disk-assisted queue. Logic would suggest that it works on all new incoming messages, regardless of where they're stored, but that's just a guess.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.