rsyslog cannot send logs to logstash

Question

Here is my rsyslog.conf (nothing in /etc/rsyslog.d/ folder):

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)

#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf


#### RULES ####

# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog

# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

 # SEND ALL THE MESSAGES TO CENTRAL LOGSTASH SERVER
#
*.* @10.38.105.18:5000

After restarting rsyslog service, I try the following to test:

logger  "Host1 kernel: device eth0 left promiscuous mode"

And tcpdump on the logstash server side (10.38.105.18) to see anything goes through (10.36.52.81 is the server that is sending the logs):

[~] # tcpdump src host 10.36.52.81 -nn -XXX
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

Nothing.

When I send data through nc:

[~] # echo -n "Host1 kernel: device eth0 left promiscuous mode" | nc -4u -w1 10.38.105.18 5000

I can see that it goes through:

03:04:19.521433 IP 10.36.52.81.42159 > 10.38.105.18.5000: UDP, length 47
        0x0000:  0050 56a6 4600 0026 981c bd42 0800 4500  .PV.F..&...B..E.
        0x0010:  004b 9560 4000 3e11 f594 0a24 3451 0a26  .K.`@.>....$4Q.&
        0x0020:  6912 a4af 1388 0037 01da 486f 7374 3120  i......7..Host1.
        0x0030:  6b65 726e 656c 3a20 6465 7669 6365 2065  kernel:.device.e
        0x0040:  7468 3020 6c65 6674 2070 726f 6d69 7363  th0.left.promisc
        0x0050:  756f 7573 206d 6f64 65                   uous.mode

So, this means that there is no obstacles in the path. What am I doing wrong?

Update:

did a tcpdump on clientside and it seems that the client is sending the log to Logstash:

03:30:20.073608 IP 10.36.52.81.39653 > 10.38.105.18.5000: UDP, length 88
        0x0000:  001b 1700 0125 0050 56a6 6b5e 0800 4500  .....%.PV.k^..E.
        0x0010:  0074 0000 4000 4011 88cc 0a24 3451 0a26  .t..@.@....$4Q.&
        0x0020:  6912 9ae5 1388 0060 b21e 3c31 333e 4465  i......`..<13>De
        0x0030:  6320 2031 2030 333a 3330 3a32 3020 6d73  c..1.03:30:20.ms
        0x0040:  7070 3170 6573 6c6f 6730 3031 2072 6f6f  pp1peslog001.roo
        0x0050:  743a 2048 6f73 7431 206b 6572 6e65 6c3a  t:.Host1.kernel:
        0x0060:  2064 6576 6963 6520 6574 6830 206c 6566  .device.eth0.lef
        0x0070:  7420 7072 6f6d 6973 6375 6f75 7320 6d6f  t.promiscuous.mo
        0x0080:  6465                                     de

There is indeed a firewall in between but why is it that the tcpdump on the logstash side does not show the packet when I send the message through "logger" command but it shows it when I send it through netcat? I'm confused.

Answer 1

I can't say for certain why you config doesn't work, but I use rsyslog to send messages to logstash and I have no issues. If you can send traffic to logstash using nc, and logstash is set up to receive logs on both UDP and TCP, and you're using the same ports, as you say then this should work.

Are you able to simplify your setup? You say there is a firewall between the 2 machines, can you disable this for testing purposes or reproduce on a VM without firewalls?

In my setup I use the following config, this should be a very reliable way to send logs. I would recommend using TCP (i.e. @@ over @) as that should be more reliable than UDP and if it fails to send with the following config the messages will be queued on disk and in memory, so you can work on fixing it without losing messages:

# start forwarding rule 2
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName logstash # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionResumeInterval 10 # Attempt resuming after 10 seconds
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
$ActionQueueMaxDiskSpace 200M #Limit the amount of space used on disk to 200M
$ActionQueueSize 20000 # Limit the amount of messages to queue in memory to 20000, average size is 512 bytes, so shouldn't be greater than 10M

*.* @@logstash:5114;RSYSLOG_SyslogProtocol23Format # end forwarding rule 2

Remember if you're sending with UDP and the messages don't get through, the sender doesn't know and will keep losing messages until someone spots the issue.