2

I'm trying to setup centralized logging with Logstash, Elasticsearch and Kibana and am having trouble getting logs to my log server.

Logstash is listening on TCP 5000 and is successfully receiving logs from one of my servers but not the rest. I can telnet to my log server on 5000 and Logstash is picking up the messages so I believe the issues is with rsyslog sending the logs.

I'm on CentOS 6.5 and have added the following to /etc/rsyslog.conf:

$WorkDirectory /var/lib/rsyslog # where to place spool files
*.* @@logs.<domain_removed>.com:5000

rsyslog restarted with no errors after making the change but nothing is being received by Logstash. The spool directory is empty so I don't believe their are send failures.

May 16 20:46:10 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-    pid="5097" x-info="http://www.rsyslog.com"] exiting on signal 15.
May 16 20:46:10 kernel: imklog 5.8.10, log source = /proc/kmsg started.
May 16 20:46:10 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3556" x-info="http://www.rsyslog.com"] start

Any ideas on how I can track down the error?

EDIT: This issue was caused by SELinux

  • can you show the rules in rsyslog.conf on your server? – phoops May 16 '14 at 21:06
  • Is logstash configured to listen for logs from rsyslog? – John May 16 '14 at 21:07
  • Have you run tcpdump on both ends to see if there is any traffic? – yoonix May 16 '14 at 21:10
  • Edvinas: Rsyslog on the server is not setup to listen for logs because Logstash should receive them. John: Logstash is configured to listen on TCP 5000. It is receiving logs from one of my other servers using the same setup. I can post the config if it's helpful. yoonix: I have not yet, thanks I'll will do that. – Merch May 16 '14 at 21:26
  • No outbound traffic to my log server was obeserved with TCP dump. On the log server the only inbound TCP traffic observed was from the one server where logs are being successfully stored in elasticsearch. – Merch May 16 '14 at 22:34
1

SELinux was blocking rsyslog from sending on port 5000. By default SELinux only allowes rsyslog to send traffic on UDP 514.

I added an exception to SELinux with:

$sudo semanage port -a -t syslogd_port_t -p tcp 5000
  • This is part of "policycoreutils-python" package, if you happen to have a minimal cloud server. :) – TommyTheKid Jan 7 '15 at 23:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.