We have a setup with one Logstash server (1.4.2, using embedded Elasticsearch) to receive logs from a number of other clients (via TCP and from their rsyslogd instances). One issue that now happens repeatedly is that one by one messages from hosts aren't received anymore, however there are no error messages anywhere to be seen.

It basically looks like the Logstash host can't process the amount of incoming messages and the connections are left in a weird state; I did lsof/strace on the client rsyslogd instances and also captured some traffic on both clients and server and it seems that while the clients still have a connection open:

rsyslogd 30088 syslog    1u  IPv4           14878202       0t0        TCP 10.129.X.X:47492->10.129.X.X:5544 (ESTABLISHED)

, that connection is basically broken as the server (port 5544) sends TCP Zero Window messages, which, as Wireshark tells me, is basically Logstash not keeping up (CPU usage on the host isn't always at max, but regularly at about 75% on all four cores).

My question is thus: Is there are flag / setting / ... to either get the rsyslogd daemons to back off / reconnect in such a situation, or for Logstash to properly close connections if it can't keep up? (Is this a known problem?, because I wasn't able to find any relevant links.)

Edit: In the meantime we've changed the input plugin to tcp instead of syslog, which, together with manual grok parsing, seems to handle the load better. I'd still like to understand the original problem though.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.