We have a setup with one Logstash server (1.4.2, using embedded Elasticsearch) to receive logs from a number of other clients (via TCP and from their
rsyslogd instances). One issue that now happens repeatedly is that one by one messages from hosts aren't received anymore, however there are no error messages anywhere to be seen.
It basically looks like the Logstash host can't process the amount of incoming messages and the connections are left in a weird state; I did
strace on the client
rsyslogd instances and also captured some traffic on both clients and server and it seems that while the clients still have a connection open:
rsyslogd 30088 syslog 1u IPv4 14878202 0t0 TCP 10.129.X.X:47492->10.129.X.X:5544 (ESTABLISHED)
, that connection is basically broken as the server (port
5544) sends TCP Zero Window messages, which, as Wireshark tells me, is basically Logstash not keeping up (CPU usage on the host isn't always at max, but regularly at about 75% on all four cores).
My question is thus: Is there are flag / setting / ... to either get the
rsyslogd daemons to back off / reconnect in such a situation, or for Logstash to properly close connections if it can't keep up? (Is this a known problem?, because I wasn't able to find any relevant links.)
Edit: In the meantime we've changed the input plugin to
tcp instead of
syslog, which, together with manual
grok parsing, seems to handle the load better. I'd still like to understand the original problem though.