2

Running logstash on Centos7 and I think the NIC might be saturated.

From the logstash server I can see the request q for servers shipping logs too it. But I'm not sure if the request q is high based on my settings or if tcp tuning would help me here.

some info:

sysctl -a | grep mem
net.core.optmem_max = 20480
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.wmem_default = 212992
net.core.wmem_max = 212992
net.ipv4.igmp_max_memberships = 20
net.ipv4.tcp_mem = 227763       303685  455526
net.ipv4.tcp_rmem = 4096        87380   6291456
net.ipv4.tcp_wmem = 4096        16384   4194304
net.ipv4.udp_mem = 229686       306249  459372
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
vm.lowmem_reserve_ratio = 256   256     32
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.nr_hugepages_mempolicy = 0
vm.overcommit_memory = 0

netstat -na --tcp | grep :9123
tcp6       0      0 :::9123                 :::*                    LISTEN
tcp6  247834      0 192.168.123.123:9123     10.16.1.82:52289       ESTABLISHED
tcp6  241242      0 192.168.123.123:9123     10.31.31.232:65293     ESTABLISHED
tcp6       0      0 192.168.123.123:9123     10.16.1.198:53693      ESTABLISHED
tcp6       0      0 192.168.123.123:9123     10.16.1.198:56751      ESTABLISHED
tcp6  331114      0 192.168.123.123:9123     10.31.35.157:53998     ESTABLISHED
tcp6  256047      0 192.168.123.123:9123     10.16.2.155:52221      ESTABLISHED
tcp6  240498      0 192.168.123.123:9123     10.19.5.166:51805      ESTABLISHED
tcp6  312648      0 192.168.123.123:9123     10.18.16.155:57975     ESTABLISHED
tcp6  234321      0 192.168.123.123:9123     10.18.19.242:51664     ESTABLISHED
tcp6  255079      0 192.168.123.123:9123     10.19.4.51:51458       ESTABLISHED
tcp6  256328      0 192.168.123.123:9123     10.18.45.89:56821      ESTABLISHED
tcp6  237167      0 192.168.123.123:9123     10.18.33.26:49278      ESTABLISHED
tcp6  248204      0 192.168.123.123:9123     10.18.30.250:54267     ESTABLISHED
tcp6  248573      0 192.168.123.123:9123     10.16.1.198:57522      ESTABLISHED
tcp6  238348      0 192.168.123.123:9123     10.18.11.169:55147     ESTABLISHED
tcp6  243762      0 192.168.123.123:9123     10.31.22.48:60425      ESTABLISHED
tcp6  258035      0 192.168.123.123:9123     10.31.46.31:60432      ESTABLISHED
tcp6  241863      0 192.168.123.123:9123     10.18.45.113:63376     ESTABLISHED
tcp6  327889      0 192.168.123.123:9123     10.18.3.219:58640      ESTABLISHED
tcp6  317363      0 192.168.123.123:9123     10.31.37.249:65162     ESTABLISHED
tcp6  252394      0 192.168.123.123:9123     10.16.1.92:56360       ESTABLISHED
tcp6  326401      0 192.168.123.123:9123     10.31.17.74:53948      ESTABLISHED
tcp6       0      0 192.168.123.123:9123     10.16.2.12:53781       ESTABLISHED
tcp6  244669      0 192.168.123.123:9123     10.18.18.100:49281     ESTABLISHED
tcp6  250264      0 192.168.123.123:9123     10.18.32.116:56795     ESTABLISHED
tcp6       0      0 192.168.123.123:9123     10.16.1.82:49304       ESTABLISHED
tcp6  310864      0 192.168.123.123:9123     10.18.11.25:64230      ESTABLISHED
tcp6  247973      0 192.168.123.123:9123     10.18.22.230:55209     ESTABLISHED
tcp6       0      0 192.168.123.123:9123     10.17.1.8:51741        ESTABLISHED
tcp6       0      0 192.168.123.123:9123     10.16.2.12:54507       ESTABLISHED
tcp6  251552      0 192.168.123.123:9123     10.18.24.83:63499      ESTABLISHED
tcp6  251481      0 192.168.123.123:9123     10.16.2.72:57268       ESTABLISHED
tcp6       0      0 192.168.123.123:9123     10.16.1.198:53406      ESTABLISHED
tcp6  312263      0 192.168.123.123:9123     10.19.12.239:52173     ESTABLISHED
tcp6  238878      0 192.168.123.123:9123     10.18.5.198:57978      ESTABLISHED
tcp6  322460      0 192.168.123.123:9123     10.18.5.124:53117      ESTABLISHED
tcp6       0      0 192.168.123.123:9123     10.16.2.72:54883       ESTABLISHED
tcp6  237717      0 192.168.123.123:9123     10.16.2.12:56387       ESTABLISHED
tcp6  315963      0 192.168.123.123:9123     10.18.26.44:49197      ESTABLISHED
tcp6  248914      0 192.168.123.123:9123     10.18.41.101:51859     ESTABLISHED
tcp6       0      0 192.168.123.123:9123     10.16.2.155:49303      ESTABLISHED
tcp6  316994      0 192.168.123.123:9123     10.18.44.120:49375     ESTABLISHED
tcp6  236421      0 192.168.123.123:9123     10.31.43.130:51590     ESTABLISHED
tcp6  240929      0 192.168.123.123:9123     10.17.1.114:63546      ESTABLISHED
tcp6  306346      0 192.168.123.123:9123     10.17.2.159:61633      ESTABLISHED
tcp6  239360      0 192.168.123.123:9123     10.18.39.43:54080      ESTABLISHED
tcp6  245361      0 192.168.123.123:9123     10.19.13.107:52629     ESTABLISHED
tcp6  244398      0 192.168.123.123:9123     10.18.11.195:53257     ESTABLISHED
1
  • Please Show ''ethtool -s | grep -vw 0'' to get a clearer picture.
    – Nils
    Apr 20, 2017 at 20:12

1 Answer 1

5

You could certainly tune your NIC, the RedHat Network Performance Tuning guide provides a fairly complete (and somewhat recent, c. 2015) set of items to consider.

That document suggests 16384 349520 16777216 for TCP rmem, for example (whether you need to tune wmem depends on your setup - from what you share, that doesn't seem necessary).

However, to my mind, this would point to logstash being a bottleneck - your receive queues are buffers for the process listening on a given port (logstash), and so increasing those buffers doesn't really address the underlying issue. It looks to me like logstash is already (comparatively) slow, and more network buffers doesn't sound like it would make it faster.

Check if your logstash config/parsers can be optimised for your situation/needs (if you use regexes, that might be a good area to investigate).

You might need to increase the resources available to logstash, and/or consider tiering your logstash infrastructure to lighten the load on your client/source-facing instance.

In other words, have a front end receiving data and doing little to no actual work (just add some tags/info, and maybe some routing to specific processors), and then have that front end pass data through for further processing to another instance (or to a message queue like Kafka, for example).

Notably, if you use DNS and geoIP plugins, it generally makes sense to do those kinds of things in a central place, once events/logs/data are fully parsed).

1
  • What makes you think logstash is slow?
    – Nils
    Apr 20, 2017 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.