I am trying to improve performance of my servers while under load. I am on a 1Gbit line and don't have an issue sending or receiving at capacity while idle.
These servers fetch, process, and then return data to users. rx:tx is almost 1:1. I noticed issues with my downstream while serving traffic to users and have been able to reproduce the issue with iperf3/wget. I've seen the issues on the web servers, and also load balancers (haproxy) sitting in front of them.
If I run iperf3 -P 4 -R (4 parallel streams receiving data) or wget alone, it goes at 1Gbit. If I run them both at the same time total server usage drops under 500Mbit and bounces all over the place. Stopping one causes the other to go back up to 1Gbit.
Transmitting works just fine @ 1Gbit while under load, as well as downloading within the datacenter.
The machines are running dual Xeon L5630 with a mix of Broadcom and Intel NICs. I've tried a variety of tweaks to improve performance nothing seems to help.
I've been told that the issue is with processing interrupts quick enough. And in the process of researching this I have determined there's no one size fits all solution for performance optimization. I would love if I could figure this out myself, but would be happy to hire someone as well.
Does anyone have recommendations as to where to start?
Is it most likely interrupts? I loaded up irqbalance, then turned it off and assigned nic queues to different cores by hand.
Tried tweaking ring buffers and sysctl parameters and countless things. Not sure where to go from here.
Would appreciate any help.
Thanks!
Here's some output of the current sysctl tweaks, ethtool, system info: fs.file-max = 4953281 kernel.panic = 10 kernel.pid_max = 4194303 net.core.netdev_max_backlog = 65535 net.core.somaxconn = 65535 net.ipv4.ip_local_port_range = 1024 65535 net.ipv4.tcp_congestion_control = htcp net.ipv4.tcp_fin_timeout = 15 net.ipv4.tcp_max_syn_backlog = 65535 net.ipv4.tcp_max_tw_buckets = 1440000 net.ipv4.tcp_rfc1337 = 1 net.ipv4.tcp_rmem = 4096 87380 33554432 net.ipv4.tcp_slow_start_after_idle = 0 net.ipv4.tcp_synack_retries = 3 net.ipv4.tcp_syn_retries = 3 net.ipv4.tcp_tw_recycle = 0 net.ipv4.tcp_tw_reuse = 0 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 0 net.netfilter.nf_conntrack_tcp_timeout_established = 60 net.nf_conntrack_max = 800000 vm.overcommit_memory = 1
ethtool -i eth0 driver: igb version: 5.0.5-k firmware-version: 1.7.2 bus-info: 0000:02:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no
ethtool -S eth0|grep -v ": 0"
NIC statistics: rx_packets: 2489111423 tx_packets: 2608653369 rx_bytes: 2862551932229 tx_bytes: 1845742794294 rx_broadcast: 187617 tx_broadcast: 1 rx_missed_errors: 102 tx_tcp_seg_good: 2977380 rx_long_byte_count: 2862551932229 rx_fifo_errors: 488 tx_queue_0_packets: 324825232 tx_queue_0_bytes: 231519953294 tx_queue_0_restart: 1093 tx_queue_1_packets: 327114999 tx_queue_1_bytes: 229028327215 tx_queue_1_restart: 1085 tx_queue_2_packets: 329640953 tx_queue_2_bytes: 232597728986 tx_queue_2_restart: 538 tx_queue_3_packets: 325861970 tx_queue_3_bytes: 224621464319 tx_queue_3_restart: 1132 tx_queue_4_packets: 323541155 tx_queue_4_bytes: 228741007431 tx_queue_4_restart: 829 tx_queue_5_packets: 330539393 tx_queue_5_bytes: 235725634025 tx_queue_5_restart: 601 tx_queue_6_packets: 320796966 tx_queue_6_bytes: 222150341311 tx_queue_6_restart: 3705 tx_queue_7_packets: 326332702 tx_queue_7_bytes: 230422874757 tx_queue_7_restart: 953 rx_queue_0_packets: 317746938 rx_queue_0_bytes: 358943545445 rx_queue_0_csum_err: 80 rx_queue_1_packets: 310501723 rx_queue_1_bytes: 354800450890 rx_queue_1_drops: 386 rx_queue_1_csum_err: 92 rx_queue_2_packets: 304246179 rx_queue_2_bytes: 346027873804 rx_queue_2_csum_err: 76 rx_queue_3_packets: 307838541 rx_queue_3_bytes: 353811008950 rx_queue_3_csum_err: 149 rx_queue_4_packets: 310871839 rx_queue_4_bytes: 359145884376 rx_queue_4_csum_err: 69 rx_queue_5_packets: 317043674 rx_queue_5_bytes: 364702136702 rx_queue_5_csum_err: 96 rx_queue_6_packets: 312282264 rx_queue_6_bytes: 358149543411 rx_queue_6_csum_err: 91 rx_queue_7_packets: 308579883 rx_queue_7_bytes: 357015009991 rx_queue_7_csum_err: 115
Dual: model name : Intel(R) Xeon(R) CPU L5630 @ 2.13GHz (16 cores total over 2 physical processors) 48GB ram
(irqbalance in charge) /proc/interrupts: 64: 2 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 PCI-MSI-edge eth0 65: 5123 13851 0 0 6115 0 221059968 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth0-TxRx-0 66: 11343 0 0 0 0 0 0 0 0 0 0 219834257 5371 0 0 0 PCI-MSI-edge eth0-TxRx-1 67: 11054 0 0 0 217177269 0 0 0 0 0 15812 0 0 0 0 0 PCI-MSI-edge eth0-TxRx-2 68: 18435 0 0 0 0 0 0 220400185 0 0 0 0 6581 17136 0 0 PCI-MSI-edge eth0-TxRx-3 69: 11528 0 0 0 4829 0 0 0 0 0 0 0 11794 215368998 0 0 PCI-MSI-edge eth0-TxRx-4 70: 11594 0 0 0 0 220027516 0 0 0 0 0 0 4306 0 0 17515 PCI-MSI-edge eth0-TxRx-5 71: 18647 214603111 0 0 4567 0 0 0 0 0 0 0 0 0 11380 0 PCI-MSI-edge eth0-TxRx-6 72: 10266 0 0 216763545 0 0 0 0 0 13147 0 0 3458 0 0 0 PCI-MSI-edge eth0-TxRx-7
cat /proc/net/softnet_stat
22d4ab38 00000000 0000002b 00000000 00000000 00000000 00000000 00000000 00000000 0952491e 00000000 23aabf38 00000000 00000017 00000000 00000000 00000000 00000000 00000000 00000000 0833ce65 00000000 22f9f396 00000000 00000010 00000000 00000000 00000000 00000000 00000000 00000000 09934c0d 00000000 2391f6b5 00000000 0000000d 00000000 00000000 00000000 00000000 00000000 00000000 0845c0a9 00000000 01418f3b 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 01768b3e 00000000 00000004 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 015a3626 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 014e9af3 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0013582d 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0014919d 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0020ba05 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0146569d 00000000 00000007 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00165ba1 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 016872b1 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0013dc0b 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0014d61d 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
