I've done the following tuning to try to eliminate
packets pruned from receive queue that I see in netstat, but I'm still seeing a couple hundred pruned every few minutes:
- increased net.core.rmem_max to 33554432
- increased net.core.rmem_default to 12582912
- increased net.core.wmem_max to 16777216
- increased net.ipv4.tcp_rmem to 4096 12582912 33554432
- increased net.core.netdev_max_backlog to 5000
- increased net.core.dev_weight to 256
- increased net.ipv4.tcp_max_syn_backlog to 8192
- increased net.core.netdev_budget to 400
- enabled rps and rfs, setting rps_sock_flow_entries to 32768, rps_flow_cnt to 8192 (4 receive queues), and rps_cpus affinity to ff (all cores)
- increased net.core.somaxconn to 8192
- set the application's rcv_buf size to 33554432
- set the application's tcp connection listen backlog to 4096
This is a graphite system running carbon-cache, carbon-c-relay, httpd and memcached - on an i3.2xlarge instance, doing a network throughput of about 250million bytes/minute.
cpu usage is reasonable, fluctuating between 1% and 30% idle, with about 10% average i/o wait.
iostat -mx 1 shows %util fluctuates btween 40% to 95%.
vmstat shows the number of processes waiting for run time fluctuates between 1 and 12.
total used free shared buffers cached
Mem: 61444 60976 467 0 7442 40391
-/+ buffers/cache: 13142 48302
Swap: 0 0 0
Any thoughts on how to determine why packets are being pruned at a rate of about 25/minute and how to fix this?