I'm running a memcached server under heavy load from multiple web servers. While investigating some odd app level behaviour, we spotted that
netstat -s on the memcached box was showing a lot of "connections reset due to unexpected data".
The value goes up by several hundred per second under load.
What does this mean, and what can I do about it?
Ip: 147533037 total packets received 2 with invalid addresses 0 forwarded 0 incoming packets discarded 147533033 incoming packets delivered 154883954 requests sent out 5 outgoing packets dropped Icmp: 27 ICMP messages received 4 input ICMP message failed. ICMP input histogram: destination unreachable: 27 0 ICMP messages sent 0 ICMP messages failed ICMP output histogram: IcmpMsg: InType3: 27 Tcp: 8010699 active connections openings 8086268 passive connection openings 6146 failed connection attempts 5030 connection resets received 195 connections established 134422776 segments received 141824483 segments send out 32169 segments retransmited 5 bad segments received. 1711889 resets sent Udp: 13110231 packets received 0 packets to unknown port received. 9 packet receive errors 13129049 packets sent UdpLite: TcpExt: 251 resets received for embryonic SYN_RECV sockets 1949831 TCP sockets finished time wait in fast timer 4587602 time wait sockets recycled by time stamp 1005360 delayed acks sent 1055 delayed acks further delayed because of locked socket Quick ack mode was activated 259 times 650992 packets directly queued to recvmsg prequeue. 60990 bytes directly in process context from backlog 6099061 bytes directly received in process context from prequeue 47684327 packet headers predicted 550456 packets header predicted and directly queued to user 26594156 acknowledgments not containing data payload received 44593118 predicted acknowledgments 2 bad SACK blocks received 22777 congestion windows recovered without slow start after partial ack 1 timeouts after reno fast retransmit 19 timeouts after SACK recovery 1 timeouts in loss state 406 retransmits in slow start 28152 other TCP timeouts 259 DSACKs sent for old packets 4234 DSACKs received 1130771 connections reset due to unexpected data 143 connections reset due to early user close 482 connections aborted due to timeout TCPDSACKIgnoredNoUndo: 18 TCPSackShiftFallback: 4233 TCPChallengeACK: 5 TCPSYNChallenge: 5 IpExt: InOctets: -414538291 OutOctets: -630924797