6

I'm running a memcached server under heavy load from multiple web servers. While investigating some odd app level behaviour, we spotted that netstat -s on the memcached box was showing a lot of "connections reset due to unexpected data".

The value goes up by several hundred per second under load.

What does this mean, and what can I do about it?

Example output:

Ip:
    147533037 total packets received
    2 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    147533033 incoming packets delivered
    154883954 requests sent out
    5 outgoing packets dropped
Icmp:
    27 ICMP messages received
    4 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 27
    0 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
IcmpMsg:
        InType3: 27
Tcp:
    8010699 active connections openings
    8086268 passive connection openings
    6146 failed connection attempts
    5030 connection resets received
    195 connections established
    134422776 segments received
    141824483 segments send out
    32169 segments retransmited
    5 bad segments received.
    1711889 resets sent
Udp:
    13110231 packets received
    0 packets to unknown port received.
    9 packet receive errors
    13129049 packets sent
UdpLite:
TcpExt:
    251 resets received for embryonic SYN_RECV sockets
    1949831 TCP sockets finished time wait in fast timer
    4587602 time wait sockets recycled by time stamp
    1005360 delayed acks sent
    1055 delayed acks further delayed because of locked socket
    Quick ack mode was activated 259 times
    650992 packets directly queued to recvmsg prequeue.
    60990 bytes directly in process context from backlog
    6099061 bytes directly received in process context from prequeue
    47684327 packet headers predicted
    550456 packets header predicted and directly queued to user
    26594156 acknowledgments not containing data payload received
    44593118 predicted acknowledgments
    2 bad SACK blocks received
    22777 congestion windows recovered without slow start after partial ack
    1 timeouts after reno fast retransmit
    19 timeouts after SACK recovery
    1 timeouts in loss state
    406 retransmits in slow start
    28152 other TCP timeouts
    259 DSACKs sent for old packets
    4234 DSACKs received
    1130771 connections reset due to unexpected data
    143 connections reset due to early user close
    482 connections aborted due to timeout
    TCPDSACKIgnoredNoUndo: 18
    TCPSackShiftFallback: 4233
    TCPChallengeACK: 5
    TCPSYNChallenge: 5
IpExt:
    InOctets: -414538291
    OutOctets: -630924797
0

From https://serverfault.com/a/527670/442063:

Do you increase the FD limit? You can obtain some info here http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.