2

I've got a host that has experienced a sudden increase in the rate of failed TCP connections. It's gone from something like 0.3/sec to 1.5/sec.

enter image description here

Regardless of the severity of my particular issue, what's the basic process for finding root cause?

5
  • How do you know that the rate of "failed TCP connections" went from 0.3/sec to 1.5/sec?
    – Mxx
    Jan 20, 2014 at 19:42
  • I've got monitoring (LogicMonitor) that graphs the rate. As far as I can tell, their measurement is just finding a simple rate of change from stats that match what I find here: netstat -st | grep failed connection attempts.
    – Larsenal
    Jan 20, 2014 at 19:52
  • Also, perhaps someone could help me understand why my question wasn't great. I'm a regular on StackOverflow, but not so much here.
    – Larsenal
    Jan 20, 2014 at 19:55
  • It would be better if you explained what the criterion for failed is, and what the impact is. "I see vague errors in my logs; how do I fix them" isn't really great, because it doesn't tell us what the meaning is, and you should try to fix the problem (not the errors). Jan 21, 2014 at 5:41
  • Understand. I'm much more dev than ops, so I perhaps falsely imagined "failed TCP connections" was a reasonable starting spot. Is the stat coming out of netstat -st specific enough?
    – Larsenal
    Jan 21, 2014 at 5:48

1 Answer 1

1

Consider contacting your hosting company/Network Admins and asking them about these failed connections. To avoid finger-pointing tell them that you see it with netstat command in addition to LogicMonitor(which shows longer term graphs and sudden increase). They might have more information into what's going on their network. If this server is protected behind a firewall/proxy, make sure that only proper traffic is allowed to pass through. (It is wild out there on the internet.)

Additionally, consider capturing some traffic using tcpdump(make sure to capture full packets, not just headers) or similar compatible tool and loading it into Wireshark. Wireshark will allow you to explore your raw captured packets. See if you can spot any traffic pattern that is not supposed to be there. Perhaps many connections from an unknown IP to an unused port or some misbehaving local networking device. Wireshark also has some general statistics about the captured, perhaps you'll spot something useful there.

Not the answer you're looking for? Browse other questions tagged .