1

Recently I started to have very frequent "connection reset by peer" on calls to an external provider. My application (client) is a Go application, doing some simple POST to an external provider over HTTPS

Some context:

  • Go client application is running on docker.
  • The "connection reset by peer" is frequent, but erratic.
  • Provider says nothing is wrong on their end. Ok, RST can come from anywhere in between us.

The host instance ifconfig:

docker0   Link encap:Ethernet  HWaddr [REDACTED]
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

eth0      Link encap:Ethernet  HWaddr [REDACTED]
          inet addr:10.208.19.134  Bcast:10.208.19.255  Mask:255.255.255.128
          inet6 addr: fe80::8d:fdff:fe90:f410/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:37685240 errors:0 dropped:0 overruns:0 frame:0
          TX packets:37927624 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:13408927179 (12.4 GiB)  TX bytes:14057395581 (13.0 GiB)

I tried:

  • Ran tcpdump -vv -i eth0 -s 65535 -n dst host [[PROVIDER IP]] -w capture.cap & on host instance (EC2)
  • Opened capture with Wireshark an looked for tcp.flags.reset==1

Couldn't find anything. And am sure there were connection reset by peer during the capture (as we have logging in place). All I wanted to understand is where the RST is coming from (if that is possible).

So, what options do I have to look for the root cause of all these sudden errors?

0

To prove that the packets originate outside, try adding this to your router:

iptables -I FORWARD -i eth0 -p tcp --tcp-flags RST RST -j DROP

Be aware that RST packets are needed in normal operation and this could quickly hog your resources, so limit it to a particular --sport, even for testing, unless the router is used very lightly and only leave it in place for this test.

Replace FORWARD with INPUT if you are testing and filtering on a single machine. Replace eth0 with your actual internet connection.

For rudimentary confirmation, you can issue iptables -nvL | grep spt and see how the number at the start increased.

  • what do you mean adding this to your router ? Are you considering the host EC2 instance the router. When a packet reaches the host heading to the docker, does it hit FORWARD, is that it? – eduardev Jun 19 '18 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.