0

I'm learning Docker, installing docker 1.12.6 on my Red Hat Enterprise Linux 7.4 machine (Linux 3.10.0-693) and going through the 'getting started' exercise.

It's been a bit of a hassle getting it to work through my company's web proxy. (I don't know what sort of web proxy we have.)

I had to set up HTTP_PROXY/HTTPS_PROXY environment variables for dockerd, so it would use the proxy; and then add the company's CA certificates to the system's pki trusted store (since the web proxy does 'man in the middle' interposing of certificates for HTTPS connections). But finally I was able to load docker images from the docker.io registry; downloads to my machine all working fine.

But my attempts to do a 'docker push' to docker.io fail. Even with a tiny 13KB docker image the upload hangs and eventually a 'connection reset by peer' error is reported.

Doing a tcpdump the basic problem seems pretty clear. After a bit of traffic, setting things up, my docker client machine sends four HTTP packets in a row to the web proxy ... and it only sends three ACK packets back. One of the four outoing packets is never acknowledged. After this both sides re-send their last ACKs for a few minutes before one of them sends a FIN.

This is the interesting tail end of the tcpdump output:

...

#   client sends HTTP to proxy, it sends an ACK back, everything fine.

17:17:17.132874 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [P.], seq 326:484, ack 3372, win 37648, options [nop,nop,TS val 175751308 ecr 3912953706], length 158: HTTP
17:17:17.133929 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [.], ack 484, win 4863, options [nop,nop,TS val 3912953725 ecr 175751308], length 0

#   Now the proxy relays a HTTP reply from upstream in two packets, my client sends a single ACK for both.

17:17:17.138783 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [P.], seq 3372:3378, ack 484, win 4863, options [nop,nop,TS val 3912953729 ecr 175751308], length 6: HTTP
17:17:17.138801 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [P.], seq 3378:3423, ack 484, win 4863, options [nop,nop,TS val 3912953729 ecr 175751308], length 45: HTTP
17:17:17.138941 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [.], ack 3423, win 37648, options [nop,nop,TS val 175751314 ecr 3912953729], length 0

#   Now we have the problem.  My client sends four packets in a row to the proxy, with sequence numbers 1692, 3140, 3605, 3639 ...

17:17:17.139139 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [P.], seq 484:1692, ack 3423, win 37648, options [nop,nop,TS val 175751314 ecr 3912953729], length 1208: HTTP
17:17:17.139157 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [.], seq 1692:3140, ack 3423, win 37648, options [nop,nop,TS val 175751314 ecr 3912953729], length 1448: HTTP
17:17:17.139160 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [P.], seq 3140:3605, ack 3423, win 37648, options [nop,nop,TS val 175751314 ecr 3912953729], length 465: HTTP
17:17:17.139179 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [P.], seq 3605:3639, ack 3423, win 37648, options [nop,nop,TS val 175751314 ecr 3912953729], length 34: HTTP

#   ... and the proxy sends separate ACKs for just three of the four, missing 3140

17:17:17.140457 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [.], ack 1692, win 6071, options [nop,nop,TS val 3912953731 ecr 175751314], length 0
17:17:17.140505 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [.], ack 3605, win 7984, options [nop,nop,TS val 3912953731 ecr 175751314], length 0
17:17:17.140513 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [.], ack 3639, win 8018, options [nop,nop,TS val 3912953731 ecr 175751314], length 0

#   Everything hangs for 30 seconds; client resends its last ACK, proxy retransmits its last ACK.

17:17:47.142582 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [.], ack 3423, win 37648, options [nop,nop,TS val 175781318 ecr 3912953731], length 0
17:17:47.143579 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [.], ack 3639, win 8018, options [nop,nop,TS val 3912983735 ecr 175751314], length 0

#   Everything hangs for another 30 seconds; client resends its last ACK, so does proxy

17:18:17.152561 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [.], ack 3423, win 37648, options [nop,nop,TS val 175811328 ecr 3912983735], length 0
17:18:17.153663 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [.], ack 3639, win 8018, options [nop,nop,TS val 3913013746 ecr 175751314], length 0

#  ... and so forth

I'm stymied by this (but I am no networking expert). In the second set of packets above, starting from time 17:17:17.138783, the proxy sends two packets, my client then sends a single ACK packet back, acknowledging the greater/last of the sequence numbers, everything is fine. You'd think my client would be able to do the same thing when the proxy skips an explicit ACK of sequence number 3140 but sends two more for the greater sequence numbers of the last two packets of the group of four. But as you can see things halt, nothing happens until my client resends its last ACK every 30 seconds.

I'd appreciate any and all advice regarding the following questions:

Q1. Can anyone give me advice on which of the two machines is doing things 'wrong'? Out of curiosity I'm intrigued as to what the TCP/IP standard requires and which one is slipping up.

Q2. Are there configuration settings I can tweak to get Linux to avoid sending the four packets at once, or otherwise fix the problem? Slow things down, serialise transmission, avoid sending queued packets until the previous transmission is acknowledged by the proxy? Thus stepping around this 'missing ACK packet' issue which I assume is the heart of my problem?

Q3. Is there any other way I can fix this and get docker working behind my corporate firewall? Anyone seen anything like this before, is it a common problem?

I can't affect/change the firewall in any way but I'm in control of the Linux docker client machine.

Many thanks for any advice!

0

This is normal and nothing is wrong. TCP does not ack individual packets/segments; it acks byte ranges. See wikipedia. At 17:17:17.138941 client ack=3423 acknowledges the data from both frames 17:17:17.138783 and 17:17:17.138801 . Similarly at 17:17:17.140505 server ack=3605 acknowledges both 17:17:17.139157 and 17:17:17.139160.

The subsequent frames every 30 seconds must be keepalives, although that is an unusually short interval, because the window is open in both directions and no data is sent. And moreover you say the connection is then closed with FIN (which should flow in both directions, not just one); if either endpoint had outstanding (unacked) data it would have used RST instead.

Your client's request has been delivered successfully and correctly to the proxy. If the proxy fails to send that request on the server and obtain and relay a response, it is not due to a problem on the link between you and the proxy.

  • Being relatively ignorant about the mysteries of TCP I thought there was a 'missing ACK' or some such but over the last few days I've come around to realising what you've said, that there is nothing wrong in the packet flow. Particularly since changing the MTU (per the answer by womble) resulted in a 3-ACKs-for-3-packets exchange at the 'hanging' point. I don't know how keepalive is implemented; happy to assume it's such. Your observation about FIN versus RST is further proof that things are normal, thank you! It must be something else ... – alex8275 Jan 19 '18 at 13:08
0

I'm pretty sure you're banging up against MTU limits (which suggests a pretty shoddy local network and/or MitM proxy, but that's not a surprise). You can limit the size of the packets your machine sends by reducing the max MTU on the NIC, like this:

ip link set mtu 1280 dev eth0

(Replace eth0 with the NIC's name)

A subsequent tcpdump should show the packet lengths never going above about that number (a few extra bytes for framing and protocol overhead should be fine). If that clears up the problem, you can either just hardcode that max MTU into the machine's config, or (preferably) use iptables to limit the MSS ("maximum segment size") for connections to just the misbehaving proxy.

  • I tried MTU sizes of 800, the default 1500 and 2500 and nothing changed; the 'docker push' attempts still hang, at exactly the same place. Thanks for the idea; the tcpdump with 2500 simplified things, three packets sent out by the client, three ACKs returned by the proxy, a simpler scenario than when one of those packets had been split in two. – alex8275 Jan 18 '18 at 9:23
  • When the MTU was 800, were the packets sent smaller, around 800 bytes? – womble Jan 18 '18 at 11:48
  • yes, the observed packet sizes varied according to the MTU settings, but the hanging problem remained. – alex8275 Jan 19 '18 at 13:03
  • Wow, definitely not an MTU issue then. You've got a stone-cold whodunnit on your hands. – womble Jan 19 '18 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.