3

I've currently got Docker running on a couple of OpenStack-hosted VMs.

When attempting to make HTTP requests from within containers on those machines, the request almost invariably hangs waiting for a response. When making HTTPS requests, the request invariably times out because it fails to complete the TLS handshake. In either case, it seems as though the requests are failing after only receiving a couple of hundred bytes of response data (if any). The same requests complete without any trouble from the host machine (i.e it isn't a host-level firewall that's the problem).

I've come across this (and some related blog posts) that suggest it may be due to the Docker network bridge having a different MTU to the host's network interface, but I've verified that both MTUs are the same (1500 bytes in each case). I've also tried toying with the --mtu switch to the Docker daemon to see if I could get it to work, with no success.

I've also come across a few similar cases that suggest that this may be due to TCP checksum and/or segmentation offloading (e.g. here), but no amount of toying around with ethtool -K {interface} tx off rx off or similar produce any positive result.

The behaviour seems to be specific to the network bridge - using --net=host when running containers does solve the problem. However, for security reasons I want to avoid having to use this workaround in our production system.

Note also that the exact same setup (same Docker version, same configuration parameters) works fine on my development machine or when running on an instance hosted in AWS - whatever the problem is, it seems to only manifest itself when running under OpenStack.

For reference, I'm using the following command for testing:

docker run -it alpine:3.3 wget http://ipv4.download.thinkbroadband.com/5MB.zip

This should (in theory) download a 5MB test file when run. However, the wget command just hangs. It's also probably worth noting that it's not the guest OS that's the issue - using a Ubuntu image (for example) still exhibits the same behaviour.

A colleague and I also ran some tests transferring files using netcat. In those tests, we were able to successfully transfer files from the container to our test server, but attempting to transfer in the opposite direction failed. We also used tcpdump to capture network activity during these tests - during the failed tests, the logs showed some packets being received but significantly fewer than expected.

Also, in case it's relevant, here's the output of docker version on the system in question:

Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:22:43 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:22:43 2016
 OS/Arch:      linux/amd64

The output of uname -a on the host is as follows:

Linux docker-builder-1 4.4.0-38-generic #57~14.04.1-Ubuntu SMP Tue Sep 6 17:20:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

UPDATE

It turns out that everything works as expected if Docker is hosted under a Ubuntu 16.04 image.

At this point I can move ahead by rebuilding all my VMs to use a Ubuntu 16.04 image, but I still don't know what the original issue was. I'm still interested in hearing any suggestions as to what the cause of the original problem was, and how to fix it.

1 Answer 1

1

TL;DR

Adapt dockerd MTU flag to your VM's MTU.

Example

One of My OpenStack instance's OS is ubuntu16.04.

  1. Check your vm's MTU with ifconfig(1450 in my case)
  2. cp /lib/systemd/system/docker.service /etc/systemd/system/docker.service
  3. vi /etc/systemd/system/docker.service
  4. ExecStart=/usr/bin/docker daemon -H fd:// --mtu 1450
  5. systemctl daemon-reload
  6. service docker restart
1
  • 1
    Please re-read the question (3rd paragraph in particular): I've already tried this without success.
    – Mac
    Mar 16, 2017 at 22:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .