How do I expose the docker API over TCP?

Question

I am using portainer and am unable to manage remote endpoints. I tried using the command line to connect to remote docker nodes, but got a message Cannot connect to the Docker daemon at tcp://<remote_ip>:<port>. Is the docker daemon running?.

Yes, they are running. I have added myself to the docker group and can access docker by SSHing into the nodes. However I cannot access any docker nodes remotely.

I modified /etc/default to add / uncomment DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock"

I also modified /etc/init.d/docker and /etc/init/docker.conf to include DOCKER_OPTS="-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock".

I restarted the docker service, logged out and logged in multiple times in the process, but still cannot connect to the remote node. I cannot even connect to the local node by passing the IP.

What did I miss out? What configuration in what file exposes the API over TCP?

user@hostname:~$ docker -H tcp://<REMOTE_IP>:2375 info
Cannot connect to the Docker daemon at tcp://<REMOTE_IP>:2375. Is the docker daemon running?
user@hostname:~$ docker -H tcp://127.0.0.1:2375 info
Cannot connect to the Docker daemon at tcp://127.0.0.1:2375. Is the docker daemon running?
user@hostname:~$ docker -H tcp://<LOCAL_IP>:2375 info
Cannot connect to the Docker daemon at tcp://<LOCAL_IP>:2375. Is the docker daemon running?
user@hostname:~$

Edit: Running ps aux | grep -i docker returns this -

root      3581  0.1  0.2 596800 41540 ?        Ssl  04:17   0:35 /usr/bin/dockerd -H fd://
root      3588  0.0  0.0 653576 14492 ?        Ssl  04:17   0:18 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc

Answer 1

I found a solution thanks to Ivan Krizsan's post.

I had to edit /lib/systemd/system/docker.service on my Ubuntu 16.04.2 LTS system to modify the line

ExecStart=/usr/bin/docker daemon -H fd:// -H tcp://0.0.0.0:

then

sudo systemctl daemon-reload
sudo systemctl restart docker.service

and everything worked :-). The next step is to figure out how to protect the docker daemon form being hijacked.

Answer 2

The /etc/default directory is where distribution maintainers put their configuration files. If you install Docker directly from Docker's repositories, this directory will not be used.

The /lib/systemd directory is where packages will install their systemd files, and they will overwrite any changes there on upgrade. If you use this, your changes will be lost.

To make your own changes to a systemd unit file that persist, you can create a unit file in /etc/systemd/system/docker.service.d/, e.g. here is my standard /etc/systemd/system/docker.service.d/override.conf:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd

That override simply unsets all of the command line flags to the dockerd daemon from systemd. Once done, you can override every setting from /etc/docker/daemon.json which is used by docker, and depending on the setting, can be reloaded without restarting the daemon. E.g. here's an example /etc/docker/daemon.json:

{
"debug": false,
"experimental": true,
"hosts": ["fd://", "tcp://0.0.0.0:2376"],
"labels": ["foo=bar", "fez=baz"],
"log-driver": "json-file",
"log-opts": {"max-size": "10m", "max-file": "3"},
"storage-driver": "overlay2",
"tlscacert": "/etc/docker/certs/ca.pem",
"tlscert": "/etc/docker/certs/host-cert.pem",
"tlskey": "/etc/docker/certs/host-key.pem",
"tlsverify": true
}

For your purposes, you only need the line in there to set the hosts.

One extremely important part of the above config file are the TLS settings. If you do not configure mutual TLS between client and server, and you open docker to listen on the network, you are running the equivalent of an open telnet server with root logins allowed without a password. If you prefer ssh over telnet, or if you prefer having a password for your root account, then you must configure TLS. The docker API ports are frequently scanned on the internet, and you will find malware installed on your host in a short while if you ever skip this configuration step.

Full details on how to configure the TLS keys for client and server can be found at: https://docs.docker.com/engine/security/https/

---

Note that with docker versions 18.09 and above on the client (both where you run your command and the remote node), you can use ssh instead of configuring TLS. This involves using a DOCKER_HOST value of ssh://user@host. E.g.

docker -H ssh://user@host container ls

Answer 3

There is an official documentation describes how to Configure where the Docker daemon listens for connections.

systemd vs daemon.json

Configuring Docker to listen for connections using both the systemd unit file and the daemon.json file causes a conflict that prevents Docker from starting.

Configuring remote access with systemd unit file

1. Use the command sudo systemctl edit docker.service to open an override file for docker.service in a text editor.

2. Add or modify the following lines, substituting your own values.

   [Service]
   ExecStart=
   ExecStart=/usr/bin/dockerd -H fd:// -H tcp://127.0.0.1:2375
   

3. Save the file.

4. Reload the systemctl configuration.

   $ sudo systemctl daemon-reload
   

5. Restart Docker.

   $ sudo systemctl restart docker.service
   

6. Check to see whether the change was honored by reviewing the output of netstat to confirm dockerd is listening on the configured port.

   $ sudo netstat -lntp | grep dockerd
   tcp        0      0 127.0.0.1:2375          0.0.0.0:*               LISTEN      3758/dockerd
   

Configuring remote access with daemon.json

1. Set the hosts array in the /etc/docker/daemon.json to connect to the UNIX socket and an IP address, as follows:

   {
   "hosts": ["unix:///var/run/docker.sock", "tcp://127.0.0.1:2375"]
   }
   

   Configuring Docker to listen for connections using both the systemd unit file and the daemon.json file causes a conflict that prevents Docker from starting.

   1. Add or modify the following lines, substituting your own values.

      [Service]
      ExecStart=
      ExecStart=/usr/bin/dockerd
      

   2. Save the file.

   3. Reload the systemctl configuration.

      $ sudo systemctl daemon-reload
      

2. Restart Docker.

3. Check to see whether the change was honored by reviewing the output of netstat to confirm dockerd is listening on the configured port.

   $ sudo netstat -lntp | grep dockerd
   tcp        0      0 127.0.0.1:2375          0.0.0.0:*               LISTEN      3758/dockerd
   

---

The Docker client will honor the DOCKER_HOST environment variable to set the -H flag for the client. Use one of the following commands:

$ docker -H tcp://127.0.0.1:2375 ps

or

$ export DOCKER_HOST="tcp://127.0.0.1:2375"
$ docker ps

Answer 4

If you don't want to reconfigure and restart your docker daemon you can simply bridge the unix socket to a TCP socket using ncat (from the nmap package):

ncat -lknvp 2375 -c "ncat -U /var/run/docker.sock"

As alternative, you can use socat or other tools.

Answer 5

For those looking for this answer in the context of Ubuntu server 20.04 that uses SNAP:

This comment github issue should give you the context you need. In my case I didn't find the $SNAP_DATA environment variable set, so I had to look for all the daemon.json files on the system and used the one with the /var prefix

$ sudo find / -name daemon.json

In my case it had two unrelated entries so I just added mine:

{
  [.....]
  "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2375"]
}

The above IP determines from where it will be accessible, and the port can be whatever port you want to expose. In my particular case, I was not able to make it work with a specific IP, instead I had to use 0.0.0.0, otherwise it would fail with the following error when restarting:

aufs aufs_fill_super:918:mount[3724]: no arg
overlayfs: missing 'lowerdir'
aufs aufs_fill_super:918:mount[3772]: no arg
overlayfs: missing 'lowerdir'
aufs aufs_fill_super:918:mount[3820]: no arg
overlayfs: missing 'lowerdir'

Weird enough, when using 0.0.0.0 it actually spits one pair of the above error lines but it does work afterwards. In my case as this is a VM this is acceptable for me.

Answer 6

It's good that you're cautious about exposing the Docker daemon's control TCP port directly to the internet, as it can pose a significant security risk. A safer and recommended approach is to use a proxy tool like docker-socket-proxy. This tool provides a secure way to manage Docker remotely without exposing the Docker daemon directly.

1. Why docker-socket-proxy?: docker-socket-proxy acts as a middleman between your Docker client and the Docker daemon. It allows you to define which Docker API endpoints are accessible, reducing the risk of exposing sensitive Docker daemon operations. This is crucial since direct exposure of the Docker daemon, especially over the internet, can lead to severe security vulnerabilities.

2. How to Implement: To use docker-socket-proxy, you run it as a Docker container, and it connects to your Docker daemon’s socket. Then, it exposes a controlled and filtered TCP port for your Docker clients to connect to. This setup ensures that you are not exposing the Docker daemon directly to the internet.

3. Additional Security Measures: Even with docker-socket-proxy, you should not expose its TCP port directly to the internet. Instead, consider additional security layers like VPN access, firewall rules, or SSH tunnels. These measures will ensure that only authorized personnel can access your Docker endpoints remotely.

For detailed setup and configuration, refer to the docker-socket-proxy GitHub repository https://github.com/Tecnativa/docker-socket-proxy.