13

I am using portainer and am unable to manage remote endpoints. I tried using the command line to connect to remote docker nodes, but got a message Cannot connect to the Docker daemon at tcp://<remote_ip>:<port>. Is the docker daemon running?.

Yes, they are running. I have added myself to the docker group and can access docker by SSHing into the nodes. However I cannot access any docker nodes remotely.

I modified /etc/default to add / uncomment DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock"

I also modified /etc/init.d/docker and /etc/init/docker.conf to include DOCKER_OPTS="-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock".

I restarted the docker service, logged out and logged in multiple times in the process, but still cannot connect to the remote node. I cannot even connect to the local node by passing the IP.

What did I miss out? What configuration in what file exposes the API over TCP?

user@hostname:~$ docker -H tcp://<REMOTE_IP>:2375 info
Cannot connect to the Docker daemon at tcp://<REMOTE_IP>:2375. Is the docker daemon running?
user@hostname:~$ docker -H tcp://127.0.0.1:2375 info
Cannot connect to the Docker daemon at tcp://127.0.0.1:2375. Is the docker daemon running?
user@hostname:~$ docker -H tcp://<LOCAL_IP>:2375 info
Cannot connect to the Docker daemon at tcp://<LOCAL_IP>:2375. Is the docker daemon running?
user@hostname:~$

Edit: Running ps aux | grep -i docker returns this -

root      3581  0.1  0.2 596800 41540 ?        Ssl  04:17   0:35 /usr/bin/dockerd -H fd://
root      3588  0.0  0.0 653576 14492 ?        Ssl  04:17   0:18 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
  • Is 2375 listening? ss -ntl – jscott Apr 7 '17 at 16:43
  • No. There is nothing listening on 2375. And I cannot figure out what configuration in what file affects this. I have included the output of ps aux in my answer if that helps. – Lord Loh. Apr 7 '17 at 16:54
  • I tried similar things and suspect that the files /etc/default/docker, /etc/init/docker.conf and /etc/init.d/docker are simply ignored on Ubuntu 16.04 with an installation of docker-ce, can anyone confirm? I think when I run "service docker status" what really happens is "systemctl status docker", a whole other management system. – chrisinmtown Jul 7 '17 at 18:44
22

I found a solution thanks to Ivan Krizsan's post.

I had to edit /lib/systemd/system/docker.service on my Ubuntu 16.04.2 LTS system to modify the line

ExecStart=/usr/bin/docker daemon -H fd:// -H tcp://0.0.0.0:

then

sudo systemctl daemon-reload
sudo systemctl restart docker.service

and everything worked :-). The next step is to figure out how to protect the docker daemon form being hijacked.

  • 1
    I confirm this change causes the Dockerd to listen for HTTP requests on Ubuntu 16.04 with docker-ce ver 17.06. It just feels wrong to modify the service script directly. – chrisinmtown Jul 7 '17 at 18:57
  • 7
    Never edit the docker service script (or any service script) directly. SystemD has a differential editing feature built in. Use systemctl edit docker.service and systemctl will create an new file with your edits. This prevents an update from wiping your changes. SystemD will merge the two files at runtime. Good document here: digitalocean.com/community/tutorials/… – Routhinator Sep 20 '17 at 2:07
  • Thanks! This is terribly useful. I keep getting warning during updates. I'll look into this :_) – Lord Loh. Sep 20 '17 at 5:48
  • 4
    On Ubuntu Server 18.04, it worked like this: ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:4243 – LUCIAN ALEXA Aug 17 '18 at 12:24
  • when i use the -H tcp:// setting by itself, without the -H fd:// setting, I can't issue and client commands, eg.., docker info, docker version, etc. – Chris F Jun 17 at 17:38
2

The /etc/default directory is where distribution maintainers put their configuration files. If you install docker directly from Docker's repositories, this directory will not be used.

The /lib/systemd directory is where packages will install their systemd files, and they will overwrite any changes there on upgrade. If you use this, your changes will be lost.

To make your own changes to a systemd unit file that persist, you can create a unit file in /etc/systemd/system/docker.service.d/, e.g. here is my standard /etc/systemd/system/docker.service.d/override.conf:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd

That override simply unsets all of the command line flags to the dockerd daemon from systemd. Once done, you can override every setting from /etc/docker/daemon.json which is used by docker, and depending on the setting, can be reloaded without restarting the daemon. E.g. here's an example /etc/docker/daemon.json:

{
"debug": false,
"experimental": true,
"hosts": ["fd://", "tcp://0.0.0.0:2376"],
"labels": ["foo=bar", "fez=baz"],
"log-driver": "json-file",
"log-opts": {"max-size": "10m", "max-file": "3"},
"storage-driver": "overlay2",
"tlscacert": "/etc/docker/certs/ca.pem",
"tlscert": "/etc/docker/certs/host-cert.pem",
"tlskey": "/etc/docker/certs/host-key.pem",
"tlsverify": true
}

For your purposes, you only need the line in there to set the hosts.

One extremely important part of the above config file are the TLS settings. If you do not configure mutual TLS between client and server, and you open docker to listen on the network, you are running the equivalent of an open telnet server with root logins allowed without a password. If you prefer ssh over telnet, or if you prefer having a password for your root account, then you must configure TLS. The docker API ports are frequently scanned on the internet, and you will find malware installed on your host in a short while if you ever skip this configuration step.

Full details on how to configure the TLS keys for client and server can be found at: https://docs.docker.com/engine/security/https/

  • 1
    A very good answer that will survive future updates to the docker service. This is the right way to do it. – Fopedush Nov 9 '18 at 17:51
2

If you don't want to reconfigure and restart your docker daemon you can simply bridge the unix socket to a TCP socket using ncat (from the nmap package):

ncat -lknvp 2375 -c "ncat -U /var/run/docker.sock"

As alternative, you can use socat or other tools.

  • amazing! can I rund this command in background? when I close my terminal connection is lost – Felix Oct 24 '18 at 6:06
  • ah i found nohup and & – Felix Oct 24 '18 at 6:10
0

There is an official documentation describes how to Configure where the Docker daemon listens for connections.

systemd vs daemon.json

Configuring Docker to listen for connections using both the systemd unit file and the daemon.json file causes a conflict that prevents Docker from starting.

Configuring remote access with systemd unit file

  1. Use the command sudo systemctl edit docker.service to open an override file for docker.service in a text editor.

  2. Add or modify the following lines, substituting your own values.

    [Service]
    ExecStart=
    ExecStart=/usr/bin/dockerd -H fd:// -H tcp://127.0.0.1:2375
    
  3. Save the file.

  4. Reload the systemctl configuration.

    $ sudo systemctl daemon-reload
    
  5. Restart Docker.

    $ sudo systemctl restart docker.service
    
  6. Check to see whether the change was honored by reviewing the output of netstat to confirm dockerd is listening on the configured port.

    $ sudo netstat -lntp | grep dockerd
    tcp        0      0 127.0.0.1:2375          0.0.0.0:*               LISTEN      3758/dockerd
    

Configuring remote access with daemon.json

  1. Set the hosts array in the /etc/docker/daemon.json to connect to the UNIX socket and an IP address, as follows:

    {
    "hosts": ["unix:///var/run/docker.sock", "tcp://127.0.0.1:2375"]
    }
    

    Configuring Docker to listen for connections using both the systemd unit file and the daemon.json file causes a conflict that prevents Docker from starting.

    1. Add or modify the following lines, substituting your own values.

      [Service]
      ExecStart=
      ExecStart=/usr/bin/dockerd
      
    2. Save the file.

    3. Reload the systemctl configuration.

      $ sudo systemctl daemon-reload
      
  2. Restart Docker.

  3. Check to see whether the change was honored by reviewing the output of netstat to confirm dockerd is listening on the configured port.

    $ sudo netstat -lntp | grep dockerd
    tcp        0      0 127.0.0.1:2375          0.0.0.0:*               LISTEN      3758/dockerd
    

The Docker client will honor the DOCKER_HOST environment variable to set the -H flag for the client. Use one of the following commands:

$ docker -H tcp://127.0.0.1:2375 ps

or

$ export DOCKER_HOST="tcp://127.0.0.1:2375"
$ docker ps

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.