I have a minimal cloud-config that works without problems on DigitalOcean. I added some hardening for SSH, which requires restarting sshd.socket
to become effective:
units:
- name: sshd.socket
command: restart
Adding this unit alone (no actual sshd configuration changes) causes provisioning with the same cloud-config to fail when trying it on Hetzner:ssh: connect to host xx.xx.xx.xx port 22: Connection refused
. It still connects fine on DigitalOcean though.
When I remove this unit then connecting to the Hetzner machine works fine, adding it again fails consistently.
Variable substitution
The only difference between both platforms that I know of is that on DigitalOcean the variables $public_ipv4
and $private_ipv4
are replaced with actual IP addresses, which is not the case on bare metal installs like Hetzner.
From the CoreOS Documentation:
Note: The $private_ipv4 and $public_ipv4 substitution variables referenced in other documents are only supported on Amazon EC2, Google Compute Engine, OpenStack, Rackspace, DigitalOcean, and Vagrant.
So I substitute the variables with the static IP address. I use the public IP address because that's the only interface available besides loopback.
However, when I provision without substituting these variables with the public IP address, then it ALSO connects fine.
Inspecting the journal reveals some errors related to name resolution:
systemd[1]: Starting etcd2...
etcd2[874]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://:2379,http://:4001
etcd2[874]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2
etcd2[874]: recognized and used environment variable ETCD_DISCOVERY=https://discovery.etcd.io/616b3957c5c78e7738207011f9c51841
etcd2[874]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://:2380
etcd2[874]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://0.0.0.0:4001
etcd2[874]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://:2380
etcd2[874]: recognized and used environment variable ETCD_NAME=39b2a003672546f8a0b648dbc66e8f6f
etcd2[874]: etcd Version: 2.2.0
etcd2[874]: Git SHA: e4561dd
etcd2[874]: Go Version: go1.4.2
etcd2[874]: Go OS/Arch: linux/amd64
etcd2[874]: setting maximum number of CPUs to 1, total number of available CPUs is 12
etcd2[874]: listening for peers on http://:2380
etcd2[874]: listening for client requests on http://0.0.0.0:2379
etcd2[874]: listening for client requests on http://0.0.0.0:4001
etcd2[874]: resolving :2380 to :2380
etcd2[874]: resolving :2380 to :2380
etcd2[874]: error #0: dial tcp: lookup discovery.etcd.io: Temporary failure in name resolution
etcd2[874]: cluster status check: error connecting to https://discovery.etcd.io, retrying in 2s
etcd2[874]: error #0: dial tcp: lookup discovery.etcd.io: Temporary failure in name resolution
etcd2[874]: cluster status check: error connecting to https://discovery.etcd.io, retrying in 4s
etcd2[874]: found self 61dbc8c9c2aca1e8 in the cluster
etcd2[874]: found 1 needed peer(s)
But they don't seem fatal: systemctl status etcd2.service
shows that the service is active:
core@localhost ~ $ systemctl status etcd2.service
● etcd2.service - etcd2
Loaded: loaded (/usr/lib64/systemd/system/etcd2.service; disabled; vendor preset: disabled)
Drop-In: /run/systemd/system/etcd2.service.d
└─20-cloudinit.conf
Active: active (running) since Tue 2016-03-22 14:10:33 UTC; 7min ago
Main PID: 874 (etcd2)
Memory: 20.3M
CPU: 1.771s
CGroup: /system.slice/etcd2.service
└─874 /usr/bin/etcd2
etcd2[874]: added local member 61dbc8c9c2aca1e8 [http://:2380] to cluster 216c373aaf11ccfa
systemd[1]: Started etcd2.
etcd2[874]: 61dbc8c9c2aca1e8 is starting a new election at term 1
etcd2[874]: 61dbc8c9c2aca1e8 became candidate at term 2
etcd2[874]: 61dbc8c9c2aca1e8 received vote from 61dbc8c9c2aca1e8 at term 2
etcd2[874]: 61dbc8c9c2aca1e8 became leader at term 2
etcd2[874]: raft.node: 61dbc8c9c2aca1e8 elected leader 61dbc8c9c2aca1e8 at term 2
etcd2[874]: published {Name:39b2a003672546f8a0b648dbc66e8f6f ClientURLs:[http://:2379 http://:4001]} to cluster 216c373aaf11ccfa
etcd2[874]: setting up the initial cluster version to 2.2
etcd2[874]: set the initial cluster version to 2.2
Containers that connect to other services like Logstash fail: the scheme http does not accept registry part: :9200 (or bad hostname?)
Cloud-config
This is a stripped-down cloud-config, but it still demonstrates the issue (verified that).
#cloud-config
ssh_authorized_keys:
- "ssh-rsa A valid SSH key here"
write_files:
coreos:
etcd2:
# NOTE: replace $discovery_url with a url generated at https://discovery.etcd.io/new?size=X
discovery: $discovery_url
listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
advertise-client-urls: http://my.public.ip.address:2379,http://my.public.ip.address:4001
initial-advertise-peer-urls: http://my.public.ip.address:2380
listen-peer-urls: http://my.public.ip.address:2380 # Remove this flag or use localhost and the connection issue goes away
units:
- name: etcd2.service
command: start
- name: fleet.service
command: start
- name: sshd.socket
command: restart # Remove this unit and all issues go away (but no SSH hardening in that case)
One thing I noticed is that when I remove the flag listen-peer-urls
the connection issue also goes away, although logstash still doesn't start for the same reason.
This document says the default value for these flags are URLs with localhost
, but the name of the substitution variables that are used on platforms like DigitalOcean seem to suggest that this should be an address that's reachable by peer machines.
When I use localhost
for these flags I can connect, but the other issues are still there.
Question 1
What should be the proper cloud-config for bare metal machines that have a public and loopback interface only (no private network)?
Question 2
What is the relationship between sshd and etcd here that causes this failure?