1

For context - I am attempting to deploy OKD in an air-gapped environment, which requires mirroring an image registry. This private, secured registry is then pulled from by other machines in the network during the installation process.

To describe the environment - the host machine where the registry container is running is running Centos 7.6. The other machines are all VMs running Fedora coreOS in using libvirt. The VMs and the host are connected using a virtual network created using libvirt which includes DHCP settings (configured through virsh net-edit) for the VMs to give them static IPs. The host machine also hosts the DNS server (bind), which, as far as I can tell is configured properly, as I can ping every machine from every other machine using its fully qualified domain name and access specific ports (such as the port the apache server on the host machine listens on). Podman is used instead of Docker for container management for OKD, but as far as I can tell the commands are exactly the same.

I have the registry running in the air-gapped environment using the following command:

sudo podman run --name mirror-registry -p 5000:5000 -v /opt/registry/data:/var/lib/registry:z \
-v /opt/registry/auth:/auth:z -v /opt/registry/certs:/certs:z -e REGISTRY_AUTH=htpasswd \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/registry.pem -e REGISTRY_HTTP_TLS_KEY=/certs/registry-key.pem \
-d docker.io/library/registry:latest

It is accessible using curl -u username:password https://host-machine.example.local:5000/v2/_catalog which returns {"repositories":[]}. I believe this confirms that my TLS and authorization configurations are correct. However, if I transfer the ca.pem file (used to sign the SSL certificates the registry uses) over to one of the VM's on the virtual network, and attempt to use the same curl command, I get an error:

connect to 192.168.x.x port 5000 failed: Connection refused
Failed to connect to host-machine.example.local port 5000: Connection refused
Closing connection 0

This is quite strange to me, as I've been able to use this method to communicate with the registry from the VMs in the past, and I'm not sure what has changed.

After some further digging, it seems like there is some sort of issue with the port itself, but I can't be sure where the issue is stemming from. For example, If I run sudo netstat -tulpn | grep LISTEN on the host, I receive a line indicating that podman (conmon) is listening on the correct port:

tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN 48337/conmon

but if I test whether the port is accessible from the VM, (nc -zvw5 192.168.x.x 5000) I get a similar error: Ncat: Connection refused. If I use the same test on any of the other listening ports on the host, it indicates successful connections to those ports.

Please note, I have completely disabled firewalld, so as far as I know, all ports are open.

I'm not sure if the issue is with my DNS settings, or the virtual network, or with the registry itself and I'm not quite sure how to further diagnose the issue. Any insight would be much appreciated.

Network definition:

<network connections='6'>
  <name>okd</name>
  <uuid>2ce10cce-9bb6-4d5d-950f-15427172b196</uuid>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='52:54:00:d9:d6:95'/>
  <domain name='okd'/>
  <ip address='192.168.2.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.2.200' end='192.168.2.254'/>
      <host mac='52:54:00:45:93:07' name='okd-bootstrap' ip='192.168.2.200'/>
      <host mac='52:54:00:f0:0a:1c' name='okd-master1' ip='192.168.2.201'/>
      <host mac='52:54:00:d1:29:9e' name='okd-master2' ip='192.168.2.202'/>
      <host mac='52:54:00:c9:a4:bb' name='okd-master3' ip='192.168.2.203'/>
      <host mac='52:54:00:25:5d:48' name='okd-worker1' ip='192.168.2.204'/>
      <host mac='52:54:00:1e:90:3c' name='okd-worker2' ip='192.168.2.205'/>
    </dhcp>
  </ip>
</network>
5
  • Please show the libvirt network definition. Also please avoid obfuscating things that do not need to be obfuscated, such as RFC1918 addresses. Mar 1, 2021 at 22:56
  • Sure thing. Are you talking about just posting the xmldump for virtual network, or is there another configuration file I should be pulling this definition from? I'll post that as soon as I can. Mar 1, 2021 at 23:49
  • Sure, the XML is fine, e.g. virsh net-dumpxml <name> Mar 1, 2021 at 23:52
  • Thanks. I don't have access to the machine at the moment but I'll post it ASAP. Mar 1, 2021 at 23:54
  • Just updating that I've appended the network definition to the post. Thanks. Mar 2, 2021 at 13:25

1 Answer 1

1

"Connection refused" is a convenient hint. Either iptables is rejecting the connection, or the software is configured to whitelist/blacklist certain ip ranges. Something in the path is actively rejecting the connection. If you were getting a "Request timed out"... it could be a much larger list of problems. (routing, packets being dropped... etc...)

5
  • Great leads, thanks. I hadn't looked at anything regarding the iptables yet, so I'll try to dig into that. Something is definitely restricting only connections to this port, but I can't seem to narrow it down any further since all other connections between the host and the VMs work just fine. Mar 1, 2021 at 23:48
  • Good point. Libvirt does both ingress and egress filtering on its virtual networks, and Connection refused is what you get if your traffic is blocked by these firewall rules. This suggests you might not be using IP addresses that are legitimately part of that virtual network. Mar 1, 2021 at 23:54
  • As far as I know, all the VM ips are configured with DHCP through virsh net-edit, but not the host machine's ip. It's only indicated as the host in this configuration as far as I can remember. The host machine also hosts the registry, DNS server, load balancer, etc. I'll get the network definition posted, likely tomorrow. Could it be that even though the host ip address is reachable from the VMs through DNS resolution, its not reachable in this particular way because of some libvirt configuration having to do with SSL? All the other methods of reaching it were successful (ping, curl to httpd). Mar 2, 2021 at 0:07
  • Just an update that I've added the network definition to the post. Mar 2, 2021 at 13:25
  • Another update - the issue was with iptables. Even though I had disabled firewalld, iptables still had forward rules that were set by Podman which, while accepting packets from virbr1 to virbr1, were rejecting any packets from virbr1 to any and vice versa that weren't established. Once changing these rules to accept those packets, I'm able to access the registry. Mar 2, 2021 at 15:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .