1

Summary

I can only resolve hostnames from and for the default namespace. Requests from and for other namespaces, like kube-system, return NXDOMAIN from coredns. Is there some authorisation that needs to be done?

Details

I have set up a learning cluster by following Kubernetes the Hard Way except using VirtualBox instead of GCP. All else appears to work except DNS queries in non-default namespaces. More concretely:

  • Works (run from default namespace): kubernetes.default
  • Does Not work (run from kube-system namespace, get NXDOMAIN): kubernetes.kube-system

Configuration

I have upgraded to Coredns 1.12 because 1.2.2 failed to resolve anything around 50% of the time. With 1.12 requests resolve 100% of the time (if in the default namespace).

I had to edit the config (in the configmap) by removing the "loop" and "proxy" (replace proxy with forward) plugins to get it working. My config is as follows:

    .:53 {
        forward . 192.168.99.254:53 # dnsmasq on host machine
        log
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        cache 30
        reload
        loadbalance

Troubleshooting

Following this guide I see:

The pods are running OK

$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-6f8fb48d88-dz9cd   1/1     Running   0          14m
coredns-6f8fb48d88-s5rvp   1/1     Running   0          14m

Checking the console I see healthy logging

2019-05-24T04:41:13.990Z [INFO] plugin/reload: Running configuration MD5 = 7c717fabd578b2514cf60fee426fabc6
2019-05-24T04:41:13.992Z [INFO] CoreDNS-1.5.0
2019-05-24T04:41:13.992Z [INFO] linux/amd64, go1.12.2, e3f9a80

The DNS service is available:

 $ kubectl get svc --namespace=kube-system
NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kube-dns         ClusterIP   10.32.0.10   <none>        53/UDP,53/TCP   18d

The endpoints are exposed:

$ kubectl get ep kube-dns --namespace=kube-system
NAME       ENDPOINTS                                                     AGE
kube-dns   10.200.21.33:53,10.200.22.17:53,10.200.21.33:53 + 1 more...   18d

As mentioned above, DNS works but only from the default namespace.

0

It appears that it was an issue with the busybox image - if I use generic Ubuntu then records resolve OK and the services that need records are able to get them. E.g. metrics-server can resolve the nodes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.