1

My microk8s cluster is running on a centos8 vm and I have some issues with DNS resolution of my pods. Nameserver are at x.x.x.101 and x.x.x.100 both are ping able from within the pods and I can also ping 8.8.8.8

nslookup from within a pod looks like this:

root@debug-7857894f66-mnklp:/# nslookup kubernetes.default
Server:         10.152.183.10
Address:        10.152.183.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.152.183.1

coredns config looks like this:

apiVersion: v1
data:
  Corefile: ".:53 {\n    errors\n    health {\n      lameduck 5s\n    }\n    ready\n
    \   log . {\n      class error\n    }\n    kubernetes cluster.local in-addr.arpa
    ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n
    \   prometheus :9153\n    forward .  x.x.x.101 x.x.x.100 \n    cache 30\n
    \   loop\n    reload\n    loadbalance\n}\n"
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors\n    health {\n      lameduck 5s\n    }\n    ready\n    log . {\n      class error\n    }\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward .  x.x.x.101 x.x.x.100 \n    cache 30\n    loop\n    reload\n    loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists","k8s-app":"kube-dns"},"name":"coredns","namespace":"kube-system"}}
  creationTimestamp: "2021-08-31T08:57:27Z"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system
  resourceVersion: "2420090"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 471b258a-253d-4b51-aaf7-7e934ab300d1

the /etc/resolv.conf in my pods looks like this:

search default.svc.cluster.local svc.cluster.local cluster.local xxx.xxxxx
nameserver 10.152.183.10
options ndots:5

when I look at the logs of my kube-dns with $ microk8s kubectl logs --namespace=kube-system -l k8s-app=kube-dns I get the following response:

[INFO] 10.1.107.105:47549 - 5288 "AAAA IN www.google.com. udp 36 false 512" NOERROR - 0 0.000256103s
[ERROR] plugin/errors: 2 www.google.com. AAAA: read udp 10.1.107.127:51486->x.x.x.101:53: read: no route to host

DNS service is up:

$ microk8s kubectl get svc --namespace=kube-system
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
[...]
kube-dns                    ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP   21d

DNS endpoints are exposed:

$ microk8s kubectl get endpoints kube-dns --namespace=kube-system
NAME       ENDPOINTS                                           AGE
kube-dns   10.1.107.127:53,10.1.107.127:53,10.1.107.127:9153   21d

The virtual machine the cluster is running on is using the same nameserver without a problem. Is this something I need to bother our admins about, or am I missing something that is wrongly configured?

EDIT:

Okay I tried out a bit more and there is some weird behavior that I think has to do with me incorrectly creating the coredns config file. I completely reset the cluster and tried again, still same error when I use 8.8.8.8 or 8.8.4.4 as DNS servers. But, when I run the following command: $ microk8s enable dns:x.x.x.101,x.x.x.100 finally it works. Then I tried to configure for this with $ microk8s kubectl -n kube-system edit configmap/coredns and I add both DNS to the config it stops working, and I again get the error: [ERROR] plugin/errors: 2 www.google.com. AAAA: read udp 10.1.107.127:51486->x.x.x.101:53: read: no route to host so what is wrong with my configuration that is automatically correctly set with $ microk8s enable dns:x.x.x.101,x.x.x.100 ?

2nd Edit

I tried to use dig here even when I specify a DNS server it is not working. Is there any reason ping could go through but DNS is blocked by the system? It only happens on the microk8s cluster, host system runs fin docker runs fine ... here are the printouts: this is from within the pod:

root@debug-865cb7fb4-wfhw4:/# dig www.google.com

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> www.google.com
;; global options: +cmd
;; connection timed out; no servers could be reached
root@debug-865cb7fb4-wfhw4:/# dig @x.x.x.101 www.google.com

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @x.x.x.101 www.google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
root@debug-865cb7fb4-wfhw4:/# dig @8.8.8.8 www.google.com

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @8.8.8.8 www.google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
root@debug-865cb7fb4-wfhw4:/# dig @x.x.x.100 www.google.com

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @x.x.x.100 www.google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

and this is from the host system:

$ dig www.google.com

; <<>> DiG 9.11.20-RedHat-9.11.20-5.el8_3.1 <<>> www.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25735
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;www.google.com.                        IN      A

;; ANSWER SECTION:
www.google.com.         113     IN      A       142.250.185.228

;; Query time: 0 msec
;; SERVER: x.x.x.101#53(x.x.x.101)
;; WHEN: Fri Oct 08 15:10:21 CEST 2021
;; MSG SIZE  rcvd: 59



$ dig @8.8.8.8 www.google.com

; <<>> DiG 9.11.20-RedHat-9.11.20-5.el8_3.1 <<>> @8.8.8.8 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3924
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.google.com.                        IN      A

;; ANSWER SECTION:
www.google.com.         300     IN      A       142.250.185.228

;; Query time: 34 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Oct 08 15:10:49 CEST 2021
;; MSG SIZE  rcvd: 59

$ dig @x.x.x.101 www.google.com

; <<>> DiG 9.11.20-RedHat-9.11.20-5.el8_3.1 <<>> @x.x.x.101 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60305
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;www.google.com.                        IN      A

;; ANSWER SECTION:
www.google.com.         70      IN      A       142.250.185.228

;; Query time: 0 msec
;; SERVER: x.x.x.101#53(x.x.x.101)
;; WHEN: Fri Oct 08 15:11:04 CEST 2021
;; MSG SIZE  rcvd: 59

I have no idea what seems to be going on ....

5
  • Which version of Kubernetes did you use? Sep 22, 2021 at 13:17
  • I'm using Kubernetes v1.22
    – Thagor
    Sep 23, 2021 at 6:20
  • Please provide your pod yaml file. Sep 24, 2021 at 12:28
  • I rebuild the cluster in an infrastructure that is fully under my control, there DNS works.
    – Thagor
    Sep 27, 2021 at 6:50
  • It seems to be an issue connected to iptables and the firewall on the host, if I disable both the problem goes away I'll post and answer as soon as I know how to correctly configure these tools.
    – Thagor
    Oct 12, 2021 at 11:26

1 Answer 1

0

I have posted a community wiki answer for better visibility. As Thagor has mentioned in the comment, problem is resolved and was related to external DNS:

I rebuild the cluster in an infrastructure that is fully under my control, where DNS works.

Try disabling the iptables and the firewall on the host and see if you are able to do via coredns config.

More information about DNS you can find in the official documentation.

2
  • Hello @Thagor. Does this answer your question? Sep 28, 2021 at 11:53
  • hey i updated my first post turns out DNS works if i set it with $ microk8s enable dns:x.x.x.100,x.x.x.101 but as soon as I try to do it via the coredns config it fails, and I have no idea why.
    – Thagor
    Oct 5, 2021 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.