I have a managed kubernetes instance on azure. I am very sure that the core dns is working and the dns pods are healthy.
I have a couple of services
frontend-service with one pod - Image [nginx-alpine] which has the static frontend files.
backend-service , with one pod - Image [ubuntu:20.04] which has the nodejs code.
I am unable to resolve the internal dns service names like frontend-service OR frontend-service.default.svc.cluster.local from the pods of the backend but nslookup , host , dig of the internal dns names resolve to the correct address. The backend pods are also able to resolve the external dns names like google.com.
curl http://frontend-service curl: (6) Could not resolve host: frontend-service curl http://frontend-service.default.svc.cluster.local curl: (6) Could not resolve host: frontend-service.default.svc.cluster.local
wget frontend-service --2020-08-31 23:36:43-- http://frontend-service Resolving frontend-service (frontend-service)... failed: Name or service not known. wget: unable to resolve host address 'frontend-service'
/etc/nsswitch.conf shows the below : passwd: files group: files shadow: files gshadow: files hosts: files dns networks: files protocols: db files services: db files ethers: db files rpc: db files
Everything works fine while trying to resolve the backend-service internal dns name from the pods of frontend service.
After some debugging and looking at the logs of coredns and the strace , I see that no call is happening to the coredns pods while doing a curl , but I can see the entry while doing an nslook up.
I also. verified that the /etc/resolv.conf has the correct configuration.
nameserver 10.3.0.10 search default.svc.cluster.local svc.cluster.local cluster.local tdghymxumodutbxfnz5m2elcog.bx.internal.cloudapp.net options ndots:5
strace does not show any entry to search for /etc/resolv.conf , so curl is not checking for /etc/resolv.conf.
From the backend service pod : dig frontend-service [It is able to resolve to the correct name server.] ; <<>> DiG 9.16.1-Ubuntu <<>> frontend-service ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 13441 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ; OPT=65436: 87 a1 ee 81 04 d8 5a 49 be 0e c4 ed 1d d8 27 41 ("......ZI......'A") ;; QUESTION SECTION: ;frontend-service. IN A ;; AUTHORITY SECTION: . 30 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2020083101 1800 900 604800 86400 ;; Query time: 20 msec ;; SERVER: 10.3.0.10#53(10.3.0.10) ;; WHEN: Tue Sep 01 10:48:00 IST 2020 ;; MSG SIZE rcvd: 142
nslookup frontend-service Server: 10.3.0.10 Address: 10.3.0.10#53 Name: frontend-service.default.svc.cluster.local Address: 10.3.0.30
host frontend-service frontend-service.default.svc.cluster.local has address 10.3.0.30
I wanted to test the deployment step by step with the same ubuntu:20.04 image , so I did the following.
I created an ephemeral pod in the cluster as below.
kubectl run -it --rm test-ubuntu --image=ubuntu:20.04 --restart=Never Installed curl (7.68) and ran the curl http://frontend-service – This is successful.
This puzzled me , so I have removed all my build steps from Dockerfile and used only the below commands.
FROM ubuntu:20.04 EXPOSE 3688 CMD [ "sleep", "infinity" ]
Pushed the image to acr and deployed the backend pods again.
kubectl exec -it <pod-name> /bin/bash I installed curl (7.68) and ran the curl http://frontend-service – Same error – unable to resolve host.
This is surprising , same image with same content – running through kubectl run and deploying through Dockerfile , has different behaviour while running curl of same version (7.68).
I wanted to see the flow in strace in both the appraches. Please find the strace links from RUN and EXEC
strace from running curl from the ephemeral pod. https://pastebin.com/NthHQacW
strace from running curl from the pod deployed through Dockerfile https://pastebin.com/6LCE5NXu
After analysing the probing paths by running
cat strace-log | grep open
I found that the strace log from the approach 2 is missing the below lines.
2844 openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 7 2844 openat(AT_FDCWD, "/etc/host.conf", O_RDONLY|O_CLOEXEC <unfinished...> 2844 <... openat resumed>) = 7 2844 openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 7 2844 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 7 2844 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 7 2844 openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 7 2844 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC <unfinished ...> 2844 <... openat resumed>) = 7 2844 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 7
So the curl command within the pod is not looking at either /etc/resolv.conf OR /etc/nsswitch.conf.
I am puzzled why the curl's behaviour within two pods with same image and same curl version in the same cluster is different.