I'm having trouble with pods not being able to 'talk' to cluster IPs (virtual IPs fronting pods) in my Kubernetes cluster.

I've been following along with "Kubernetes the hard way" by Kelsey Hightower, however I've converted it all to run the infrastructure in AWS.

I have pretty much everything working, except I have a problem where my pods are unable to talk to clusterIP virtual IPs.

  • service-cluster-ip-range is:
  • Pod CIDR for worker nodes is:

I've tried with both CoreDNS and Kube-dns initially, thinking it might have been an issue at that level, however I've since diagnosed down to the fact that I cannot talk to service cluster IPs from pods, but on the actual worker nodes I can indeed talk to cluster IPs.

I've verified that kube-proxy is working as expected. I'm running that in iptables mode and can see it writing out iptables rules on worker nodes correctly. I even tried switching to ipvs mode and in that mode it also wrote out rules correctly.

If I do nslookup inside a test pod (e.g. busybox 1.28) and let it use it's standard nameserver setting pointing to my coredns installation, it fails to resolve google.com or the clusterkubernetes.default`. However if I tell nslookup to use the POD IP address of the coredns pod, it works just fine.


This does not work:

kubectl exec -it busybox -- nslookup google.com               
Address 1:

nslookup: can't resolve 'google.com'
command terminated with exit code 1

This works (pointing nslookup to coredns pod IP address rather that cluster IP):

kubectl exec -it busybox -- nslookup google.com                   
Address 1: kube-dns-67d45fcb87-2h2dz

Name:      google.com
Address 1: 2607:f8b0:4004:810::200e iad23s63-in-x0e.1e100.net
Address 2: iad30s24-in-f14.1e100.net

To clarify, I've tried this with both CoreDNS and kube-dns - same result in both cases. It seems like a higher up networking issue.

My AWS EC2 instances have source/destination checking disabled. All my configuration and settings are forked from the official kubernetes-the-hard-way repo, but I've updated things to run on AWS. Source with all my config / settings etc is here

Edit: providing the /etc/resolv.conf that my pods are getting from kube-dns / coredns for info (this looks absolutely fine though):

# cat /etc/resolv.conf
search kube-system.svc.cluster.local svc.cluster.local cluster.local ec2.internal
options ndots:5

I am able to ping the kube-dns pod IP directly from pods, but the cluster IP for kube-dns does not work for ping or anything else. (same for other services with cluster IPs). E.g.

me@mine ~/Git/kubernetes-the-hard-way/test kubectl get pods -n kube-system -o wide
NAME                                READY   STATUS    RESTARTS   AGE    IP            NODE             NOMINATED NODE   READINESS GATES
hello-node1-55cc74b4b8-2hh4w        1/1     Running   2          3d1h   ip-10-240-0-22   <none>           <none>
hello-node2-66b5494599-cw8hx        1/1     Running   2          3d1h   ip-10-240-0-22   <none>           <none>
kube-dns-67d45fcb87-2h2dz           3/3     Running   6          3d1h   ip-10-240-0-22   <none>           <none>

 me@mine ~/Git/kubernetes-the-hard-way/test kubectl exec -it hello-node1-55cc74b4b8-2hh4w sh
Error from server (NotFound): pods "hello-node1-55cc74b4b8-2hh4w" not found
 me@mine ~/Git/kubernetes-the-hard-way/test kubectl -n kube-system exec -it hello-node1-55cc74b4b8-2hh4w sh
# ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.080 ms
64 bytes from icmp_seq=2 ttl=64 time=0.044 ms
64 bytes from icmp_seq=3 ttl=64 time=0.045 ms
--- ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.044/0.056/0.080/0.017 ms

# ip route get via dev eth0  src

Am I missing something obvious here?

  • What do you have in /etc/resolv.conf ? – fg78nc Nov 3 '19 at 22:26
  • Added the content of /etc/resolv.conf to my original question in the edit. (It looks fine). The point I have is that the cluster service vip is not reachable from my pods, but is from worker nodes. If it was reachable, then this /etc/resolv.conf would be working for my pods. If I manually change the IP in the resolv.conf to point at the kube-dns or coredns pod IP address then everything works as expected. – Shogan Nov 4 '19 at 9:26
  • 1
    And what about ip route get run in your pod ? Are you able to ping it ? – mario Nov 6 '19 at 12:43
  • @mario I have added the results of ip route get to my original question at the bottom (as well as ping results). As expected ping to the services' cluster IP does not work. However direct to the pod IP works fine. – Shogan Nov 6 '19 at 23:05
  • 1
    @JackTheKnife unfortunately no, I didn't. I got distracted with other projects and moved on from this. If I ever boot up a new environment again I'll take another crack at this and update here if I find anything. – Shogan Aug 28 '20 at 21:10

Try to add the following to kube-dns ConfigMap

  upstreamNameservers: |
    [“”, “”]
  • It's not an upstream DNS issue- as per my question, DNS works just fine if I point anything directly at my coredns or kubedns pod IP address. In other words coredns/kubedns are able to do DNS resolution themselves just fine. – Shogan Nov 4 '19 at 8:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.