2

I'm having trouble with pods not being able to 'talk' to cluster IPs (virtual IPs fronting pods) in my Kubernetes cluster.

I've been following along with "Kubernetes the hard way" by Kelsey Hightower, however I've converted it all to run the infrastructure in AWS.

I have pretty much everything working, except I have a problem where my pods are unable to talk to clusterIP virtual IPs.

  • service-cluster-ip-range is: 10.32.0.0/24
  • Pod CIDR for worker nodes is: 10.200.0.0/16

I've tried with both CoreDNS and Kube-dns initially, thinking it might have been an issue at that level, however I've since diagnosed down to the fact that I cannot talk to service cluster IPs from pods, but on the actual worker nodes I can indeed talk to cluster IPs.

I've verified that kube-proxy is working as expected. I'm running that in iptables mode and can see it writing out iptables rules on worker nodes correctly. I even tried switching to ipvs mode and in that mode it also wrote out rules correctly.

If I do nslookup inside a test pod (e.g. busybox 1.28) and let it use it's standard nameserver setting pointing to my coredns installation, it fails to resolve google.com or the clusterkubernetes.default`. However if I tell nslookup to use the POD IP address of the coredns pod, it works just fine.

Example

This does not work:

kubectl exec -it busybox -- nslookup google.com               
Server:    10.32.0.10
Address 1: 10.32.0.10

nslookup: can't resolve 'google.com'
command terminated with exit code 1

This works (pointing nslookup to coredns pod IP address rather that cluster IP):

kubectl exec -it busybox -- nslookup google.com 10.200.2.2                   
Server:    10.200.2.2
Address 1: 10.200.2.2 kube-dns-67d45fcb87-2h2dz

Name:      google.com
Address 1: 2607:f8b0:4004:810::200e iad23s63-in-x0e.1e100.net
Address 2: 172.217.164.142 iad30s24-in-f14.1e100.net

To clarify, I've tried this with both CoreDNS and kube-dns - same result in both cases. It seems like a higher up networking issue.

My AWS EC2 instances have source/destination checking disabled. All my configuration and settings are forked from the official kubernetes-the-hard-way repo, but I've updated things to run on AWS. Source with all my config / settings etc is here

Edit: providing the /etc/resolv.conf that my pods are getting from kube-dns / coredns for info (this looks absolutely fine though):

# cat /etc/resolv.conf
search kube-system.svc.cluster.local svc.cluster.local cluster.local ec2.internal
nameserver 10.32.0.10
options ndots:5

I am able to ping the kube-dns pod IP directly from pods, but the cluster IP for kube-dns does not work for ping or anything else. (same for other services with cluster IPs). E.g.

me@mine ~/Git/kubernetes-the-hard-way/test kubectl get pods -n kube-system -o wide
NAME                                READY   STATUS    RESTARTS   AGE    IP            NODE             NOMINATED NODE   READINESS GATES
hello-node1-55cc74b4b8-2hh4w        1/1     Running   2          3d1h   10.200.2.14   ip-10-240-0-22   <none>           <none>
hello-node2-66b5494599-cw8hx        1/1     Running   2          3d1h   10.200.2.12   ip-10-240-0-22   <none>           <none>
kube-dns-67d45fcb87-2h2dz           3/3     Running   6          3d1h   10.200.2.11   ip-10-240-0-22   <none>           <none>

 me@mine ~/Git/kubernetes-the-hard-way/test kubectl exec -it hello-node1-55cc74b4b8-2hh4w sh
Error from server (NotFound): pods "hello-node1-55cc74b4b8-2hh4w" not found
 me@mine ~/Git/kubernetes-the-hard-way/test kubectl -n kube-system exec -it hello-node1-55cc74b4b8-2hh4w sh
# ping 10.200.2.11
PING 10.200.2.11 (10.200.2.11) 56(84) bytes of data.
64 bytes from 10.200.2.11: icmp_seq=1 ttl=64 time=0.080 ms
64 bytes from 10.200.2.11: icmp_seq=2 ttl=64 time=0.044 ms
64 bytes from 10.200.2.11: icmp_seq=3 ttl=64 time=0.045 ms
^C
--- 10.200.2.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.044/0.056/0.080/0.017 ms

# ip route get 10.32.0.10
10.32.0.10 via 10.200.2.1 dev eth0  src 10.200.2.14
    cache
#

Am I missing something obvious here?

15
  • What do you have in /etc/resolv.conf ?
    – fg78nc
    Nov 3, 2019 at 22:26
  • Added the content of /etc/resolv.conf to my original question in the edit. (It looks fine). The point I have is that the 10.32.0.10 cluster service vip is not reachable from my pods, but is from worker nodes. If it was reachable, then this /etc/resolv.conf would be working for my pods. If I manually change the 10.32.0.10 IP in the resolv.conf to point at the kube-dns or coredns pod IP address then everything works as expected.
    – Shogan
    Nov 4, 2019 at 9:26
  • 1
    And what about ip route get 10.32.0.10 run in your pod ? Are you able to ping it ?
    – mario
    Nov 6, 2019 at 12:43
  • @mario I have added the results of ip route get 10.32.0.10 to my original question at the bottom (as well as ping results). As expected ping to the services' cluster IP does not work. However direct to the pod IP works fine.
    – Shogan
    Nov 6, 2019 at 23:05
  • 1
    @JackTheKnife unfortunately no, I didn't. I got distracted with other projects and moved on from this. If I ever boot up a new environment again I'll take another crack at this and update here if I find anything.
    – Shogan
    Aug 28, 2020 at 21:10

2 Answers 2

1

Had this exact same problem, here's the solution:

modprobe br_netfilter
sysctl net.bridge.bridge-nf-call-iptables=1
0

Try to add the following to kube-dns ConfigMap

data:
  upstreamNameservers: |
    [“8.8.8.8”, “8.8.4.4”]
1
  • It's not an upstream DNS issue- as per my question, DNS works just fine if I point anything directly at my coredns or kubedns pod IP address. In other words coredns/kubedns are able to do DNS resolution themselves just fine.
    – Shogan
    Nov 4, 2019 at 8:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .