I have the following project that I use to create my own Kubernetes cluster on the local machine (macOS) via KubeAdm: https://github.com/sasadangelo/k8s-cluster Basically, I started from this project and did the following update:

  • Configure the Vagrant deployment via a YAML file.
  • Support the K8s 1.6 APIs, K8s 1.18, and Calico 3.8.8
  • Automatic K8s dashboard deploy.

Now I am fine with this cluster but I have some DNS issues I would like to understand how to solve. The tool uses a Vagrant file to start 3 Vagrant boxes then I use mainly 3 scripts:

  • configure_box.sh, I use to install on each box the required software like docker, kubeadm, kubelet, kubectl, etc.
  • configure_master.sh, used to configure only the master node, install network plugin, and run the kubeadm init.
  • configure_worker.sh, used to configure the worker node joining the cluster.

My problem is that resolv.conf of my Pods looks like this:

search default.svc.cluster.local svc.cluster.local cluster.local <other domain coming from host machine>
options ndots:5

This file has two problems:

  1. doesn't allow me to access the Internet. In fact, I have to add manually add it adding this line at the beginning of the file:

and remove it when not required anymore (if I keep it internal DNS doesn't work)

  1. in the search clause there are domains coming from my host machines: <other domain coming from host machine> that cause issues.

I read a lot of documentation but I still have problems figuring out how to change my code to avoid these two problems.

  • If you don't have a specific reason to set up a three node development cluster, consider using minikube instead. Jul 21, 2021 at 16:16
  • Minikube, as far as I know, is a single node. I need to test my application with Pods running on different workers. Jul 21, 2021 at 16:19
  • @SalvatoreD'angelo with latest minikube version you could add multiple nodes
    – c4f4t0r
    Jul 22, 2021 at 2:05
  • Hi, yes I noticed this. I didn't know it. But it's not clear to me if they are real nodes (I mean Virtual Machine like Virtual Box) or just containers. If they are VM which is the underlying technology used by minikube? I tried to search for an answer but no luck. Jul 22, 2021 at 8:06

1 Answer 1

  1. doesn't allow me to access the Internet.

Your default configuration seems fine. As you've found, you don't want to change your pod resolv.conf as it needs to use the coredns service within the cluster for local dns lookups. The coredns service should be able to forward dns requests to the resolver configured in your k8s-master /etc/resolv.conf. It seems like it may be a problem with the k8s-master server internet access or with the virtualbox networking. I cloned your repo and was able to resolve internet dns from pods running on the cluster as expected.

If you take a look at the coredns configuration you will see that it's set to use /etc/resolv.conf for any unspecified domains. From k8s-master you can run kubectl -n kube-system get configmap coredns -o yaml to see the coredns config. There will be a line, forward . /etc/resolv.conf. The kubernetes docs here explain the coredns corefile configuration. You will also find that the resolve.conf in the coredns pod is the same as on the host. To verify this, you can run ps -aux | grep coredns to get the pid of one of the coredns processes. Then run nsenter -t <pid> -n cat /etc/resolv.conf and you should see that the content is the same as the host system. It should have a resolver from the virtualbox network. You can test this resolver from the host or within the coredns process to validate if it can resolve dns. You could also test against the google resolver from within the coredns process to see what you get. I would try a dns lookup as well as just a ping. This should help you narrow down what's causing your issue. Again, I tried your vagrantfile on a brand new ubuntu 20.04 server and it worked fine.

  1. in the search clause there are domains coming from my host machines that cause issues.

This is also the expected configuration. I wouldn't typically expect this to cause issues, but I don't know your exact configuration or what you're trying to achieve. If the solution requires that you modify the list of search domains in the pod, the place to do that would be in the pod dns policy. Hopefully solving your first issue will allow you to avoid having to customize this.

  • Hi Mike, thank you very much for your exhaustive answer. To be honest I am not an expert in DNS problem resolution (this is a handicap that I should address) and I need time to analyze and test your feedback. For the moment I can only upvote your answer and I'll come back if I have doubts. Jul 22, 2021 at 8:04
  • Hi Mike Frank, after a long time I am still here with the problem. Can you tell me exactly what to do to overwrite my search clause in the Pod /etc/resolve.conf. I tried to use dnsConfig with the searches clause but in the end, the extra domain coming from the VirtualBox is still there. Is there a way to only use the one I specify in dnsConfig without K8s mix them with the one in Vagrant/VirtualBox host? Oct 15, 2021 at 7:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.