- You are trying to replicate an issue that you observe in a different environment, in your dev K8s cluster. To successfully replicate the issue, you need to ensure that you use the same hostnames, application FQDNs that were used in the original environment.
- You are testing an application running on Kubernetes that requires access to third party external endpoints over the internet. You need to ensure that these third-party external connections are created towards some known, test services, not the actual ones.
- You need an alternative and a more centralized way to control the responses received to DNS queries by the application pods. (e.g. You do not like to use “hostAliases” option with K8s pods)
First, we are going to run a “dnsmasq” server in the same Kubernetes cluster as a pod. It will be associated with a configmap through which we can manipulate the DNS records. Other than that, the dnsmasq pod will be exposed via a clusterIP type Kubernetes service.
To run a “dnsmasq” server as a container, we can first create docker image using following Dockerfile.
As you may notice in the ENTRYPOINT of above, we are running “dnsmasq” in foreground (-d) as well as we pass a configuration file (-C) and a hosts (-H) file to it. During the build, these files do not necessarily need to exist in the docker image. However, during the runtime we will mount relevant configuration files to /var/dnsmasq/conf/dnsmasq.conf and /var/dnsmasq/hosts location accordingly.
Now you can build the docker image with “docker build –t dnsmasq:latest ” command. You may push it to a docker image repository that is accessible by your Kubernetes cluster, afterwards
Now let us create a couple of configmaps to pass the dnsmasq.conf and hosts files to the running dnsmasq pod.
In below example, we are creating a fake DNS record, that resolves www.zinkworks.com to 10.100.1.1 address. Also, we will create a host’s file record that maps example.zinkworks.com to 10.100.1.2
You can create above configmaps in the namespace where you will run the dnsmasq pod, using “kubectl create –f“or “kubectl apply –f” commands.
Next let us create a dnsmasq pod using the docker image built in step 1. We will also mount the configmaps created previously to /var/dnsmasq/conf/dnsmasq.conf and /var/dnsmasq/hosts as files.
Above is an example pod definition to run the dnsmasq container with desired configurations. Notice how, dnsmasq-conf and dnsmasq-hosts configmaps are mounted as files to the pod. Also notice the “NET_ADMIN” capability given to the container. This will allow you to run the container with UDP port 53.
You can create the pod by running “kubectl create –f” or “kubectl apply –f” commands on above pod definition.
Once the dnsmasq pod is created, you can confirm if it is running fine by looking at the pod logs.
Here is an example output. In this case, it is assumed that dnsmsq pod is created in dnsmasq namespace.
Finally, we will expose the dnsmasq pod via a kubernetes service.
In the future steps, we may need to use the service IP assigned to the dnsmsq service to send our DNS queries to the dnsmasq container. To find the service IP of the dnsmasq service, you can run “kubectl get svc” command on the namespace where you run dnsmasq pod.
In our example the dnsmasq service IP is 10.108.96.48.
You can further check if the domain name resolution is working as expected in the dnsmasq container by running a few nslookup commands from a different pod.
In below example, we are running a pod called ubuntu-debugger in the same cluster from which we can check the responses sent by the dnsmasq. It is assumed that ubuntu-debugger pod has nslookup utility.
Notice how in the first nslookup command for www.zinkworks.com has returned the actual public IP instead of the fake one we provided to the dnsmasq container in dnsmasq.conf file. This is because this external DNS queries are handled by the coredns and they are by default forwarded to the nameservers specified in /etc/resolv.conf file.
Now in the subsequent nslookup commands we specify the service IP of the dnsmasq service which is 10.108.96.48. This ensures that the DNS query is processed by our dnsmasq container directly. As you notice in the second nslookup command for www.zinkworks.com, We have received the fake IP we configured in the dnsmasq container through the dnsmasq.conf file.
Any other query that dnsmasq service cannot answer will be sent to the next nameserver in the chain. In our case it is 8.8.8.8
Now as you noticed in step 2 above, all external DNS queries are by default not forwarded to our dnsmasq container, unless we specify the nameserver in our query. This is not convenient. Therefore, next we look at how we can configure the coredns to forward all our external DNS queries to dnsmasq container by default.
First, we open the coredns configmap in edit mode by running the following command.
$ kubectl edit configmap –namespace kube-system coredns
This will open a similar configuration as shown below on your editor.
The highlighted “forward” section in above configuration should now be modified to forward the external DNS queries to our dnsmasq container.
After the modification the configmap should appear as shown below.
Next save and exit from the editor so that above change is now reflected to coredns configmap.
Finally, you can restart the coredns pods in kube-system namespace by deleting them. This will ensure that our modified configuration is loaded into the new instances of coredns pods that will come up after deleting the old replicas.
Here is the command to delete the current coredns pods.
$ kubectl delete pods –namespace kube-system –force –grace-period=0 $(kubectl get pods –namespace kube-system | grep coredns | awk -F ‘ ‘ ‘{print $1}’)
Now, let us run the same nslookup command on www.zinkworks.com from our ubuntu-debugger container to see if we are getting the expected response.
As expected, the queries for www.zinkworks.com and example.zinkworks.com return the fake IPs we configured in dnsmasq.conf and hosts file in dnsmasq pod.
To find out more about careers at Zinkworks click here.