Container Service for Kubernetes:Best practices for DNS services

Optimize DNS queries

Use proper container images

If containers that are deployed in a Kubernetes cluster use Alpine as the base image, domain names may not be resolved due to the use of musl libc. We recommend that you replace the image with an image that is based on Debian or CentOS.

Reduce the adverse effect of occasional DNS resolution timeouts caused by IPVS defects

If the load balancing mode of kube-proxy is set to IPVS in your cluster, DNS resolution timeouts may occur when CoreDNS pods are scaled in or restarted. The issues are caused by the kernel bugs of Linux. For more information, see IPVS.

Use NodeLocal DNSCache to optimize DNS resolution

Container Service for Kubernetes (ACK) allows you to deploy NodeLocal DNSCache to improve the stability and performance of service discovery. NodeLocal DNSCache is implemented as a DaemonSet and runs a DNS caching agent on cluster nodes to improve the efficiency of DNS resolution for ACK clusters.

For more information about NodeLocal DNSCache and how to deploy NodeLocal DNSCache in ACK clusters, see Configure NodeLocal DNSCache.

Use proper CoreDNS versions

Monitor the status of CoreDNS

Monitoring metrics

CoreDNS uses the standard Prometheus API to collect metrics such as DNS resolution results. This allows you to identify exceptions in CoreDNS and upstream DNS servers at the earliest opportunity.

By default, monitoring metrics and alert rules related to CoreDNS are predefined in Managed Service for Prometheus provided by Alibaba Cloud. You can log on to the ACK console to enable Managed Service for Prometheus and dashboards. For more information, see CoreDNS monitoring.

If you use open source Prometheus to monitor the Kubernetes cluster, you can view the related metrics in Prometheus and create alert rules based on the following key metrics. For more information, see CoreDNS Prometheus official documentation.

Operational log

When DNS resolution errors occur, you can view the log of CoreDNS to identify the causes. We recommend that you enable logging for CoreDNS and use Log Service to collect log data. For more information, see Collect and analyze CoreDNS logs.

Sink Kubernetes events

CoreDNS v1.9.3.6-32932850-aliyun and later versions allow you to enable the k8s_event plug-in to sink Kubernetes events that contain the Info, Error, and Warning logs of CoreDNS to the event center. For more information, see k8s_event.

By default, the k8s_event plug-in is enabled after you deploy CoreDNS v1.9.3.6-32932850-aliyun or a later version. If you deployed an earlier CoreDNS version and then updated it to v1.9.3.6-32932850-aliyun or later, you need to modify the ConfigMap of CoreDNS to enable the k8s_event plug-in.

1. Run the following command to open the coredns ConfigMap:

   kubectl -n kube-system edit configmap/coredns

2. Add the kubeAPI and k8s_event plug-ins.

   apiVersion: v1
   data:
     Corefile: |
       .:53 {
           errors
           health {
               lameduck 15s
           }
   
           // The beginning of the plug-in configuration. Ignore other settings. 
           kubeapi
           k8s_event {
             level info error warning // Report Info, Error, and Warning logs to Kubernetes events. 
           }
           // The end of the plug-in configuration. 
   
           kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods verified
               fallthrough in-addr.arpa ip6.arpa
           }
           // Details are not shown. 
       }

3. Check the status and logs of the CoreDNS pods. If the log data contains the reload keyword, the new configuration is loaded.

Modify the CoreDNS Deployment

Modify the number of CoreDNS pods

We recommend that you provision at least two CoreDNS pods. You must make sure that the number of CoreDNS pods is sufficient to handle DNS queries within the cluster.

The DNS QPS of CoreDNS is related to CPU usage. A single CPU can handle more than 10,000 DNS QPS if you enable DNS caching. The DNS QPS required by different workloads may vary. You can evaluate the DNS QPS based on the peak CPU usage of each CoreDNS pod. We recommend that you increase the number of CoreDNS pods if a CoreDNS pod occupies more than one vCPU during peak hours. If you cannot confirm the peak CPU usage of each CoreDNS pod, you can set the ratio of CoreDNS pods to cluster nodes to 1:8. This way, each time you add eight nodes to a cluster, a CoreDNS pod is created. The total number of CoreDNS pods must not exceed 10. If your cluster contains more than 100 nodes, we recommend that you use NodeLocal DNSCache. For more information, see Use NodeLocal DNSCache to optimize DNS resolution.

• You can manually increase the number of CoreDNS pods if the number of cluster nodes remains unchanged for a long period of time. For more information, see Manually increase the number of CoreDNS pods.

• You can set the system to automatically increase the number of CoreDNS pods if the number of cluster nodes increases. For more information, see Use cluster-autoscaler to automatically increase the number of CoreDNS pods.

Schedule CoreDNS pods to proper nodes

When you deploy CoreDNS pods in a cluster, we recommend that you deploy the CoreDNS pods on different cluster nodes across multiple zones. This prevents service disruptions when a single node or zone fails. By default, soft anti-affinity settings based on nodes are configured for CoreDNS. Some or all CoreDNS pods may be deployed on the same node due to insufficient nodes. In this case, we recommend that you delete the CoreDNS pods and reschedule the pods.

CoreDNS pods must not be deployed on cluster nodes whose CPU and memory resources are fully utilized. Otherwise, DNS QPS and response time are adversely affected. If the cluster contains sufficient nodes, you can schedule CoreDNS pods to exclusive nodes by configuring custom parameters. This allows CoreDNS to provide stable domain name resolution services. For more information about how to schedule CoreDNS pods to exclusive cluster nodes, see Deploy CoreDNS pods on exclusive nodes by configuring custom parameters.

Manually increase the number of CoreDNS pods

kubectl scale --replicas={target} deployment/coredns -n kube-system

Use cluster-autoscaler to automatically increase the number of CoreDNS pods

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dns-autoscaler
  namespace: kube-system
  labels:
    k8s-app: dns-autoscaler
spec:
  selector:
    matchLabels:
      k8s-app: dns-autoscaler
  template:
    metadata:
      labels:
        k8s-app: dns-autoscaler
    spec:
      serviceAccountName: admin
      containers:
      - name: autoscaler
        image: registry.cn-hangzhou.aliyuncs.com/acs/cluster-proportional-autoscaler:1.8.4
        resources:
          requests:
            cpu: "200m"
            memory: "150Mi"
        command:
        - /cluster-proportional-autoscaler
        - --namespace=kube-system
        - --configmap=dns-autoscaler
        - --nodelabels=type!=virtual-kubelet
        - --target=Deployment/coredns
        - --default-params={"linear":{"coresPerReplica":64,"nodesPerReplica":8,"min":2,"max":100,"preventSinglePointFailure":true}}
        - --logtostderr=true
        - --v=9

{
      "coresPerReplica": 64,
      "nodesPerReplica": 8,
      "min": 2,
      "max": 100,
      "preventSinglePointFailure": true
}

Use HPA to increase the number of CoreDNS pods based on CPU loads

---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: coredns-hpa
  namespace: kube-system
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: coredns
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 50

Deploy CoreDNS pods on exclusive nodes by using custom parameters

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, choose Nodes > Nodes.

3. On the Nodes page, click Manage Labels and Taints.

4. On the Labels tab of the Manage Labels and Taints page, select the nodes that you want to manage and click Add Label.

   Note

   The number of nodes that you select must be greater than the number of CoreDNS pods. This avoids deploying multiple CoreDNS pods to the same node.

5. In the Add dialog box, set the following parameters and click OK.

   • Name: Enter node-role-type.

   • Value: Enter coredns.

6. In the left-side navigation pane, choose Operations > Add-ons. On the page that appears, search for CoreDNS.

7. Click Configuration in the CoreDNS section. On the CoreDNS Parameters dialog box, click + Add to the right of NodeSelector, set the following parameters, and then click OK.

   • Key: Enter node-role-type.

   • Value: Enter coredns.

   CoreDNS pods are rescheduled to nodes with the specified label.

Properly configure CoreDNS

ACK provides only the default settings for CoreDNS. You can modify the parameters to optimize the settings and enable CoreDNS to provide normal DNS services for your client pods. You can modify the configurations of CoreDNS on demand. For more information, see DNS policies and domain name resolution and CoreDNS official documentation.

You can go to the Container Intelligence Service console and use the Regular Inspection and Diagnosis components to check the configurations of CoreDNS. If the result shows errors related to the coredns ConfigMap, we recommend that perform the preceding operations.

Adjust resource requests and limits for CoreDNS

The following table describes the default values of resource requests and limits when the system deploys CoreDNS. You can modify the values based on the status of your cluster. For more information, see O&M management - Component management - Network - CoreDNS - Parameter configurations.

Disable the affinity settings for the kube-dns Service

Disable the autopath plug-in

The autopath plug-in is enabled for CoreDNS of earlier versions and may cause DNS resolution errors in specific scenarios. If the autopath plug-in is enabled, you must disable the plug-in in the coredns ConfigMap. For more information, see Autopath.

1. Run the kubectl -n kube-system edit configmap coredns command to modify the coredns ConfigMap.

2. Delete autopath @kubernetes. Then, save the change and exit.

3. Check the status and logs of the CoreDNS pods. If the log data contains the reload keyword, the new configuration is loaded.

Configure graceful shutdown for CoreDNS

Configure the default protocol for the forward plug-in and upstream DNS servers of a VPC

NodeLocal DNSCache uses the TCP protocol to communicate with CoreDNS. CoreDNS communicates with the upstream DNS servers based on the protocol used by the source of DNS queries. Therefore, DNS queries sent from a client pod for external domain names pass through NodeLocal DNSCache and CoreDNS, and then arrive at the DNS servers in a VPC over TCP. The IP addresses of the DNS servers are 100.100.2.136 and 100.100.2.138. These IP addresses are automatically configured on the Elastic Compute Service (ECS) instances.

# The original setting
forward . /etc/resolv.conf
# The modified setting
forward . /etc/resolv.conf {
  prefer_udp
}

Configure the ready plug-in

You must configure the ready plug-in for CoreDNS versions later than 1.5.0.

1. Run the following command to open the coredns ConfigMap:

   kubectl -n kube-system edit configmap/coredns

2. Check whether a line that contains only ready exists. If the line does not exist, add a line and specify ready, press Esc, enter :wq!, and then press Enter to save the file and exit the edit mode.

   apiVersion: v1
   data:
    Corefile: |
     .:53 {
       errors
       health {
         lameduck 15s
       }
       ready # Add this line and make sure that the word "ready" is aligned with the word "kubernetes". 
       kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods verified
         fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf {
         max_concurrent 1000
               prefer_udp
       }
       cache 30
       loop
       log
       reload
       loadbalance
     }

3. Check the status and logs of the CoreDNS pods. If the log data contains the reload keyword, the new configuration is loaded.