Container Service for Kubernetes:DNS best practices

Metrics and dashboards

CoreDNS exposes health and performance metrics through a standard Prometheus interface to detect exceptions on CoreDNS and upstream DNS servers.

• ACK managed clusters:Managed Service for Prometheus provides built-in metrics monitoring dashboards and alerting rules for CoreDNS. You can enable Prometheus and dashboard features in the ACK console. See Monitor the CoreDNS component.

• Self-managed Prometheus: Scrape CoreDNS metrics and configure alerts for critical indicators. See the official CoreDNS Prometheus documentation.

Log analysis

In the event of a DNS exception, CoreDNS logs are essential for root cause diagnosis. We recommend enabling DNS resolution logging and Simple Log Service (SLS) collection. For details, see Analyze and monitor CoreDNS logs.

Kubernetes event delivery

In CoreDNS v1.9.3.6-32932850-aliyun and later, use the k8s_event plugin to send critical logs to Event Center as Kubernetes events. See k8s_event.

Newly deployed CoreDNS enables this feature by default. If upgrading from an earlier version to v1.9.3.6-32932850-aliyun or later, manually modify the configuration file to enable it.

1. Edit the CoreDNS ConfigMap.

   kubectl -n kube-system edit configmap/coredns

2. Add the kubeAPI and k8s_event plugins to the configuration.

   apiVersion: v1
   data:
     Corefile: |
       .:53 {
           errors
           health {
               lameduck 15s
           }
   
           # --- Addition Start (ignore other differences) ---
           kubeapi
           k8s_event {
             level info error warning // Deliver key logs with the info, error, and warning statuses.
           }
           # --- Addition End ---
   
           kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods verified
               fallthrough in-addr.arpa ip6.arpa
           }
           # (Remaining configuration omitted)
       }

3. Verify the update by checking the CoreDNS pod logs. If the logs contain the word reload, the modification is successful.

Assess CoreDNS pressure

Use open-source tools such as DNSPerf to benchmark your DNS performance. If you cannot perform a customized assessment, follow these baseline recommendations:

• Minimum replicas: Always maintain at least 2 pods for redundancy.

• Resource limits: Set resource limits to at least 1 Core CPU and 1 GiB memory per pod.

• Performance scaling: CoreDNS performance scales linearly with CPU. With NodeLocal DNSCache enabled, a single CPU core can typically handle over 10,000 QPS.

• Replica ratio: If you cannot monitor peak CPU usage, use a conservative 1:8 Pod-to-Node ratio (add one CoreDNS pod for every 8 cluster nodes).

Scale CoreDNS pods

The number of CoreDNS pods determines available computing resources. Adjust the pod count based on the assessment results.

• Automatically scale based on the recommended policy

  Deploy dns-autoscaler to automatically adjust CoreDNS replicas based on the recommended policy (one pod for every eight cluster nodes).

  Formula: replicas = max(ceil(cores × 1/coresPerReplica), ceil(nodes × 1/nodesPerReplica))

  This ensures that the 1:8 ratio is maintained as the cluster grows.

  dns-autoscaler

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: dns-autoscaler
    namespace: kube-system
    labels:
      k8s-app: dns-autoscaler
  spec:
    selector:
      matchLabels:
        k8s-app: dns-autoscaler
    template:
      metadata:
        labels:
          k8s-app: dns-autoscaler
      spec:
        serviceAccountName: admin
        containers:
        - name: autoscaler
          image: registry.cn-hangzhou.aliyuncs.com/acs/cluster-proportional-autoscaler:1.8.4
          resources:
            requests:
              cpu: "200m"
              memory: "150Mi"
          command:
          - /cluster-proportional-autoscaler
          - --namespace=kube-system
          - --configmap=dns-autoscaler
          - --nodelabels=type!=virtual-kubelet
          - --target=Deployment/coredns
          - --default-params={"linear":{"coresPerReplica":64,"nodesPerReplica":8,"min":2,"max":100,"preventSinglePointFailure":true}}
          - --logtostderr=true
          - --v=9

• Manually adjust

  Run the following command to manually adjust the number of CoreDNS pods:

  kubectl scale --replicas={target} deployment/coredns -n kube-system # Replace {target} with the desired pod count

• Do not use HPA or CronHPA

  Although workload auto-scaling mechanisms (such as HPA and CronHPA) can automatically adjust pod counts, they perform frequent scaling operations. Due to resolution exceptions caused by pod scale-in (the IPVS UDP issues mentioned above), do not use workload auto-scaling to control CoreDNS pod count.

Optimize CoreDNS pod specifications

Another way to adjust CoreDNS resources is by modifying pod specifications. In an ACK managed Pro cluster, the default memory limit for CoreDNS pods is 2 GiB, with no CPU limit. To ensure consistent performance, set the CPU limit to 4096m (minimum 1024m). You can adjust these resource requests and limits in the console.

Schedule CoreDNS pods

Disable the affinity configuration of the kube-dns service

Disable the autopath plugin

Some early CoreDNS versions enabled the autopath plugin, which can cause resolution errors in specific edge cases. Verify whether it is enabled and edit the configuration file to disable it. For more information, see Autopath issue #3765.

1. Run the kubectl -n kube-system edit configmap coredns command to open the CoreDNS configuration file.

2. Remove the line autopath @kubernetes and save the change.

3. Check the running status and logs of the CoreDNS pod. If the logs contain the word reload, the modification is successful.

Configure graceful shutdown

The lameduck mechanism ensures that when a CoreDNS process is about to terminate (during updates or restarts), it handles existing requests before exiting.

• When the CoreDNS process is about to terminate, it enters lameduck mode.

• In lameduck mode, CoreDNS continues to process already-received requests for a specified duration while signaling the system to stop sending new requests.

Optimize upstream protocol (prefer_udp)

When using NodeLocal DNSCache, the communication chain is: Application -> NodeLocal DNSCache (TCP) -> CoreDNS (TCP). By default, CoreDNS will then attempt to reach the upstream VPC DNS (100.100.2.136/138) via TCP.

# Before modification
forward . /etc/resolv.conf
# After modification
forward . /etc/resolv.conf {
  prefer_udp
}

Configure the ready plugin for readiness probe

For CoreDNS v1.5.0 and later, the ready plugin is mandatory for the readiness probe to function.

1. Run the following command to open the CoreDNS ConfigMap.

   kubectl -n kube-system edit configmap/coredns

2. Check whether the file contains the ready directive. If not, add it, press the Esc key, enter :wq!, then press Enter to save the modified configuration file and exit edit mode.

   apiVersion: v1
   data:
    Corefile: |
     .:53 {
       errors
       health {
         lameduck 15s
       }
       ready # If this line does not exist, add it. Make sure that the indentation is consistent with Kubernetes.
       kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods verified
         fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf {
         max_concurrent 1000
               prefer_udp
       }
       cache 30
       loop
       log
       reload
       loadbalance
     }

3. Check the running status and logs of the CoreDNS pod. If the logs contain the word reload, the modification is successful.

Enhance resolution performance with the multisocket plugin

CoreDNS v1.12.1 introduced the multisocket plugin. Enable this plugin to allow CoreDNS to listen on the same port using multiple sockets, enhancing performance in high-CPU scenarios. For details, see the community documentation.

Enable multisocket in the coredns ConfigMap:

.:53 {
        ...
        prometheus :9153
        multisocket [NUM_SOCKETS]
        forward . /etc/resolv.conf
        ...
}

NUM_SOCKETS determines the number of sockets listened on the same port.

Recommended configuration: Align NUM_SOCKETS with estimated CPU utilization, CPU resource limits, and available cluster resources. Examples:

• If peak consumption is 4 cores with 8 cores available, set NUM_SOCKETS to 2.

• If peak consumption is 8 cores with 64 cores available, set NUM_SOCKETS to 8.

To determine the optimal configuration, test QPS and load with different settings.

Default: If not specified, it defaults to GOMAXPROCS, which equals the CoreDNS pod's CPU limit. If the pod's CPU limit is not set, it equals the number of CPU cores on the node where the pod resides.