Container Service for Kubernetes:Manually update CoreDNS

Update notes

• If the load balancing mode of kube-proxy is set to IP Virtual Server (IPVS), all DNS queries within the cluster may fail or time out after CoreDNS is updated. This situation lasts about 5 minutes. You can use one of the following methods to avoid this issue:

  • Modify the timeout period for IPVS UDP sessions in the kube-proxy configuration. For more information, see Change the UDP timeout period in IPVS mode.

  • Use NodeLocal DNSCache to optimize DNS resolution. For more information, see Use NodeLocal DNSCache to optimize DNS resolution.

• ACK requires about 2 minutes to update CoreDNS. The duration varies based on the number of CoreDNS pods. If the newly provisioned CoreDNS pods cannot be scheduled or launched, submit a ticket. DNS resolution is not interrupted because the existing CoreDNS pods are still running.

Query the CoreDNS version

kubectl get deployment coredns -n kube-system -o jsonpath="{.spec.template.spec.containers[0].image}"

registry-vpc.cn-hangzhou.aliyuncs.com/acs/coredns:1.6.2 # Version 1.6.2 is used in this example.

Confirm the update version

Confirm the CoreDNS version to which you want to update. The following table describes the Kubernetes versions that are compatible with CoreDNS. We recommend that you use the latest CoreDNS version that is compatible with the Kubernetes version.

Manually update

Change the UDP timeout period in IPVS mode

ACK clusters that run Kubernetes 1.18 or later

Use the ACK console

1. Log on to the ACK console.

2. In the left-side navigation pane of the ACK console, click Clusters.

3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.

4. In the left-side navigation pane of the details page, choose Configurations > ConfigMaps.

5. On the ConfigMap page, select the kube-system namespace, find the kube-proxy-worker ConfigMap, and then click Edit YAML in the Actions column.

6. In the View in YAML panel, add udpTimeout: 10s to the ipvs field and click OK.

   apiVersion: v1
   data:
     config.conf: |
       apiVersion: kubeproxy.config.k8s.io/v1alpha1
       kind: KubeProxyConfiguration
       # Irrelevant fields are not shown. 
       mode: ipvs
       # If the ipvs field does not exist, you must add the field. 
       ipvs:
         udpTimeout: 10s

7. Recreate all of the pods named kube-proxy-worker.

   1. In the left-side navigation pane of the details page, choose Workloads > DaemonSets.

   2. On the DaemonSets page, find and click kube-proxy-worker.

   3. On the Pods tab of the kube-proxy-worker page, select a pod and choose More > Delete in the Actions column. In the message that appears, click Confirm.

      Repeat the preceding steps to delete all of the pods. After you delete the pods, the system automatically recreates the pods.

8. Check whether the timeout period of UDP sessions is changed.

   1. Run the following command to install ipvsadm.

      ipvsadm is a tool that you can use to manage IPVS. For more information, see ipvsadm.

      yum install -y ipvsadm

   2. Log on to an Elastic Compute Service (ECS) instance in your cluster and then run the following command to check the third value in the output:

      ipvsadm -L --timeout

      If the third value in the output is 10, the timeout period of UDP sessions is changed.

      Note

      After the timeout period of UDP sessions is changed, wait at least 5 minutes before you proceed.

Use the CLI

1. Run the following command to modify the kube-proxy-worker ConfigMap:

   kubectl -n kube-system edit configmap kube-proxy-worker

2. Add udpTimeout: 10s to the ipvs field of the kube-proxy-worker ConfigMap. Then, save the modification and exit.

   apiVersion: v1
   data:
     config.conf: |
       apiVersion: kubeproxy.config.k8s.io/v1alpha1
       kind: KubeProxyConfiguration
       # Irrelevant fields are not shown. 
       mode: ipvs
       # If the ipvs field does not exist, you must add the field. 
       ipvs:
         udpTimeout: 10s

3. Run the following command to recreate all of the pods named kube-proxy-worker.

   1. Run the following command to query the pods:

      kubectl -n kube-system get pod -o wide | grep kube-proxy-worker

   2. Run the following command to delete all of the pods that are returned in the preceding step. Then, the system recreates the pods named kube-proxy-worker.

      kubectl -n kube-system delete pod <kube-proxy-worker-****>

      Note

      Replace <kube-proxy-worker-****> with the name of a pod that is returned in the preceding step.

4. Check whether the timeout period of UDP sessions is changed.

   1. Run the following command to install ipvsadm.

      ipvsadm is a tool that you can use to manage IPVS. For more information, see ipvsadm.

      yum install -y ipvsadm

   2. Log on to an Elastic Compute Service (ECS) instance in your cluster and then run the following command to check the third value in the output:

      ipvsadm -L --timeout

      If the third value in the output is 10, the timeout period of UDP sessions is changed.

      Note

      After the timeout period of UDP sessions is changed, wait at least 5 minutes before you proceed.

ACK clusters that run Kubernetes 1.16 or earlier

sudo yum install -y ipvsadm
sudo ipvsadm -L --timeout > /tmp/ipvsadm_timeout_old
sudo ipvsadm --set 900 120 10
sudo ipvsadm -L --timeout > /tmp/ipvsadm_timeout_new
diff /tmp/ipvsadm_timeout_old /tmp/ipvsadm_timeout_new

For more information about how to use OOS to manage multiple ECS instances at the same time, see Manage multiple instances at a time.