Container Service for Kubernetes:Usage notes and instructions on high-risk operations

Item

References

Usage notes

• Data plane components

• Cluster update

• Kubernetes configurations

• ACK Serverless clusters

• Cluster registration

• App catalogs

High-risk operations

• High-risk operations on clusters

• High-risk operations on node pools

• High-risk operations on networks and load balancing

• High-risk operations on storage

• High-risk operations on logs

Category

High-risk operation

Impact

How to recover

API Server

Delete the SLB instance that is used to expose the API server.

You cannot manage the cluster.

Unrecoverable. You must create a new cluster. For more information about how to create a cluster, see Create an ACK managed cluster.

Worker nodes

Modify the security group of nodes.

The nodes may become unavailable.

Add the nodes to the original security group again. The security group is created when you create the cluster. For more information, see Manage ECS instances in security groups.

The subscriptions of nodes expire or nodes are removed.

The nodes become unavailable.

Unrecoverable.

Reinstall the node OS.

Components are uninstalled from nodes.

Remove the nodes and then add the nodes to the cluster again. For more information, see Remove nodes and Add existing ECS instances to an ACK cluster.

Update component versions.

The nodes may become unavailable.

Roll back to the original component versions.

Change the IP addresses of nodes.

The nodes become unavailable.

Change the IP addresses of the nodes to the original IP addresses.

Modify the parameters of key components, such as kubelet, docker, and containerd.

The nodes may become unavailable.

Refer to the ACK official documentation and configure the component parameters.

Modify node OS configurations.

The nodes may become unavailable.

Restore the configurations, or remove the worker nodes and then purchase new nodes.

Modify the system time of nodes.

The components on the nodes do not work as expected.

Reset the system time of the nodes.

Master nodes in ACK dedicated clusters

Modify the security group of master nodes.

The master nodes may become unavailable.

Add the master nodes to the original security group again. The security group is created when you create the cluster. For more information, see Manage ECS instances in security groups.

The subscriptions of master nodes expire or master nodes are removed.

The master nodes become unavailable.

Unrecoverable.

Reinstall the node OS.

Components are uninstalled from master nodes.

Unrecoverable.

Update master nodes or the etcd component.

The cluster may become unavailable.

Roll back to the original component versions.

Delete or format the directories that store business-critical data on nodes, for example, /etc/kubernetes.

The master nodes become unavailable.

Unrecoverable.

Change the IP addresses of master nodes.

The master nodes become unavailable.

Change the IP addresses of the master nodes to the original IP addresses.

Modify the parameters of key components, such as etcd, kube-apiser, and docker.

The master nodes may become unavailable.

Refer to the ACK official documentation and configure the component parameters.

Replace the certificates of master nodes or the etcd component.

The cluster may become unavailable.

Unrecoverable.

Increase or decrease the number of master nodes.

The cluster may become unavailable.

Unrecoverable.

Modify the system time of nodes.

The components on the nodes do not work as expected.

Reset the system time of the nodes.

Other services

Use Resource Access Management (RAM) to modify permissions.

Resources such as SLB instances may fail to be created.

Restore the permissions.

High-risk operation

Impact

How to recover

Delete scaling groups.

Node pool exceptions occur.

Unrecoverable. You must create new node pools. For more information about how to create a node pool, see Procedure.

Use kubectl to remove nodes from a node pool.

The number of nodes in the node pool that is displayed in the ACK console is different from the actual number.

Remove nodes in the ACK console, by calling the ACK API, or by configuring the Expected Nodes parameter of the node pool. For more information, see Remove nodes and Create a node pool.

Manually release ECS instances.

Incorrect information may be displayed on the node pool details page. A node pool is configured with the Expected Nodes parameter when the node pool is created. After you release the ECS instances, ACK automatically scales out the node pool to the value of the Expected Nodes parameter.

Unrecoverable. To release ECS instances in a node pool, configure the Expected Nodes parameter of the node pool in the ACK console or by calling the ACK API. You can also remove the nodes that are deployed on the ECS instances. For more information, see Create a node pool and Remove nodes.

Manually scale in or scale out a node pool that has auto scaling enabled.

The auto scaling component automatically adjusts the number of nodes in the node pool after you manually scale in or scale out the node pool.

Unrecoverable. You do not need to manually scale a node pool that has auto scaling enabled.

Change the upper or lower limit of instances that a scaling group can contain.

Scaling errors may occur.

• For a node pool that has auto scaling disabled, the default upper limit of instances for the scaling group is 2,000, and the default lower limit of instances is 0.

• For a node pool that has auto scaling enabled, make sure that the upper and lower limits of instances for the scaling group are the same as the upper and lower limits of instances for the node pool.

Add existing nodes to a cluster without backing up the data on the nodes.

The data on the nodes is lost after the nodes are added to the cluster.

Unrecoverable.

• If you want to add existing nodes in Manual mode, you must first back up the data on the nodes.

• If you want to add existing nodes in Auto mode, you must first back up the data on the system disks because the system disks will be replaced when the nodes are added to the cluster.

Store business-critical data on the system disk.

If you enable auto repair for a node pool, the system may handle node exceptions by resetting node configurations. As a result, data on the system disk is lost.

Unrecoverable. Store business-critical data to data disks, cloud disks, File Storage NAS (NAS) file systems, or Object Storage Service (OSS) buckets.

High-risk operation

Impact

How to recover

Specify the following kernel parameter setting: net.ipv4.ip_forward=0.

Network connectivity issues occur.

Replace the setting with the following content: net.ipv4.ip_forward=1.

Specify the following kernel parameter settings:

• net.ipv4.conf.all.rp_filter = 1|2

• net.ipv4.conf.[ethX].rp_filter = 1|2

  Note

  ethX specifies the network interface controllers whose names start with eth.

Network connectivity issues occur.

Replace the settings with the following content:

• net.ipv4.conf.all.rp_filter = 0

• net.ipv4.conf.[ethX].rp_filter = 0

Specify the following kernel parameter setting: net.ipv4.tcp_tw_reuse = 1.

Pods fail to pass health checks.

Replace the setting with the following content: net.ipv4.tcp_tw_reuse = 0.

Specify the following kernel parameter setting: net.ipv4.tcp_tw_recycle = 1.

Network address translation errors occur.

Replace the setting with the following content: net.ipv4.tcp_tw_recycle = 0.

Specify the following kernel parameter setting: net.ipv4.ip_local_port_range.

Network connectivity issues occasionally occur.

Replace the setting with the following content: net.ipv4.ip_local_port_range="32768 60999".

Install firewall software, such as Firewalld or ufw.

The container network becomes inaccessible.

Uninstall the firewall software and restart the nodes.

The security group of a node does not open UDP port 53 for the pod CIDR block.

DNS cannot work as expected in the cluster.

Refer to the ECS official documentation and modify the security group configuration to open UDP port 53 for the pod CIDR block.

Modify or delete the tags that ACK added to SLB instances.

The SLB instances do not work as normal.

Restore the tags.

Modify the configurations of the SLB instances that are managed by ACK, including the configurations of the instances, listeners, and vServer groups.

The SLB instances do not work as normal.

Restore the SLB configurations.

Remove the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: ${YOUR_LB_ID} annotation that is used to specify an existing SLB instance from the Service configuration.

The SLB instances do not work as normal.

Add the annotation to the Service configuration.

Delete the SLB instances that are created by ACK in the SLB console.

Errors may occur in the cluster network.

Delete SLB instances by deleting the Services that are associated with the SLB instances. For more information about how to delete a Service, see Delete a Service.

Manually delete the nginx-ingress-lb Service in the kube-system namespace of a cluster that has the NGINX Ingress controller installed.

The NGINX Ingress controller does not run as normal or may stop running.

Use the following YAML template to create a Service that has the same name:

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: nginx-ingress-lb
  name: nginx-ingress-lb
  namespace: kube-system
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: ingress-nginx
  type: LoadBalancer

Configure the nameserver parameter in the DNS configuration file of an ECS node. The DNS configuration file is named /etc/resolv.conf.

If the DNS server is not configured properly, DNS resolution may fail. As a result, the cluster cannot run as expected.

If you want to use a self-managed DNS server, we recommend that you configure the DNS server in CoreDNS. For more information, see Configure CoreDNS.

High-risk operation

Impact

How to recover

Unmount cloud disks that are mounted to pods in the ECS console.

I/O errors occur when you write data to the pods.

Restart the pods and clear residual data on the nodes.

Unmount disks from their mount paths on nodes.

Pod data is written to local disks.

Restart the pods.

Manage cloud disks on the nodes.

Pod data is written to local disks.

Unrecoverable.

Mount a cloud disk to multiple pods.

Pod data is written to local disks or I/O errors occur when you write data to the pods.

Mount the cloud disk only to one pod.

Manually delete the NAS directories that are mounted to pods.

I/O errors occur when you write data to the pods.

Restart the pods.

Delete the NAS file systems that are mounted to pods or delete the mount targets that are used to mount NAS file systems.

I/O hangs occur when you write data to the pods.

Restart the ECS instances. For more information about how to restart an ECS instance, see Restart an instance.

High-risk operation

Impact

How to recover

Delete the /tmp/ccs-log-collector/pos directory on a node.

Duplicate logs are collected.

Unrecoverable. The /tmp/ccs-log-collector/pos directory contains information about the path from which logs are collected.

Delete the /tmp/ccs-log-collector/buffer directory on a node.

Logs are lost.

Unrecoverable. The /tmp/ccs-log-collector/buffer directory stores cached log files that need to be consumed.

Delete the aliyunlogconfig CustomResourceDefinition (CRD) objects.

Logs cannot be collected.

Recreate the aliyunlogconfig CRD objects that are deleted and the related resources. Logs that are generated within the period of time during which the aliyunlogconfig CRD objects do not exist cannot be collected.

If you delete the aliyunlogconfig CRD objects, the related log collection tasks are also deleted. After you recreate the aliyunlogconfig CRD objects, you must also relaunch the log collection tasks.

Uninstall logging components.

Logs cannot be collected.

Reinstall the logging component and manually create the aliyunlogconfig CRD objects. Logs that are generated within the period of time during which the logging component and the aliyunlogconfig CRD objects do not exist cannot be collected.

If you delete the logging component, the aliyunlogconfig CRD objects and Logtail are also deleted. Logs that are generated within the period of time during which the logging component does not exist cannot be collected.