FAQ about nodes and node pools

• How do I change the operating system for a node pool?

• Can I leave the Expected Nodes parameter empty when I create a node pool?

• What are the differences between node pools that are configured with the Expected Nodes parameter and those that are not configured with this parameter?

• How do I add free nodes to a node pool?

• How do I use preemptible instances in a node pool?

• Can I configure different ECS instance types in a node pool?

• How do I calculate the maximum number of pods on a node?

• How do I adjust the maximum number of pods that can be used when the number of pods reaches the upper limit?

• How do I modify the configurations of a node?

• How do I release a specific ECS instance?

• How do I update the container runtime of a worker node that does not belong to a node pool?

• What do I do if a timeout error occurs after I add an existing node?

• How do I change the hostname of a worker node in an ACK cluster?

• When a cluster that contains nodes in different zones fails, how does the cluster evict pods from nodes?

• What is the path of the kubelet in an ACK cluster? Can I use a custom path?

• Can I mount a data disk to a custom directory on a node in a node pool?

• How do I modify the maximum number of file handles?

How do I change the operating system for a node pool?

You can change the operating system based on your business requirements. For example, you can replace an operating system that has reached its end of life (EOL) with a supported operating system. Before you change the operating system, you must learn about the types of operating systems supported by ACK, the latest versions of operating system images, and the limits of some operating systems. For more information, see Release notes for OS images.

For more information about how to change the operating system and the usage notes, see Change the operating system.

Can I leave the Expected Nodes parameter empty when I create a node pool?

If the Scaling Mode parameter of the node pool is set to Manual, you must configure the Expected Nodes parameter for the node pool. This feature cannot be disabled.

For more information about how to remove or release a node, see Remove a node. For more information about how to add a node, see Add existing ECS instances to an ACK cluster. After you remove nodes from or add existing nodes to a cluster, the value of the Expected Nodes parameter is automatically set to the actual number of nodes after the modification.

What are the differences between node pools that are configured with the Expected Nodes parameter and those that are not configured with this parameter?

The Expected Nodes parameter specifies the number of nodes that you want to keep in a node pool. You can change the value of this parameter to modify the number of nodes in the node pool. This feature is disabled for existing node pools that are not configured with the Expected Nodes parameter.

Node pools that are configured with the Expected Nodes parameter and those that are not configured with this parameter have different reactions to operations such as removing nodes and releasing Elastic Compute Service (ECS) instances. The following table describes the details.

How do I add free nodes to a node pool?

Free nodes exist in clusters created before the node pool feature was released. If you no longer need free nodes, you can release the Elastic Compute Service (ECS) instances that are used to deploy the nodes. If you want to retain free nodes, we recommend that you add them to node pools. This way, you can manage the nodes in groups.

You can create and scale out a node pool, remove and add the free nodes to the node pool. For more information, see Add free nodes to a node pool.

How do I use preemptible instances in a node pool?

Can I configure different ECS instance types in a node pool?

Yes, you can. We recommend that you configure multiple vSwitches for a node pool, select multiple ECS instance types from multiple zones, or specify the instance types such as vCPUs and memory. This prevents nodes from being added to the node pool due to unavailable instance types or insufficient inventory. The ACK console automatically evaluates the scalability of the node pool. You can check the scalability of the node pool when you create the node pool or after you create the node pool.

For more information about instance types that are not supported by ACK, see ECS specification recommendations for ACK clusters.

How do I calculate the maximum number of pods on a node?

The maximum number of pods varies based on the network plug-in. For more information, see Maximum number of pods.

• Terway: Maximum number of pods on a node = Maximum number of pods in the node network + Number of pods in the host network.

• Flannel: The maximum number of pods on a node depends on the Number of Pods per Node parameter that you specified when you create the cluster.

You can view the maximum number of pods in the Pod (Allocated/Quota) column of the Nodes page in the ACK console.

You cannot modify the maximum number of pods on a node. When the number of pods in your cluster reaches the upper limit, we recommend that you scale out the node pool in the cluster to increase the number of pods in the cluster. For more information, see Increase the maximum number of pods in a cluster.

How do I adjust the maximum number of pods that can be used when the number of pods reaches the upper limit?

The maximum number of pods on a worker node varies based on the network plug-in and cannot be adjusted in most cases. In Terway mode, the maximum number of pods on a node depends on the number of elastic network interfaces (ENIs) provided by the Elastic Compute Service (ECS) instance. In Flannel mode, the maximum number of pods on a node depends on the cluster configurations that you specify when you create the cluster. The upper limit cannot be modified after the cluster is created. When the number of pods in your cluster reaches the upper limit, we recommend that you scale out the node pool in the cluster to increase the number of pods in the cluster.

For more information, see Increase the maximum number of pods in a cluster.

How do I modify the configurations of a node?

• Some node pool parameters, especially the parameters related to the node pool availability and network, cannot be modified after the node pool is created. This ensures the stability of your business. For example, the container runtime and the virtual private cloud (VPC) to which the node belongs can be modified after the node pool is created.

• For parameters that can be modified, after you modify a node pool, the modified configurations apply only to newly added nodes. In specific scenarios, such as when you update the ECS tags or labels and taints of existing nodes, the modified configurations also apply to existing nodes in the node pool.

For more information about which parameters can or cannot be modified and how they take effect, see Modify a node pool.

If you want to run a new node, you can create a node pool based on the configurations of the new node, set the nodes in the old node pool to the Unschedulable state, and then drain the old nodes. Wait until the service is full migration to the new node before releasing the old node. For more information, see Node draining and scheduling status.

How do I release a specific ECS instance?

You can release a specific ECS instance by removing the corresponding node. After an ECS instance is released, the expected number of nodes automatically changes to the actual number of nodes. You do not need to modify the expected number of nodes. You cannot release an ECS instance by modifying the expected number of nodes.

How do I update the container runtime of a worker node that does not belong to a node pool?

Free nodes exist in clusters created before the node pool feature was released. To update the container runtime of a node, you can add the node to a node pool for management.

Perform the following steps:

1. Create a node pool: If the cluster does not have a node pool, you can create a node pool that has the same configurations as the free nodes.

2. Remove a node: When you remove the worker node, the system sets the node to the Unschedulable state and drains the node. If the node fails to be drained, the system stops removing the node. If the node is drained, the system continues to remove the node from the cluster.

3. Add existing nodes: You can add the node to an existing node pool. Alternatively, you can create an empty node pool and add the node to the node pool. After the node is added to a node pool, the container runtime of the node automatically becomes the same as that of the node pool.

   Note

   Node pools are free of charge. However, you must pay for the cloud resources, such as ECS instances, used in node pools. For more information, see Cloud resource fee.

What do I do if a timeout error occurs after I add an existing node?

Check whether the network of the node and the network of the Classic Load Balancer (CLB) instance of the API server are connected. Check whether the security groups meet the requirement. For more information about the limits on security groups, see Limits on security groups. For more information about other network connectivity issues, see FAQ about network management.

How do I change the hostname of a worker node in an ACK cluster?

1. For more information about how to remove a node, see Remove a node.

2. For more information about how to add the removed nodes to the node pool, see Manually add nodes.

   After you add the node to the node pool, the node is renamed based on the new node naming rule of the node pool.

How do I manually update the kernel version of GPU-accelerated nodes in a cluster?

1. Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

2. Set the GPU-accelerated node that you want to manage to the Unschedulable state. In this example, the node cn-beijing.i-2ze19qyi8votgjz12345 is used.

   kubectl cordon cn-beijing.i-2ze19qyi8votgjz12345
   
   node/cn-beijing.i-2ze19qyi8votgjz12345 already cordoned

3. Migrate the pods on the GPU-accelerated node to other nodes.

   kubectl drain cn-beijing.i-2ze19qyi8votgjz12345 --grace-period=120 --ignore-daemonsets=true
   
   node/cn-beijing.i-2ze19qyi8votgjz12345 cordoned
   WARNING: Ignoring DaemonSet-managed pods: flexvolume-9scb4, kube-flannel-ds-r2qmh, kube-proxy-worker-l62sf, logtail-ds-f9vbg
   pod/nginx-ingress-controller-78d847fb96-5fkkw evicted

4. Uninstall the existing nvidia-driver.

   Note

   In this example, the uninstalled driver version is 384.111. If your driver version is not 384.111, download the installation package of your driver from the official NVIDIA website and update the driver to 384.111 first.

   1. Log on to the GPU-accelerated node and run the nvidia-smi command to check the driver version.

      sudo nvidia-smi -a | grep 'Driver Version'
      Driver Version                      : 384.111

   2. Download the driver installation package.

      sudo cd /tmp/
      sudo curl -O https://cn.download.nvidia.cn/tesla/384.111/NVIDIA-Linux-x86_64-384.111.run

      Note

      The installation package is required for uninstalling the NVIDIA driver.

   3. Uninstall the driver.

      sudo chmod u+x NVIDIA-Linux-x86_64-384.111.run
      sudo sh ./NVIDIA-Linux-x86_64-384.111.run --uninstall -a -s -q

5. Update the kernel.

   Update the kernel version based on your business requirements.

6. Restart the GPU-accelerated node.

   sudo reboot

7. Log on to the GPU node and run the following command to install the kernel-devel package.

   sudo yum install -y kernel-devel-$(uname -r)

8. Go to the official NVIDIA website to download the required driver and install it on the GPU-accelerated node. In this example, the driver version 410.79 is used.

   sudo cd /tmp/
   sudo curl -O https://cn.download.nvidia.cn/tesla/410.79/NVIDIA-Linux-x86_64-410.79.run
   sudo chmod u+x NVIDIA-Linux-x86_64-410.79.run
   sudo sh ./NVIDIA-Linux-x86_64-410.79.run -a -s -q
   
   warm up GPU
   sudo nvidia-smi -pm 1 || true
   sudo nvidia-smi -acp 0 || true
   sudo nvidia-smi --auto-boost-default=0 || true
   sudo nvidia-smi --auto-boost-permission=0 || true
   sudo nvidia-modprobe -u -c=0 -m || true

9. Make sure that the /etc/rc.d/rc.local file includes the following configurations. Otherwise, add the following configurations to the file.

   sudo nvidia-smi -pm 1 || true
   sudo nvidia-smi -acp 0 || true
   sudo nvidia-smi --auto-boost-default=0 || true
   sudo nvidia-smi --auto-boost-permission=0 || true
   sudo nvidia-modprobe -u -c=0 -m || true

10. Restart the kubelet and Docker.

    sudo service kubelet stop
    sudo service docker restart
    sudo service kubelet start

11. Set the GPU-accelerated node to schedulable.

    kubectl uncordon cn-beijing.i-2ze19qyi8votgjz12345
    
    node/cn-beijing.i-2ze19qyi8votgjz12345 already uncordoned

12. Run the following command in the nvidia-device-plugin container to check the version of the driver installed on the GPU-accelerated node.

    kubectl exec -n kube-system -t nvidia-device-plugin-cn-beijing.i-2ze19qyi8votgjz12345 nvidia-smi
    Thu Jan 17 00:33:27 2019
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: N/A      |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |===============================+======================+======================|
    |   0  Tesla P100-PCIE...  On   | 00000000:00:09.0 Off |                    0 |
    | N/A   27C    P0    28W / 250W |      0MiB / 16280MiB |      0%      Default |
    +-------------------------------+----------------------+----------------------+
    
    +-----------------------------------------------------------------------------+
    | Processes:                                                       GPU Memory |
    |  GPU       PID   Type   Process name                             Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+

    Note

    If no container is launched on the GPU-accelerated node after you run the docker ps command, see What do I do if no container is launched on a GPU-accelerated node?

What do I do if no container is launched on a GPU-accelerated node?

sudo service kubelet stop
Redirecting to /bin/systemctl stop kubelet.service
sudo service docker stop
Redirecting to /bin/systemctl stop docker.service
sudo service docker start
Redirecting to /bin/systemctl start docker.service
sudo service kubelet start
Redirecting to /bin/systemctl start kubelet.service

sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

sudo docker info | grep -i cgroup
Cgroup Driver: cgroupfs

To resolve the issue, perform the following steps:

1. Create a copy of /etc/docker/daemon.json. Then, run the following commands to update /etc/docker/daemon.json.

   sudo cat >/etc/docker/daemon.json <<-EOF
   {
       "default-runtime": "nvidia",
       "runtimes": {
           "nvidia": {
               "path": "/usr/bin/nvidia-container-runtime",
               "runtimeArgs": []
           }
       },
       "exec-opts": ["native.cgroupdriver=systemd"],
       "log-driver": "json-file",
       "log-opts": {
           "max-size": "100m",
           "max-file": "10"
       },
       "oom-score-adjust": -1000,
       "storage-driver": "overlay2",
       "storage-opts":["overlay2.override_kernel_check=true"],
       "live-restore": true
   }
   EOF

2. Run the following commands to restart the Docker runtime and the kubelet:

   sudo service kubelet stop
   Redirecting to /bin/systemctl stop kubelet.service
   sudo service docker restart
   Redirecting to /bin/systemctl restart docker.service
   sudo service kubelet start
   Redirecting to /bin/systemctl start kubelet.service

3. Run the following command to check whether the cgroup driver is set to systemd.

   sudo docker info | grep -i cgroup
   Cgroup Driver: systemd

How do I migrate multiple pods to other nodes when a node fails?

You can set the faulty node to unschedulable and drain the node. This way, ACK migrates application pods from the faulty node to other nodes.

1. Log on to the ACK console. On the Nodes page, choose More > Drain in the Actions column. ACK sets the node to unschedulable and migrates applications from the node to other nodes.

2. Troubleshoot node exceptions. For more information, see Troubleshoot node exceptions.

   To contact technical support, submit a ticket.

When a cluster that contains nodes in different zones fails, how does the cluster evict pods from nodes?

In most scenarios, when a node fails, the node controller evicts pods from the node. The value of --node-eviction-rate is 0.1 pod per second, which indicates that pods are evicted from at most one node every 10 seconds.

When an ACK cluster that contains nodes residing in multiple zones fails, the node controller determines how to evict pods based on the zone status and the cluster size.

A zone can be in one of the following states:

• FullDisruption: No healthy node resides in the zone and at least one unhealthy node exists.

• PartialDisruption: At least two unhealthy nodes exist in the zone, and the ratio of unhealthy nodes (unhealthy nodes/(unhealthy nodes + healthy nodes) is greater than 0.55.

• Normal: All nodes in the zone are healthy.

A cluster can be classified into two types based on the cluster size:

• Large cluster: The cluster contains more than 50 nodes.

• Small cluster: The cluster contains 50 or fewer nodes.

The eviction rate of the node controller is calculated based on the following rules:

• If all zones are in the FullDisruption state, the eviction feature is disabled for all zones.

• If not all zones are in the FullDisruption state, the eviction rate is determined in the following ways.

  • If a zone is in the FullDisruption state, the eviction rate is set to the default value (0.1), regardless of the cluster size.

  • If a zone is in the PartialDisruption state, the eviction rate depends on the cluster size. In a large cluster, the eviction rate of the zone is 0.01. In a small cluster, the eviction rate of the zone is 0, which indicates that no pod is evicted.

  • If a zone is in the Normal state, the eviction rate is set to the default value (0.1), regardless of the cluster size.

For more information, see Rate limits on eviction.

What is the path of the kubelet in an ACK cluster? Can I use a custom path?

ACK does not allow you to customize the path of the kubelet. The default path of the kubelet is /var/lib/kubelet. Do not change the path.

Can I mount a data disk to a custom directory on a node in a node pool?

This feature is in canary release. To use this feature, submit a ticket. After you enable this feature, you can automatically format the data disks attached to a node pool and mount the data disks to specified custom directories on the operating system. When you use this feature, the following limits apply:

• Do not mount data disks to the following directories on the operating system:

  • /

  • /etc

  • /var/run

  • /run

  • /boot

• Do not mount data disks to the following directories or their subdirectories used by the system and the container runtime:

  • /usr

  • /bin

  • /sbin

  • /lib

  • /lib64

  • /ostree

  • /sysroot

  • /proc

  • /sys

  • /dev

  • /var/lib/kubelet

  • /var/lib/docker

  • /var/lib/containerd

  • /var/lib/container

• Multiple data disks cannot be mounted to the same directory.

• The mount directory must be an absolute path that starts with a forward slash (/).

• The mount directory cannot contain carriage returns (the \r escape character in C) or line feeds (the \n escape character in C), and cannot end with a backslash (\).

How do I modify the maximum number of file handles?

The maximum number of file handles equals the maximum number of files that can be opened. Alibaba Cloud Linux and CentOS have two file handle limits:

• System level: The maximum number of files that can be opened simultaneously by all user processes.

• User level: The maximum number of files that can be opened by a single user process.

In a container environment, there is an additional file handle limit, which limits the maximum number of file handles for a single process within a container.

Modify the maximum number of system-level file handles for a node

For more information, see Customize the OS parameters of a node pool.

Modify the maximum number of file handles for a single process on a node

1. Log on to the node and view the /etc/security/limits.conf file.

   cat /etc/security/limits.conf

   Use the following parameters to configure the maximum number of file handles for a single process on a node:

   ...
   root soft nofile 65535
   root hard nofile 65535
   * soft nofile 65535
   * hard nofile 65535

2. Run the sed command to modify the maximum number of file handles. We recommend that you set the maximum number of file handles to 65535.

   sed -i "s/nofile.[0-9]*$/nofile 65535/g" /etc/security/limits.conf

3. Log on to the node again and run the following command to check whether the modification takes effect:

   If the returned value is the same as the modified value, the modification takes effect.

   # ulimit -n
   65535

Modify the maximum number of file handles for a container

1. Log on to the node and run the following command to view the configuration file:

   • Nodes that use containerd: cat /etc/systemd/system/containerd.service

   • Nodes that use Docker: cat /etc/systemd/system/docker.service

   Configure the maximum number of file handles for a single process in a container by using the following parameters:

   ...
   LimitNOFILE=1048576 ****** Maximum number of file handles for a single process
   LimitNPROC=1048576 ****** Maximum number of processes
   ...

2. Run the following command to modify the value of the corresponding parameter. We recommend that you set the maximum number of file handles to 1048576.

   • Nodes that use containerd:

      sed -i "s/LimitNOFILE=[0-9a-Z]*$/LimitNOFILE=65536/g" /etc/systemd/system/containerd.service;sed -i "s/LimitNPROC=[0-9a-Z]*$/LimitNPROC=65537/g" /etc/systemd/system/containerd.service && systemctl daemon-reload && systemctl restart containerd

   • Nodes that use Docker:

     sed -i "s/LimitNOFILE=[0-9a-Z]*$/LimitNOFILE=1048576/g" /etc/systemd/system/docker.service;sed -i "s/LimitNPROC=[0-9a-Z]*$/LimitNPROC=1048576/g" /etc/systemd/system/docker.service && systemctl daemon-reload && systemctl restart docker

3. Run the following command to view the maximum number of file handles for a single process in a container:

   If the returned value is the same as the modified value, the modification takes effect.

   • Nodes that use containerd:

     # cat /proc/`pidof containerd`/limits | grep files
     Max open files            1048576              1048576              files

   • Nodes that use Docker:

     # cat /proc/`pidof dockerd`/limits | grep files
     Max open files            1048576              1048576              files