Pod troubleshooting

Error message

Description

Suggested solution

no nodes available to schedule pods.

The cluster does not have nodes available for pod scheduling.

1. Check whether the nodes in the cluster are in the NotReady state. If a node is in the NotReady state, troubleshoot and repair the node.

2. Check whether the node selector, node affinity rules, or taint tolerations are configured for the pod. If no affinity rules are configured for the pod, increase the number of nodes in the node pool.

• 0/x nodes are available: x Insufficient cpu.

• 0/x nodes are available: x Insufficient memory.

The cluster does not have nodes that can fulfill the CPU or memory request of the pod.

On the Nodes page, view the pod, CPU, and memory usage and check the cluster resource utilization.

When the CPU or memory resources in the cluster are exhausted, use the following methods:

• Delete unnecessary pods. For more information, see Manage pods.

• Modify the resource configuration of the pod. For more information, see Modify the upper and lower limits of CPU and memory resources for a pod. Enable resource profiling to obtain the suggested resource requests and limits for containers.

• Add nodes to the cluster based on the affinity rules of the pod. For more information, see Create and manage a node pool.

• Upgrade nodes. For more information, see Upgrade the configurations of a worker node.

• x node(s) didn't match node selector.

• x node(s) didn't match pod affinity/anti-affinity.

The cluster does not have nodes that match the node affinity rules (nodeSelector) or affinity and anti-affinity rules (podAffinity and podAnitiAffinity) of the pod.

1. Check and modify the node affinity rules of the pod, including node labels, nodeSelector, nodeAffinity, taints, and tolerations.

2. Check and modify the pod affinity rules of the pod. Assess whether nodes can match the pod affinity rules. If podAffinity is configured, check for pods that match podAffinity on the desired node. If podAntiAffinity is configured, check for pods that match podAntiAffinity on the desired node.

0/x nodes are available: x node(s) had volume node affinity conflict.

The volume used by the pod has encounters a node affinity conflict because disk volumes cannot be mounted across zones. Consequently, the pod failed to be scheduled.

• If the pod uses a statically provisioned persistent volume (PV) and you want to schedule the pod to a node in the zone of the PV, you must configure a node affinity rule for the pod.

• If the pod uses a dynamically provisioned PV, you must set the volumeBindingMode in the StorageClass of the disk volume to WaitForFirstConsumer. This way, the volume is created only after the pod is scheduled to a node. This ensures that the volume resides in the zone of the node that hosts the pod.

InvalidInstanceType.NotSupportDiskCategory

The disk type is not supported by Elastic Compute Service (ECS) instances.

Refer to Overview of instance families and check the disk types supported by ECS instances. Mount a disk that is supported by ECS instances to the pod.

0/x nodes are available: x node(s) had taints that the pod didn't tolerate.

The pod failed to be scheduled to the desired node because the node has a taint.

• If the taint is manually added by you, delete the taint. If the taint must be retained, you can add a toleration to the pod. For more information, see Taints and Tolerations and Manage taints.

• If the taint is added by the system, refer to the following solution and wait for the scheduler to reschedule the pod.

  View taints added by the system

  • node.kubernetes.io/not-ready: The node is in the NotReady state.

  • node.kubernetes.io/unreachable: The node controller failed to access the node. The value of the Ready field of the node is Unknown.

  • node.kubernetes.io/memory-pressure: The node does not have sufficient memory resources.

  • node.kubernetes.io/disk-pressure: The node does not have sufficient disk space.

  • node.kubernetes.io/pid-pressure: The node does not have sufficient process IDs (PIDs).

  • node.kubernetes.io/network-unavailable: The network of the node is unavailable.

  • node.kubernetes.io/unschedulable: The node is in the Unschedulable state.

0/x nodes are available: x Insufficient ephemeral-storage.

The node does not have sufficient ephemeral storage space.

1. Check whether the pod has a limit on ephemeral volumes, which is specified in spec.containers.resources.request.ephemeral-storage of the pod YAML file. If the value exceeds the ephemeral storage capacity of the node, the pod failed to be scheduled.

2. Run the kubectl describe node | grep -A10 Capacity command to view the total ephemeral storage space of all nodes. If the storage space does not meet your requirement, expand the disks of the nodes or add nodes.

0/x nodes are available: pod has unbound immediate PersistentVolumeClaims.

The persistent volume claim (PVC) failed to be bound to the pod.

Check whether the PVC or PV specified for the pod is created. You can run the kubectl describe pvc <pvc-name> or kubectl describe pv <pv-name> command to view the events of the PVC or PV. For more information, see Why does the system generate the "0/x nodes are available: x pod has unbound immediate PersistentVolumeClaims" event for a pod?

Error message

Description

Suggested solution

The pod remains in the Init:N/M state

The pod contains M Init containers. N containers have been started. M-N Init containers have not started.

1. Run the kubectl describe pod -n <ns> <pod name> command to view the events of the pod. Make sure that no anomalies occur in the Init containers that have not started.

2. Run the kubectl logs -n <ns> <pod name> -c <container name> command to view the logs of the Init containers that have not started in the pod and troubleshoot the error based on the log data.

3. Check whether the Init container configuration in the pod configuration contains errors, such as the health check configuration.

For more information about init containers, see Debug init containers.

The pod remains in the Init:Error state

Init containers in the pod failed to start.

The pod remains in the Init:CrashLoopBackOff state

Init containers in the pod failed to start and repetitively restart.

Error message

Description

Suggested solution

failed to allocate for range 0: no IP addresses available in range set: xx.xxx.xx.xx-xx.xx.xx.xx

This error is expected due to the design of the Flannel network plug-in.

Update Flannel to v0.15.1.11-7e95fe23-aliyun or later. For more information, see Flannel.

If the cluster runs a Kubernetes version earlier than 1.20 and a pod repetitively restarts or pods created by a CronJob complete the tasks and exit after a short period of time, an IP leak may occur.

Update the Kubernetes version of the cluster to 1.20 or later. We recommend that you update to the latest version. For more information, see Manually update ACK clusters.

containerd and runC have defects.

For more information, see What do I do if a pod fails to be started and the "no IP addresses available in range" error message appears?

error parse config, can't found dev by mac 00:16:3e:01:c2:e8: not found

The internal database status maintained by Terway and used to track and manage elastic network interfaces (ENIs) on the node of the pod is inconsistent with the actual network device configuration. Consequently, ENIs failed to be allocated.

1. ENIs are asynchronously loaded. When you configure the CNI, the system may still be loading ENIs. This can cause the CNI to retry ENI allocation. This does not affect the final ENI allocation result. You can determine whether the operation is successful based on the final state of the pod.

2. If the pod failed to be created for a long period of time and the preceding error is thrown, the driver failed to be loaded during ENI attaching because of insufficient high-level memory. To address this issue, restart the corresponding ECS instance. For more information, see Restart an instance.

• cmdAdd: error alloc ip rpc error: code = DeadlineExceeded desc = context deadline exceeded

• cmdAdd: error alloc ip rpc error: code = Unknown desc = error wait pod eni info, timed out waiting for the condition

Terway failed to request an IP address from the vSwitch.

1. View the container log of the Terway pod on the node that hosts the pod and check the ENI allocation process.

2. Run the kubectl logs -n kube-system <terwayPodName > -c terway | grep <podName> command to view the ENI information of the Terway pod. Obtain the request ID of the operation that is performed to request an IP address and the error message returned by the API.

3. Locate the cause based on the request ID and error message.

Error message

Description

Suggested solution

The log displays exit(0).

1. Log on to the node where the abnormal workload is deployed.

2. Run the docker ps -a | grep $podName command. If no persistent process exists in the pod, exit (0) is displayed.

The event information displays Liveness probe failed: Get http….

A health check failure occurred.

Check the liveness probe policy of the pod. Make sure that the liveness probing result can indicate the actual status of the applications running in the containers of the pod.

The pod log displays no left space.

The disk space is insufficient.

• Refer to Step 1: Resize a disk to extend its capacity and expand the disk.

• Delete unused images to free disk space and configure imageGCHighThresholdPercent to adjust the threshold for triggering image garbage collection on the node.

The pod failed to start and no event is generated

The resource limits of the pod are lower than the resources requested by the pod. Consequently, containers in the pod failed to start.

Check the resource configuration of the pod. You can enable resource profiling to obtain suggested resource requests and resource limits.

The pod log displays Address already in use.

Containers in the pod encounter a port conflict.

1. Check whether hostNetwork: true is configured for the pod. If yes, containers in the pod share ENIs and ports with the host. If you do not want to use the host network, specify hostNetwork: false.

2. If hostNetwork: true is configured, configure pod anti-affinity rules to ensure that pods of the same replica set are scheduled to different nodes.

3. Make sure that pods using the same port are not scheduled to the same node.

The pod log displays container init caused \"setenv: invalid argument\"": unknown.

The workload is mounted with a Secret. The key value of the Secret is not encoded by using Base64.

• The key values of Secrets created in the console are automatically encoded by using Base64. For more information, see Manage Secrets.

• Use YAML to create a Secret and run the echo -n "xxxxx" | base64 command to encrypt the key value by using Base64.

A business issue exists.

Locate the cause based on the pod log.

OOM level

Description

Suggested solution

OS level

The kernel log (/var/log/messages) of the node that hosts the pod displays a killed process but no cgroup log is generated. This means that the OOM error is OS-level.

Possible causes are insufficient system global memory, insufficient node memory, or insufficient buddy system memory during memory fragmentation. For more information about the causes of memory shortage, see Possible causes. For more information about the solutions, see Solutions.

cgroup level

The kernel log (/var/log/messages) of the node that hosts the pod displays an error message similar to Task in /kubepods.slice/xxxxx killed as a result of limit of /kubepods.slice/xxxx. This means that the OOM error is cgroup-level.

If the process runs as normal, increase the memory limit of the pod accordingly. Make sure that the actual memory usage of the pod does not exceed 80% of the memory limit. For more information, see Modify the upper limit and lower limit of CPU and memory resources for pods. You can enable resource profiling to obtain suggested resource requests and resource limits for containers.

Possible cause

Description

Suggested solution

The node is abnormal and in the NotReady state.

A NotReady node is automatically removed after it recovers.

Finalizers are configured for the pod.

If finalizers are configured for the pod, Kubernetes performs the cleanup operations specified by the finalizers before deleting the pod. If no responses are returned for the cleanup operations, the pod remains in the Terminating state.

Run the kubectl get pod -n <ns> <pod name> -o yaml command to view the finalizer configuration of the pod and locate the cause.

The preStop hook of the pod is invalid.

If the preStop hook is configured for the pod, Kubernetes performs the operations specified by the preStop hook before terminating the pod. The pod is at the preStop stage and will enter the Terminating state.

Run the kubectl get pod -n <ns> <pod name> -o yaml command to view the preStop configuration of the pod and locate the cause.

The pod is configured with a graceful shutdown period.

If the pod is configured with a graceful shutdown period (terminationGracePeriodSeconds), the pod enters the Terminating state after receiving a termination command, such as kubectl delete pod <pod_name>. Kubernetes considers that the pod is terminated after the graceful shutdown period (terminationGracePeriodSeconds) ends or all containers in the pod exit before the graceful shutdown period ends.

After the containers gracefully exit, Kubernetes deletes the pod.

Containers do not respond.

After you initiate a request to terminate or delete the pod, Kubernetes sends the SIGTERM signal to the containers in the pod. If the containers do not respond to the SIGTERM signal, the pod may remain in the Terminating state.

1. Run the kubectl delete pod <pod-name> --grace-period=0 --force command to forcefully delete the pod.

2. Check the containerd or Docker log of the node that hosts the pod and locate the cause.

Possible cause

Description

Suggested solution

The node does not have sufficient resources, such as memory or disk space resources. Consequently, the kubelet evicts one or more pods on the node to reclaim resources.

The node may not have sufficient memory, disk space, or PIDs.

• Run the kubectl describe node <node name> | grep Taints command to query node taints. The following information is returned:

  • Insufficient memory: The node has the node.kubernetes.io/memory-pressure taint.

  • Insufficient disk space: The node has the node.kubernetes.io/disk-pressure taint.

  • Insufficient PIDs: The node has the node.kubernetes.io/pid-pressure taint.

• The pod may display the following states:

  • Evicted.

  • ContainerStatusUnknown, and the reason field in the pod YAML file indicates Evicted.

• Insufficient memory:

  • Adjust the resource configuration of the pod. For more information, see Manage pods.

  • Upgrade the node. For more information, see Upgrade the configurations of a worker node.

• Insufficient disk space:

  • Periodically delete the logs of application pods on the node.

  • Expand the disk of the node. For more information, see Step 1: Resize a disk to extend its capacity.

• Insufficient PIDs: Modify the resource settings of the pod. For more information, see Process ID Limits And Reservations.

An unexpected eviction occurs.

The node that hosts the pod has the NoExecute taint. Consequently, an unexpected eviction occurs.

Run the kubectl describe node <node name> | grep Taints command to check whether the node has the NoExecute taint. If yes, delete the taint.

The pod is not evicted as expected.

• --pod-eviction-timeout: the pod eviction timeout period. When the node downtime exceeds the specified timeout period, pods are evicted from the node. The default timeout period is 5 minutes.

• --node-eviction-rate: the number of pods evicted per second. The default is 0.1, which means that at least one pod is evicted from the node every 10 seconds.

• --secondary-node-eviction-rate: the secondary pod eviction rate. If an excessive number of nodes are down, pod eviction is tuned to the secondary rate. The default value is 0.01.

• --unhealthy-zone-threshold: the unhealthy zone threshold. The default value is 0.55. When the number of failed nodes exceeds 55% of the total number of nodes, the zone is considered unhealthy.

• --large-cluster-size-threshold: the large cluster threshold. The default value is 50. When the number of cluster nodes exceeds 50, the cluster is considered a large cluster.

In a small cluster which contains no more than 50 nodes, if more than 55% of the nodes are down, pod eviction is stopped. For more information, see Rate limits on eviction.

In a large cluster which contains more than 50 nodes, if the ratio of unhealthy nodes to total nodes exceeds the value of --unhealthy-zone-threshold (the default is 0.55), the eviction rate is set to the value of --secondary-node-eviction-rate. This parameter specifies the maximum proportion of pods evicted per minute. The default value is 0.01. For more information, see Rate limits on eviction.

The pod is still frequently scheduled to the original node after it is evicted.

Pods are evicted from a node based on the resource usage of the node. The pod scheduling rule depends on the allocated resources on the node. An evicted pod may be scheduled to the original node again.

Check whether the resource requests of the pod are properly configured based on the allocatable resources on the node. For more information, see Modify the upper and lower limits of CPU and memory resources for a pod. You can enable resource profiling to obtain suggested resource requests and resource limits for containers.