Container Service for Kubernetes：Cluster check items and suggestions on how to fix cluster issues

Type

Check item

Description

Cluster resources

APIServer SLB

Checks whether the SLB instance exists.

Checks whether the status of the SLB instance is normal.

Checks whether the configurations of the SLB listeners are valid, including the listener ports and protocol.

Checks whether the configurations of the SLB backend server groups are valid.

Checks whether the configuration of SLB access control is valid. If no access control is configured, this check item displays Normal.

VPC

Checks whether the VPC exists.

Checks whether the status of the VPC is normal.

vSwitch

Checks whether the vSwitch exists.

Checks whether the status of the vSwitch is normal.

Checks whether the vSwitch can provide no less than two idle IP addresses.

ECS

Checks whether the ECS instance exists.

Checks whether the status of the ECS instance is normal.

Checks whether the security group of the ECS instance is normal.

Checks whether the ECS instance has expired.

Checks whether the instance type of the ECS instance meets the requirement.

Checks whether the status of the Cloud Assistant client is normal.

Cluster components

Kube Proxy Master

Checks whether the component exists.

Kube Proxy Worker

Checks whether the component exists.

API Service

Checks whether unavailable API Services exist.

Cluster instances

Checks whether the number of master instances in the cluster is three or five.

Cluster components

Checks whether the version of Terway meets the requirement.

Checks whether the version of CoreDNS meets the requirement.

Checks whether the version of the cloud controller manager meets the requirement.

Checks whether the version of the NGINX Ingress controller meets the requirement.

Checks whether the version of ACK Virtual Node meets the requirement.

Checks whether the version of the metric server meets the requirement.

Nodes

Checks whether the node IP address exists.

Checks whether the node is schedulable.

Checks whether the node is ready.

Checks whether the operating system of the node can be updated.

Checks whether the number of available pods on the node is greater than two.

Deprecated APIs

Checks whether the cluster uses deprecated APIs.

Cluster configurations

iptables configurations

Checks whether the iptables configurations are valid.

Operating systems

Checks whether the operating system can be updated.

yum

Checks whether Yum is normal.

Disks

Checks whether the file system of the node is normal.

Checks whether the free disk space of the node exceeds 5% of the total disk space.

Swap

Checks whether the node has Swap enabled.

NTP

Checks whether the NTP of the node is normal.

Systemd

Checks whether the Systemd version of the node is later than systemd-219-67.

Kubelet

Checks whether the kubelet configuration meets the requirement.

Container runtime

Checks whether the Docker runtime or Containerd runtime is normal.

Kernel configuration

Checks whether the kernel configuration of the node is normal.

Manifest configuration

Checks whether the manifest file meets the requirement.

Type

Check item

Description

Cluster resources

APIServer SLB

Checks whether the SLB instance exists.

Checks whether the status of the SLB instance is normal.

Checks whether the configurations of the SLB listeners are valid, including the listener ports and protocol.

Checks whether the configurations of the SLB backend server groups are valid.

Checks whether the configuration of SLB access control is normal. If no access control is configured, this check item displays Normal.

VPC

Checks whether the VPC exists.

Checks whether the status of the VPC is normal.

vSwitch

Checks whether the vSwitch exists.

Checks whether the status of the vSwitch is normal.

Checks whether the vSwitch can provide no less than two idle IP addresses.

ECS

Checks whether the ECS instance exists.

Checks whether the status of the ECS instance is normal.

Checks whether the security group of the ECS instance is normal.

Checks whether the status of the Cloud Assistant client is normal.

Cluster components

Kube Proxy Master

Checks whether the component exists.

Kube Proxy Worker

Checks whether the component exists.

API Service

Checks whether unavailable API Services exist.

Cluster instances

Checks whether the number of master instances in the cluster is three or five.

Nodes

Checks whether the node IP address exists.

Checks whether the node is schedulable.

Checks whether the node is ready.

Checks whether the operating system of the node can be updated.

Checks whether the number of available pods on the node is greater than two.

Cluster configurations

Operating systems

Checks whether the operating system can be updated.

yum

Checks whether Yum is normal.

Use of components

Cloud Controller Manager

Checks whether the cloud controller manager is normal.

Type

Check item

Description

cloud-controller-manager

Addon_CCM

Checks whether the update causes SLB changes.

Component_Block_Version

Checks whether the cloud controller manager can be updated.

csi-plugin

DaemonSet_Annotation

Checks whether the annotations of the DaemonSet meet the requirement.

Csi_Driver_Attributes

Checks whether the CSI driver attribute meets the requirement.

Node_Status_Ready

Checks whether the node is ready.

csi-provisioner

Stateful_Set_Exist

Checks whether the resource is a StatefulSet.

Deployment_Annotation

Checks whether the annotations of the Deployment meet the requirement.

Storage_Class_Attributes

Checks whether the StorageClass attribute meets the requirement.

Csi_Provisioner_Node_Count

Checks whether the number of ready nodes is equal to or greater than two.

terway-eniip

Systemd

Checks whether the Systemd version of the node is later than systemd-219-67.

nginx-ingress-controller

Deployment_Healthy

Checks whether the NGINX Ingress Deployment is healthy.

Deployment_Not_Under_HPA

Checks whether a horizontal pod autoscaler (HPA) is configured for the Deployment.

Deployment_Not_Modified

Checks whether the Deployment is changed.

Nginx_Ingress_Pod_Error_Log

Checks whether NGINX error logs are generated.

LoadBalancer_Service_Healthy

Checks whether the NGINX Services are healthy.

Nginx_Ingress_Configuration

Checks whether incompatible configurations exist in Ingresses.

aliyun-acr-credential-helper

RamRole_Exist

Checks whether the component is assigned the AliyunCSManagedAcrRole role.

ack-cost-exporter

RamRole_Exist

Checks whether the component is assigned the AliyunCSManagedCostRole role.

Type

Check item

Description

Cluster resources

APIServer SLB

Checks whether the SLB instance exists.

Checks whether the status of the SLB instance is normal.

Checks whether the configurations of the SLB listeners are valid, including the listener ports and protocol.

Checks whether the configurations of the SLB backend server groups are valid.

Checks whether the configuration of SLB access control is valid. If no access control is configured, this check item displays Normal.

VPC

Checks whether the VPC exists.

Checks whether the status of the VPC is normal.

vSwitch

Checks whether the vSwitch exists.

Checks whether the status of the vSwitch is normal.

Checks whether the vSwitch can provide no less than two idle IP addresses.

Cluster components

API Service

Checks whether unavailable API Services exist.

Cluster instances

Checks whether the number of master instances in the cluster is three or five.

Nodes

Checks whether the node is ready.

Checks whether the number of available pods on the node is greater than two.

HostPath

Checks whether pods that use hostPath exist on the node.

Cluster configurations

iptables configurations

Checks whether the iptables configurations are valid.

Operating systems

Checks whether the operating system can be updated.

yum

Checks whether Yum is normal.

Disks

Checks whether the file system of the node is normal.

Free disk space on nodes

Checks whether the free disk space of the node exceeds 5% of the total disk space.

Swap

Checks whether the node has Swap enabled.

NTP

Checks whether the NTP of the node is normal.

Systemd

Checks whether the Systemd version of the node is later than systemd-219-67.

Kubelet

Checks whether the kubelet configuration meets the requirement.

Container runtime

Checks whether the Docker runtime or Containerd runtime is normal.

Kernel configuration

Checks whether the kernel configuration of the node is normal.

Manifest configuration

Checks whether the manifest file meets the requirement.

Issue

Suggestion

Role Aliyun_ARMS_CMonitor_Role missing

Grant the cluster permissions on Managed Service for Prometheus. For more information about how to manually grant permissions on Application Real-Time Monitoring Service (ARMS) and Tracing Analysis, see Enable Kubernetes Monitoring for a Kubernetes cluster.

Outdated Systemd version

Update Systemd.

Outdated component version

Update the component. For more information, see Manage components.

Yum timeout

Run the following command to check whether Yum times out. The default timeout period is 10 seconds.

time if type yum&>/dev/null; then yum list yum; fi

Unavailable API Services

1. Run the following command to check for unavailable API Services:

   kubectl -n kube-system get apiservices |grep -i false

2. Confirm the purpose of the unavailable API service. If the API Service is no longer needed, run the following command to delete the Service.

   Important

   Proceed with caution because cluster exceptions may occur if you delete API Services that are still in use. If you cannot determine the purpose of the API Service, submit a ticket.

   kubectl -n kube-system delete apiservices ${your-abnormal-apiservice-name}

Pods using hostPath

When the system updates a node by replacing its system disk, if the pods on the node use hostPath to mount the container directory to the host, data lost may occur. You need to check the directory that is mounted by the pods. If hostPath is not used or no risk of data loss exists, you can proceed with the update. The check result is for reference only.

Use of deprecated APIs

Identify the resource that uses the deprecated APIs and take actions accordingly. For more information, see Deprecated APIs.

Type

Suggestion

Example

core

Key Kubernetes components: ACK automatically updates key Kubernetes components. You do not need to update the components. Information about the components is not displayed on the precheck page.

apiserver, scheduler, and kube-controller-manager

ack

ACK components: ACK components require manual update. You can update ACK components based on the instructions on the Add-ons page of the ACK console.

metrics-server, nginx-ingress-controller, and coredns

opensource

Open source components: Some open source components are listed in the ACK console. You can decide whether to update the components. These components can only be manually updated. Other open source components may be classified into the unknown type.

rancher and elasticsearch-operator

unknown

Unknown sources: Deprecated APIs that do not belong to the preceding types are considered unknown resources and listed in the ACK console. You can decide whether to update the components. These components can only be manually updated.

kubectl, agent, Go-http-client, and okhttp