All Products
Search
Document Center

Container Service for Kubernetes:Cluster check items and suggestions on how to fix cluster issues

Last Updated:Sep 01, 2023

The Container Service for Kubernetes (ACK) console supports cluster update check, cluster migration check, and component check. This topic describes cluster check items and provides suggestions on how to fix cluster issues.

Table of contents

Cluster check items

Cluster update check

Kubernetes is complex. In new Kubernetes versions, changes may be made to the runtime, certain Kubernetes APIs may be deprecated, and new features may be introduced. Due to these updates, high risks exist when you update clusters. To ensure that you can smoothly update your cluster, ACK provides the cluster update check feature. A precheck is automatically triggered before a cluster is updated. The cluster is updated only if the cluster passes the precheck.

Cluster update check inspects the following types of resources:

  • Cluster resources: cloud resources related to the ACK Serverless cluster, such as Server Load Balancer (SLB) instances and virtual private clouds (VPCs).

  • Cluster components: the configurations of the ACK Serverless cluster, components, and applications. For example, the system checks whether the component versions meet the requirement and when applications use deprecated APIs.

The check items vary based on the type, runtime, and version of the cluster. The check items in the following table are for reference only. The actual check items in the console shall prevail.

Type

Check item

Description

Cluster resources

APIServer SLB

Checks whether the SLB instance exists.

Checks whether the status of the SLB instance is normal.

Checks whether the configurations of the SLB listeners are valid, including the listener ports and protocol.

Checks whether the configurations of the SLB backend server groups are valid.

Checks whether the configuration of SLB access control is valid. If no access control is configured, this check item displays Normal.

VPC

Checks whether the VPC exists.

Checks whether the status of the VPC is normal.

vSwitch

Checks whether the vSwitch exists.

Checks whether the status of the vSwitch is normal.

Checks whether the vSwitch can provide no less than two idle IP addresses.

Cluster components

Kube Proxy Master

Checks whether the component exists.

Kube Proxy Worker

Checks whether the component exists.

API Service

Checks whether unavailable API Services exist.

Cluster components

Checks whether the version of Terway meets the requirement.

Checks whether the version of CoreDNS meets the requirement.

Checks whether the version of the cloud controller manager meets the requirement.

Checks whether the version of the NGINX Ingress controller meets the requirement.

Checks whether the version of the metric server meets the requirement.

Deprecated APIs

Checks whether the cluster uses deprecated APIs.

Cluster configurations

iptables configurations

Checks whether the iptables configurations are valid.

Operating systems

Checks whether the operating system of the node can be updated.

yum

Checks whether Yum is normal.

Kubelet

Checks whether the kubelet configuration meets the requirement.

Container runtime

Checks whether the Docker runtime or Containerd runtime is normal.

Manifest configuration

Checks whether the manifest file meets the requirement.

Cluster migration check

A precheck is automatically triggered before a cluster is migrated. The cluster is migrated only if the cluster passes the precheck. Cluster migration check is suitable for the following scenarios:

Migrate from an ACK Serverless Basic cluster to an ACK Serverless Pro cluster.

Cluster migration check inspects the following types of resources:

  • Cluster resources: cloud resources related to the ACK Serverless cluster, such as SLB instances and VPCs.

  • Cluster components: the configurations of the components in the ACK Serverless cluster, such as whether unavailable API Services exist.

The cluster migration check items vary based on the type, runtime, and version of the cluster. The check items in the following table are for reference only. The actual check items in the console shall prevail.

Type

Check item

Description

Cluster resources

APIServer SLB

Checks whether the SLB instance exists.

Checks whether the status of the SLB instance is normal.

Checks whether the configurations of the SLB listeners are valid, including the listener ports and protocol.

Checks whether the configurations of the SLB backend server groups are valid.

Checks whether the configuration of SLB access control is normal. If no access control is configured, this check item displays Normal.

VPC

Checks whether the VPC exists.

Checks whether the status of the VPC is normal.

vSwitch

Checks whether the vSwitch exists.

Checks whether the status of the vSwitch is normal.

Checks whether the vSwitch can provide no less than two idle IP addresses.

Cluster components

Kube Proxy Master

Checks whether the component exists.

Kube Proxy Worker

Checks whether the component exists.

API Service

Checks whether unavailable API Services exist.

Component check

Component check is suitable for component update scenarios. A precheck is automatically triggered before a component is updated. The component is updated only if the cluster passes the precheck.

The component check items vary based on the type, runtime, and version of the cluster. The check items in the following table are for reference only. The actual check items in the console shall prevail.

Type

Check item

Description

cloud-controller-manager

Addon_CCM

Checks whether the update causes SLB changes.

Component_Block_Version

Checks whether the cloud controller manager can be updated.

csi-plugin

DaemonSet_Annotation

Checks whether the annotations of the DaemonSet meet the requirement.

Csi_Driver_Attributes

Checks whether the CSI driver attribute meets the requirement.

csi-provisioner

Stateful_Set_Exist

Checks whether the resource is a StatefulSet.

Deployment_Annotation

Checks whether the annotations of the Deployment meet the requirement.

Storage_Class_Attributes

Checks whether the StorageClass attribute meets the requirement.

nginx-ingress-controller

Deployment_Healthy

Checks whether the NGINX Ingress Deployment is healthy.

Deployment_Not_Under_HPA

Checks whether a horizontal pod autoscaler (HPA) is configured for the Deployment.

Deployment_Not_Modified

Checks whether the Deployment is changed.

Nginx_Ingress_Pod_Error_Log

Checks whether NGINX error logs are generated.

LoadBalancer_Service_Healthy

Checks whether the NGINX Services are healthy.

Nginx_Ingress_Configuration

Checks whether incompatible configurations exist in Ingresses.

aliyun-acr-credential-helper

RamRole_Exist

Checks whether the component is assigned the AliyunCSManagedAcrRole role.

ack-cost-exporter

RamRole_Exist

Checks whether the component is assigned the AliyunCSManagedCostRole role.

Suggestions on how to fix cluster issues

Issue

Suggestion

Role Aliyun_ARMS_CMonitor_Role missing

Grant the cluster permissions on Prometheus Service. For more information about how to manually grant permissions on Application Real-Time Monitoring Service (ARMS) and Tracing Analysis, see Enable Kubernetes Monitoring for a Kubernetes cluster.

Outdated component version

Update the component. For more information, see Manage components.

Unavailable API Services

  1. Run the following command to check for unavailable API Services:

    kubectl -n kube-system get apiservices |grep -i false
  2. Confirm the purpose of the unavailable API Service. If the API Service is no longer needed, run the following command to delete the Service.

    Important

    Proceed with caution because cluster exceptions may occur if you delete API Services that are still in use. If you cannot determine the purpose of the API Service, submit a ticket.

    kubectl -n kube-system delete apiservices ${your-abnormal-apiservice-name}

Use of deprecated APIs

Identify the resource that uses the deprecated APIs and take actions accordingly. For more information, see Deprecated APIs.

Deprecated APIs

If your cluster runs Kubernetes 1.20 or later, the precheck checks whether deprecated APIs are used in your cluster. You can view the deprecated APIs that are used by the cluster in the check report.

For example, before you update your cluster from Kubernetes 1.20 to Kubernetes 1.22, the system checks whether deprecated APIs are used in your cluster by scanning the audit logs that were generated the previous day.

  • The precheck result is for reference only. You can proceed with the update even if your cluster runs Kubernetes 1.20 and uses deprecated APIs.

  • If you continue to use the deprecated APIs in Kubernetes 1.22, potential security risks may exist.

The following table describes the types of deprecated APIs. Before you update a cluster that uses deprecated APIs, we recommend that you refer to the Type column of the following table and perform operations that correspond to the type of deprecated API used by the cluster.

Type

Suggestion

Example

core

Key Kubernetes components: ACK automatically updates key Kubernetes components. You do not need to update the components. Information about the components is not displayed on the precheck page.

apiserver, scheduler, and kube-controller-manager

ack

ACK components: ACK components require manual update. You can update ACK components based on the instructions on the Add-ons page of the ACK console.

Note
  • You can log on to the ACK console and choose Operations > Add-ons to update components. The next day after the components are updated, the deprecated APIs are not displayed.

  • The CoreDNS component may use deprecated APIs in clusters that run Kubernetes 1.24 and later. If the check report includes CoreDNS, refer to Why does CoreDNS use deprecated APIs?

  • Deprecated APIs in the precheck result are for reference only. You can proceed with the update even if your cluster uses deprecated APIs. After your cluster is updated, your cluster uses new APIs. To avoid security risks, we recommend that you do not use deprecated APIs after you update your cluster.

metrics-server, nginx-ingress-controller, and coredns

opensource

Open source components: ACK lists some open source components in the console. You can decide whether to update the components. Other open source components may be classified into the unknown type.

Note

Deprecated APIs in the precheck result are for reference only. You can proceed with the update even if your cluster uses deprecated APIs. Update the components based on your business requirements.

rancher and elasticsearch-operator

unknown

Unknown sources: Deprecated APIs that do not belong to the previous types are classified into the unknown type. ACK lists the unknown components in the console. You can decide whether to update the components. The components require manual update.

Note

Deprecated APIs in the precheck result are for reference only. You can proceed with the update even if your cluster uses deprecated APIs. Update the components based on your business requirements.

kubectl, agent, Go-http-client, and okhttp