Container Service for Kubernetes:ACK console troubleshooting (cluster access exceptions)

An API server request exception occurs when you access a cluster resource, and the ErrorQueryClusterNamespace or APIServer.500 error code is returned

Issue

Cause

The load balancing configuration of the API server is invalid or the status of the API server is abnormal. As a result, the ACK management services fail to connect to the API server.

Solution

1. Log on to the ACK console. In the left-side navigation pane, click Cluster.

2. On the Clusters page, click the cluster that you want to manage.

3. On the cluster details page, click the Cluster Resources tab and then click the hyperlink to the right of API Server SLB to log on to the Server Load Balancer (SLB) console.

   • If the system displays the The specified SLB ID does not exist. message, the SLB instance for the API server is deleted or released. You cannot restore the SLB instance. To resolve this issue, you must recreate the cluster. For more information, see Create an ACK managed cluster.

   • If the system does not display the preceding message, proceed to the next step.

4. Check whether the Status column of the SLB instance displays Running.

   • If the SLB instance is not in the Running state, the SLB instance may be suspended or locked. A pay-as-you-go SLB instance is suspended if the SLB instance is overdue. A subscription SLB instance is locked if the subscription of the SLB instance expires. You must settle the overdue payment or renew the subscription, and then restart the SLB instance. For more information about overdue payments related to SLB instances, see Overdue payments.

   • If the SLB instance is in the Running state, proceed to the next step.

5. Check whether the SLB instance has a listener whose Frontend Protocol/Port and Backend Protocol/Port settings are set to TCP:6443, and whether the Status column of the listener displays Running.

   • If the preceding listener does not exist, the configuration of the listener for the API server is modified.

     • If the preceding listener exists but the listener is in the Stopped state, select the listener and click Start.

     • If the preceding listener does not exist, perform one of the following operations:

       • If your cluster is an ACK managed cluster, submit a ticket.

       • If your cluster is an ACK dedicated cluster, make sure that all master nodes are added to the default server group. Then, create a listener whose Frontend Protocol/Port and Backend Protocol/Port settings are set to TCP:6443, associate the listener with the default server group, and start the listener. For more information about how to add a listener, see Add a TCP listener.

   • If the preceding listener exists and runs as normal, proceed to the next step.

6. Check whether the Health Check Status column of the listener displays Normal.

   • If the health check status is not Normal, the backend servers of the SLB instance for the API server are abnormal.

     • If your cluster is an ACK managed cluster, submit a ticket.

     • If your cluster is an ACK dedicated cluster, perform the following operations to troubleshoot the issue. If the issue persists, submit a ticket.

       • On the cluster details page of the ACK console, click the Cluster Resources tab. Click the hyperlink to the right of Master Instance to log on to the Elastic Compute Service (ECS) console and check whether the status of the ECS instance is Running.

         Note

         Repeat the steps to check all the master instances.

       • Use the ECS console to log on to a master node and check whether the container of the API server runs as normal.

         1. For more information about how to log on to a master node, see Connection method overview.

         2. After you log on to a master node, use one of the following methods to check whether the container of the API server runs as normal:

            • If your cluster uses the Docker runtime, run the docker ps | grep kube-apiserver command. Then, run the docker inspect command to check the status of the container based on the output returned by the first command.

            • If your cluster uses the containerd runtime, run the crictl ps | grep kube-apiserver command. Then, run the crictl inspect command to check the status of the container based on the output returned by the first command.

   • If the health check status of the listener is Normal, proceed to the next step.

7. Check whether access control is enabled for the preceding listener.

   • If access control is enabled for the listener, this indicates that the whitelist of the listener is not correctly configured. To resolve this issue, add the CIDR block 100.104.0.0/16 to the whitelist. The CIDR block specifies the source IP addresses of the internal requests that are sent by the ACK management services to the API server. For more information about access control, see Access control.

   • If access control is disabled for the listener, proceed to the next step.

8. If the exception is not caused due to the preceding reasons, submit a ticket.

An API server request exception occurs when you access the log of a pod

The current account does not have the required RBAC permissions to perform the operation, and the ForbiddenQueryClusterNamespace or APISERVER.403 error code is returned

Issue

Cause

The account that you use does not have the required RBAC permissions to perform the operation.

Solution

1. Use an Alibaba Cloud account or an account that has administrator permissions to log on to the ACK console. In the left-side navigation pane, click Authorizations.

2. On the RAM Users tab, find the Resource Access Management (RAM) user that causes the error and click Modify Permissions for the RAM user.

3. On the Permission Management page, click Add Permissions, select a cluster, a namespace, and a predefined RBAC role, and then click Submit.

The current account does not have the required RAM permissions to perform the operation, and the StatusForbidden error code is returned

Issue

Cause

The account that you use does not have the required RAM permissions to perform the operation.

Solution

1. Use an Alibaba Cloud account or an account that has RAM permissions to log on to the RAM console.

2. Grant your account the required permissions based on the CS information that is returned by the system, such as cs:DescribeKubernetesVersionMetadata. For more information, see Create a custom RAM policy.