Container Service for Kubernetes：Service troubleshooting

Procedure

Make sure that the CCM version is 1.9.3.276-g372aa98-aliyun or later before troubleshooting. For more information about how to update the CCM, see Update the CCM. For more information about the release notes of the CCM, see Cloud Controller Manager.

1. Run the following command to query the Service that is associated with the CLB instance:

   kubectl get svc -A | grep -i LoadBalancer | grep {XXX.XXX.XXX.XXX}# XXX.XXX.XXX.XXX is the IP address of the CLB instance.

2. Run the following command to check whether events are generated for Service errors:

   kubectl -n {your-namespace} describe svc {your-svc-name}

   Important

   If no events are generated for Service errors, check whether the CMM version is 1.9.3.276-g372aa98-aliyun or later. For more information about how to update the CCM, see Update the CCM.

   • If events are generated for Service errors, refer to Service errors and solutions.

   • If no events are generated for Service errors, refer to Troubleshooting.

3. If the issue persists, submit a ticket.

Service errors and solutions

Troubleshooting

You can refer to the information provided in the following table to troubleshoot errors other than Service errors.

The CLB instance does not evenly distribute traffic

Cause

The scheduling algorithm specified for the SLB instance is improper.

Issue

Traffic is not evenly distributed to the backend servers of a CLB instance.

The 503 error occurs when I access the CLB instance during application updates

Cause

Connection draining is not configured for the CLB listener or graceful shutdown is not configured for the pod.

Issue

The 503 error occurs when I access the CLB instance during application updates.

The CLB instance cannot be accessed from within the cluster

Cause

externalTrafficPolicy: Local is set for the LoadBalancer. In Local mode, the IP address of the LoadBalancer is accessible only from pods that are provisioned on the local node (the node that runs the LoadBalancer). The IP address of the LoadBalancer is inaccessible from pods on other nodes in the cluster. The IP address of the LoadBalancer is external to the Kubernetes cluster. If nodes or pods in the ACK cluster cannot access the IP address without using a second hop, requests do not pass through the LoadBalancer. As a result, the IP address of the LoadBalancer is considered an extended IP address of the Service that uses the LoadBalancer. Requests are forwarded by kube-proxy based on iptables or IP Virtual Server (IPVS).

In this scenario, if the requested pod is not provisioned on the local node, a connectivity issue occurs. The IP address of the LoadBalancer is accessible only if the requested pod is provisioned on the local node. For more information, see Why kube-proxy add external-lb's address to node local iptables rule?.

Issue

The CLB instance cannot be accessed from within the cluster.

Solution

• Use the cluster IP or Ingress name.

  The Ingress name is nginx-ingress-lb.kube-system.

• Set externalTrafficPolicy of the LoadBalancer Service to Cluster. However, client source IP addresses cannot be preserved in this case. You need to run the following command to modify the Ingress.

  Note

  If the CLB instance is used by the Ingress, only the pods on the node that hosts the Ingress pod can access the Services exposed by the Ingress or CLB instance.

  kubectl edit svc nginx-ingress-lb -n kube-system

• If the cluster uses Terway in ENI mode, set externalTrafficPolicy of the LoadBalancer Service to Cluster and add an annotation in the following format, such as annotation: service.beta.kubernetes.io/backend-type:"eni", to directly access the ENI. This way, client source IP addresses can be preserved and the CLB instance can be accessed from within the cluster. For more information, see Add annotations to the YAML file of a Service to configure CLB instances.

  apiVersion: v1
  kind: Service
  metadata:
    annotations:
      service.beta.kubernetes.io/backend-type: eni
    labels:
      app: nginx-ingress-lb
    name: nginx-ingress-lb
    namespace: kube-system
  spec:
    externalTrafficPolicy: Cluster

The CLB instance cannot be accessed from outside the cluster

Cause

You configured access control list (ACL) rules for the CLB instance or the CLB instance does not run as normal.

Issue

The CLB instance cannot be accessed from outside the cluster.

Solution

1. Run the following command to query Service events and troubleshoot errors. For more information, see Service errors and solutions.

   kubectl -n {your-namespace} describe svc {your-svc-name}

2. Check whether ACL rules are configured for the CLB instance.

   If ACL rules are configured for the CLB instance, check whether the client IP address is allowed to access the CLB instance. For more information about how to configure ACL rules for a CLB instance, see Access control.

3. Check whether the CLB instance is associated with a vServer group.

   If no vServer group is associated, check whether the application pods are associated with the Service and whether the application pods run as normal. If the application pods do not run as normal, identify the causes and troubleshoot the errors. For more information, see Pod troubleshooting.

4. Check whether unhealthy backend servers are detected by the CLB listeners.

   If unhealthy backend servers are detected, check whether the application pods run as normal. For more information about health checks for CLB, see Execute a health check script.

5. If the issue persists, submit a ticket.

Backend HTTPS services cannot be accessed

Cause

After you specify the certificate information in the CLB instance, the CLB instance decrypts HTTPS requests and then sends HTTP requests to the backend pods.

Issue

You cannot access backend HTTPS services.

Solution

Set targetPort to an HTTP port in the Service. targetPort specifies the port to which the HTTPS port is mapped. For example, the HTTPS port is 443 in the following NGINX Service. In this case, you must change the value of targetPort to 80.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer