Container Service for Kubernetes：Service troubleshooting

Procedure

Make sure that the CCM version is 1.9.3.276-g372aa98-aliyun or later before troubleshooting. For more information about how to update the CCM, see Update the CCM. For more information about the release notes of the CCM, see Cloud Controller Manager.

1. Run the following command to query the Service that is associated with the SLB instance:

   kubectl get svc -A |grep -i LoadBalancer|grep ${XXX.XXX.XXX.XXX}  #XXX.XXX.XXX.XXX is the IP address of the SLB instance.

2. Run the following command to check whether events are generated for Service errors:

   kubectl -n {your-namespace} describe svc {your-svc-name}

   Important

   If no events are generated for Service errors, check whether the CMM version is 1.9.3.276-g372aa98-aliyun or later. For more information about how to update the CCM, see Update the CCM.

   • If events are generated for Service errors, refer to Service errors and solutions.

   • If no events are generated for Service errors, refer to Troubleshooting.

3. If the issue persists, Submit a ticket.

Service errors and solutions

Troubleshooting

You can refer to the information provided in the following table to troubleshoot errors other than Service errors.

The SLB instance does not evenly distribute traffic

Cause

The scheduling algorithm specified for the SLB instance is improper.

Issue

Traffic is not evenly distributed to the backend servers of an SLB instance.

The 503 error occurs when I access the SLB instance during application updates

Cause

Connection draining is not configured for the SLB listener or graceful shutdown is not configured for the pod.

Issue

The 503 error occurs when you access the SLB instance during application updates.

The SLB instance cannot be accessed from within the cluster

Cause

externalTrafficPolicy: Local is set for the LoadBalancer. In Local mode, the IP address of the LoadBalancer is accessible only from pods that are provisioned on the local node (the node that runs the LoadBalancer). The IP address of the LoadBalancer is inaccessible from pods on other nodes in the cluster. The IP address of the LoadBalancer is external to the Kubernetes cluster. If nodes or pods in the ACK cluster cannot access the IP address without using a second hop, requests do not pass through the LoadBalancer. As a result, the IP address of the LoadBalancer is considered an extended IP address of the Service that uses the LoadBalancer. Requests are forwarded by kube-proxy based on iptables or IP Virtual Server (IPVS).

In this scenario, if the requested pod is not provisioned on the local node, a connectivity issue occurs. The IP address of the LoadBalancer is accessible only if the requested pod is provisioned on the local node. For more information, see Why kube-proxy add external-lb's address to node local iptables rule?.

Issue

The SLB instance cannot be accessed from within the cluster.

Solution

• Use the cluster IP or Ingress name.

  The Ingress name is nginx-ingress-lb.kube-system.

• Set externalTrafficPolicy of the LoadBalancer Service to Cluster. However, client source IP addresses cannot be preserved in this case. You need to run the following command to modify the Ingress.

  Note

  If the SLB instance is used by the Ingress, only the pods on the node that hosts the Ingress pod can access the Services exposed by the Ingress or SLB instance.

  kubectl edit svc nginx-ingress-lb -n kube-system

• If the cluster uses Terway in ENI mode, set externalTrafficPolicy of the LoadBalancer Service to Cluster and add an annotation in the following format, such as annotation: service.beta.kubernetes.io/backend-type:"eni", to directly access the ENI. This way, client source IP addresses can be preserved and the SLB instance can be accessed from within the cluster. For more information, see Add annotations to the YAML file of a Service to configure CLB instances.

  apiVersion: v1
  kind: Service
  metadata:
    annotations:
      service.beta.kubernetes.io/backend-type: eni
    labels:
      app: nginx-ingress-lb
    name: nginx-ingress-lb
    namespace: kube-system
  spec:
    externalTrafficPolicy: Cluster

The SLB instance cannot be accessed from outside the cluster

Cause

You configured access control list (ACL) rules for the SLB instance or the SLB instance does not run as normal.

Issue

You cannot access the SLB instance from outside the cluster.

Solution

1. Run the following command to query Service events and troubleshoot errors. For more information, see Service errors and solutions.

   kubectl -n {your-namespace} describe svc {your-svc-name}

2. Check whether ACL rules are configured for the SLB instance.

   If ACL rules are configured for the SLB instance, check whether the client IP address is allowed to access the SLB instance. For more information about how to configure ACL rules for an SLB instance, see Access control overview.

3. Check whether the SLB instance is associated with a vServer group.

   If no vServer group is associated, check whether the application pods are associated with the Service and whether the application pods run as normal. If the application pods do not run as normal, identify the causes and troubleshoot the errors. For more information, see Pod troubleshooting.

4. Check whether unhealthy backend servers are detected by the SLB listeners.

   If unhealthy backend servers are detected, check whether the application pods run as normal. For more information about health checks for SLB, see Execute a health check script.

5. If the issue persists, Submit a ticket.

Backend HTTPS services cannot be accessed

Cause

After you specify the certificate information in the SLB instance, the SLB instance decrypts HTTPS requests and then sends HTTP requests to the backend pods.

Issue

You cannot access backend HTTPS services.

Solution

Set targetPort to an HTTP port in the Service. targetPort specifies the port to which the HTTPS port is mapped. For example, the HTTPS port is 443 in the following NGINX Service. In this case, you must change the value of targetPort to 80.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer