:How can I troubleshoot network issues of Kubernetes clusters?

Problem description

In a Kubernetes cluster, multiple errors may occur when you connect to a pod that runs on a node. The errors include:

• An error occurs when the pod accesses the Internet.
• An error occurs when the pod accesses other services.
• An error occurs when the pod accesses the pods the run on other nodes.

Causes

• An error or conflict occurs when you configure the Elastic Compute Service (ECS) security group for the Kubernetes cluster.
• An error or conflict occurs when you configure route entries in a Virtual Private Cloud (VPC) network for the Kubernetes cluster.

Solutions

Modify the ECS security group and the route entries of the VPC network based on your requirements.

ECS security group

When you create a Kubernetes cluster, a security group is created. The security group supports the following features:

• All nodes in the cluster use the security group.
• The inbound rules of the security group allow data packets that are transferred through the CIDR block of the pod. All ports and all protocols are supported.

  Note: This CIDR block is the pod CIDR block that you specify when you create the cluster.

• The outbound rules allow all data packets.

  Note: We recommend that you do not modify the outbound rules.

After the required security group is created, check the following items:

• You can modify the inbound rules as needed. Ensure that the rules that allow the data packets transferred through the pod CIDR block have a higher priority over the rules that deny all requests. Otherwise, a network error may occur in the Kubernetes cluster.
• Ensure that the inbound rules of the security group for the Kubernetes cluster allow the data packets transferred through the pod CIDR block and the outbound rules allow all data packets.

Route tables of the VPC network

The Kubernetes cluster is created based on the Terway network plug-in.

In this mode, the pod CIDR block serves a vSwitch that belongs to the VPC CIDR block. This mode enables cross-host communication for pods without the need for the VPC route tables. You do not need to configure route tables for the VPC.

The Kubernetes cluster is created based on another network plug-in.

In this mode, the pod CIDR block does not belong to the VPC CIDR block. The cross-host communication of pods depends on the route entries that are specified in the VPC route tables. The route entries for the Kubernetes cluster must meet the following requirements:

• The number of route entries that are specified for the Kubernetes cluster must be the same as the number of nodes in the cluster.

  Note: The differences between both numbers are not allowed.

• Each route entry contains the following content:
  • The destination address is the pod CIDR block of the Kubernetes cluster.

    Note: On the command line, you can enter kubectl describe node [$Node_Name] |grep PodCIDR to obtain the pod CIDR block. [$Node_Name] specifies a node name.

  • The next hop IP address is the ID of the ECS instance that runs on the node.
• The route entry must be accurate. If the route entries have errors or overlap with each other, network issues may occur in the cluster.

Conclusion

To troubleshoot network connection issues of Kubernetes clusters, we recommend that you check the configurations that are described in this topic. Most of the issues are caused by conflicts or errors in the configurations.

Application scope

• Dedicated clusters of Container Service for Kubernetes
• Managed clusters of Container Service for Kubernetes