Analysis of Alibaba Cloud Container Network Data Link (5): Terway ENI-Trunking

By Yu Kai

Co-Author: Xieshi, Alibaba Cloud Container Service

This article is the fifth part of the series. It mainly introduces the forwarding links of data plane links in Kubernetes Terway Elastic Network Interface (ENI) – Trunking. On the one hand, we can find out the reasons for different access results that customers receive in different scenarios and help customers further optimize the business architecture by understanding the forwarding links of the data plane in different scenarios. On the other hand, by understanding the forwarding links in-depth, customer O&M and staff from Alibaba Cloud can choose the link points to manual deployment and observation to further classify and locate the problems.

1. Architecture Design for Terway ENI-Trunking

Trunk ENIs are virtual network interfaces that can be bound with Elastic Computer Service (ECS) instances deployed in virtual private clouds (VPCs). Compared with Elastic Network Interface (ENI), the instance resource density of Trunk ENI is significantly improved. After you enable the Terway Trunk ENI, the pods you specified will use Trunk ENI resources. Custom configurations are optional for pods. By default, the Terway Trunk ENI feature is disabled for pods. In this case, pods use IP addresses allocated from a shared ENI. Terway can simultaneously use shared ENI and Trunk ENI to allocate IP addresses to a pod only after you claim the pod to use custom configurations. The two modes share the maximum number of pods on the node. The total deployment density is the same as before the mode is enabled.

Industries (such as finance, telecommunications, and government) have strict data security requirements for data information security. Generally, core important data is placed in a self-built data center, and there is a strict whitelist of the clients that access this data. Generally, specific IP access sources are restricted. When the business architecture is migrated to the cloud, we usually connect user-created data centers to cloud resources through leased lines, VPNs, etc. Since the PodIP in traditional containers is not fixed, Network Policies can only take effect in clusters, which poses a great challenge to the whitelist. When ENI is in the Trunk mode, you can configure independent security groups and vSwitches to provide a more refined network configuration and a highly competitive container network solution.

In the trunking namespace, you can see the relevant pod information and node information. The network of the IP address of the pod application will be described in detail later.

The pod has a default route that only points to eth0. This indicates that the pod accesses all address fields with eth0 as the unified gateway.

How do pods communicate with ECS OS? At the OS level, when we see the network interface card of calicxxxx, we can see it is attached to eth1. The communication connection between nodes and pods is similar to what's described in Panoramic Analysis of Alibaba Cloud Container Network Data Link (3) — Terway ENIIP. No more details are provided here. Through the OS Linux Routing, we can see that all traffic destined for Pod IP will be forwarded to pod's corresponding calico virtual network interface card. Here, ECS OS and Pod's network namespace have established a complete inbound and outbound link configuration.

Let's focus on ENI Trunking itself. How does ENI Trunking configure the switches and the security groups of pods? Terway allows you to use custom resource definitions (CRD) named PodNetworking to describe network configurations. You can create multiple PodNetworkings to design different network planes. After the PodNetworking is created, Terway automatically synchronizes the network configurations. The PodNetworking takes effect on pods only after the status of the PodNetworking changes to Ready. As shown in the following figure, the type is Elastic. As long as the tag of namespace conforms to tryunking:zoneb, the pod can use the specified security groups and switches.

When a pod is created, the pod will match PodNetworkings through labels. If a pod does not match a PodNetworking, the pod is assigned an IP address from a shared ENI by default. If a pod matches a PodNetworking, an ENI is assigned to the pod based on the configurations specified in the PodNetworking. Please see labels for more information about the Pod label.

Terway automatically creates CRDs named PodENI for pods that match PodNetworkings to track the resource usage of pods. PodENIs are managed by Terway and cannot be modified. The following centos-59cdc5c9c4-l5vf9 pod in the trunking namespace matches the podnetworking settings and is assigned with the corresponding member ENI, corresponding trunking ENI, security group, switch, and bound ECS instance. This way, we can configure and manage the switch and the security group in the Pod dimension.

In the ECS console, we can clearly see the relationship between the member ENI and the Trunking ENI with the information on the corresponding security groups and switches.

The configurations above allow us to know how to configure switches and security groups for each pod, and these configurations will take effect if we can enable each pod to automatically go to the corresponding configured Member ENI after it leaves the ECS through Trunking ENI. All configurations are implemented on the host through relevant policies. How does the Trunking ENI network interface card forward the traffic of the corresponding pod to the correct corresponding Member ENI? This is done through vlan. The VLAN ID can be seen at the tc level. Therefore, the VLAN ID is marked or removed at the egress or ingress stage.

Therefore, the overall Terway ENI-Trunking model can be summarized as:

Trunk ENIs are virtual network interface cards that can be bound with Elastic Computer Service (ECS) instances deployed in virtual private clouds (VPCs). Compared with ENIs, the instance resource density of Trunk ENIs is significantly improved.
The Terway Trunk ENI allows you to specify a fixed IP address, a separate Virtual Switch (vSwitch), and a separate security group for each pod. This allows you to manage and isolate user traffic, configure network policies, and manage IP addresses in a fine-grained manner.
You need to purchase 5th or 6th generation ECS Bare Metal Instances with eight cores or above to use the Terway plug-in. In addition, the instances must support Trunk ENIs. Please see Overview of Instance Family for more information.
The maximum number of pods that each node supports depends on the number of ENIs assigned to the node. The maximum number of pods supported by each shared ENI = (Number of ENIs supported by each ECS instance-1) × Number of private IP addresses supported by each ENI.
Pod security group rules do not apply to the traffic between pods on the same node and between nodes and pods on the same node. If any restrictions are needed, you can use NetworkPolicy to do it.
Pods and the corresponding MemberENI traffic are mapped by VLAN ID.

2. Terway ENI-Trunking Mode Container Network Data Link Analysis

Since the security group and switch can be set in the Pod dimension, access to different links will inevitably become more complicated in macro terms. We can roughly divide the network links in Terway ENI-TRunking mode into two large SOP scenarios (Pod IP and SVC), or we can further divide them into ten different small SOP scenarios.

The data links of these 11 scenarios can be summarized into the following ten typical scenarios:

Access pods through nodes (same or different security groups)
Mutual access between Trunk pods on the same node and security group (including access to SVC IP and the source and SVC backend deployed on the same node)
Mutual access between Trunk pods in different security groups on the same node (including access to SVC IP and the source and SVC backend deployed on the same node)
Mutual access between different nodes and Trunk pods in the same security group
Mutual access between Eth-Trunk pods on different nodes and different security groups
Source accesses to the SVC IP address in the cluster (different source and SVC backend nodes and the same security group, including access to the external IP address in Local mode)
Source accesses to the SVC IP in the cluster (different source and SVC backend nodes and different security groups, including access to external IP in Local mode)
In Cluster mode, the source in the cluster accesses the SVC ExternalIP (different source and SVC backend nodes and different security groups)
In Cluster mode, the source in the cluster accesses the SVC ExternalIP (same security group)
Access the SVC IP address outside the cluster.

2.1 Scenario 1: Use a Node to Access Pods (Same or Different Security Groups)

Environment

nginx-6f545cb57c-kt7r8 and 10.0.4.30 exist on the cn-hongkong.10.0.4.22 node.

Kernel Routing

The IP address of nginx-6f545cb57c-kt7r8 is 10.0.4.30. The PID of the container on the host is 1734171, and the container network namespace has a default route pointing to container eth0.

The container eth0 is a tunnel established in ECS through ipvlan tunnel and ECS's subsidiary ENI eth1. The subsidiary ENI eth1 also has a virtual calxxx network interface card.

In the ECS OS, there is a route that points to the IP address of the pod with calixxxxx as the next hop. As you can see from the preceding section, the calixxx network interface card is a pair that consists of veth1 in each pod. Therefore, the CIDR of the pod that accesses the SVC will point to veth1 instead of the default eth0 route. Therefore, the calixx network interface card here works to:

Help node access the pod
When the node or pod accesses the CIDR of the SVC, it helps the node or pod go through the ECS OS kernel protocol stack conversion to calixxx and veth1 to access the pod.

The nginx-6f545cb57c-kt7r8 pod in the trunking namespace matches the corresponding podnetworking settings and is assigned with the corresponding member ENI, corresponding trunking ENI, security group, switch, and bound ECS instance, thus realizing the configuration and management of the switch and security group in the Pod dimension.

VLAN ID 1027 can be seen at the tc level, so data traffic will be tagged or removed from VLAN IDs at the egress or ingress stage.

From the security group to which the network interface card of the ENI belongs, we can see that only the specified IP address is allowed to access port 80 of the nginx pod.

The traffic forwarding logic placed on the data surface at the OS level is similar to what's described in Panoramic Analysis of Alibaba Cloud Container Network Data Link (3) — Terway ENIIP. No more details are provided here.

Summary

Access to Target

Data Link Forwarding Diagram

It will pass through the calico network interface card. Each pod that is not the host network will form a veth pair with the calico network interface card, which is used to communicate with other pods or nodes.
The entire link and the request will not pass through the ENI allocated by the pod. The IP rule is hit directly in the ns of the OS and is forwarded.
The entire request link is ECS1 OS → calixxxx → ECS1 Pod1.
Since forwarding is performed through the OS kernel routing and does not pass through the member ENI, the security group does not take effect. This link has nothing to do with the member ENI security group to which the pod belongs.