All Products
Search

This Product

    Container Service for Kubernetes:Work with Terway

    Document Center

    Container Service for Kubernetes:Work with Terway

    Last Updated:Oct 11, 2024

    Terway is an open source Container Network Interface (CNI) plug-in developed by Alibaba Cloud. Terway works with Virtual Private Cloud (VPC) and allows you to use standard Kubernetes network policies to regulate how containers communicate with each other.

    Before you begin

    To better understand the working mode of Terway, we recommend that you read this topic before you use the Terway network plug-in.

    Before you read this topic, we recommend that you understand the terms used in container network plug-ins and select a container network plug-in. For more information, see Networking overview and Comparison between Terway and Flannel.

    Before you create a cluster, you must plan the CIDR blocks of the cluster. For more information, see Plan the network of an ACK cluster.

    Billing

    Terway is free of charge. However, pods used by Terway are deployed on each node. These pods consume a small amount of node resources. For more information about the billing of Alibaba Cloud services used by ACK, see Cloud service billing.

    Method for calculating the maximum number of pods on a node

    In Terway mode, the maximum number of pods on a node depends on the number of elastic network interfaces (ENIs) provided by the Elastic Compute Service (ECS) instance. Terway has a lower limit for the pods on a node. The maximum number of pods on a node must be at least equal to the lower limit to that the node can join a cluster. The following table describes the details.

    Terway mode

    Maximum number of pods on a node

    Example

    Maximum number of pods that support static IP addresses, separate vSwitches, and separate security groups on a node

    Shared ENI mode

    (EniQuantity - 1) × EniPrivateIpAddressQuantity, where EniQuantity is the number of ENIs provided by an ECS instance type, and EniPrivateIpAddressQuantity is the number of private IP addresses provided by an ENI.

    Note

    The maximum number of pods on a node must be greater than 11 so that the node can join a cluster.

    In this example, the general-purpose ecs.g7.4xlarge instance type is used. This instance type provides 8 ENIs, and each ENI provides 30 private IP addresses. The maximum number of pods on a node is (8 - 1) × 30 = 210.

    Important

    The maximum number of pods that can use ENIs on a node is a fixed value determined by the instance type. If you modify the value of the maxPods parameter, only the maximum number of pods that use the hostNetwork mode is affected.

    0

    Shared ENI + Trunk ENI

    EniTotalQuantity -EniQuantity, where EniTotalQuantity is the maximum number of network interfaces supported by an ECS instance type, and

    Exclusive ENI

    EniQuantity - 1

    Note

    To join the cluster, the maximum number of pods on a node must be greater than 6.

    In this example, the general-purpose ecs.g7.4xlarge instance type is used. This instance type supports 8 ENIs. The maximum number of pods supported by a node is (8 - 1) = 7.

    EniQuantity - 1

    Important

    In Terway v1.11.0 and later versions, you can create node pools that run in exclusive ENI mode or shared ENI mode, and these two types of node pools can exist in the same cluster. For more information, see Terway.

    View the maximum number of pods supported by the node network

    • Method 1: When you create a node pool in the ACK console, you can view the maximum number of pods supported by an instance type in the Terway Mode (Supported Pods) column of the Instance Type section.

    • Method 2: Perform the following steps to manually calculate the maximum number of pods supported by an ECS instance type.

      • Read the relevant documentation to obtain the number of ENIs provided by the instance type. For more information, see Overview of instance families.

      • Query the information in OpenAPI Explorer. Specify the instance type of the node in the InstanceTypes parameter and click Initiate Call. The returned value of the EniQuantity parameter indicates the number of ENIs provided by the instance type. The EniPrivateIpAddressQuantity parameter indicates the number of private IP addresses provided by each ENI. The EniTotalQuantity parameter indicates the maximum number of network interfaces supported by an instance type.

    Install Terway when you create a cluster

    You can select Terway as the network plug-in only when you create a cluster. You cannot change the plug-in after the cluster is created.

    1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

    2. In the upper-right corner of the Clusters page, click Create Kubernetes Cluster.

    3. Configure the key network parameters for Terway. For more information about other cluster parameters, see Create an ACK managed cluster.

      Parameter

      Description

      IPv6 Dual-stack

      Select Enable to create a dual-stack cluster that supports IPv4 and IPv6.

      If you enable IPv4/IPv6 dual-stack, a dual-stack cluster is created. This feature is in public preview. To use this feature, submit an application in the Quota Center console.

      Important

      • Only clusters that run Kubernetes 1.22 and later support IPv4/IPv6 dual-stack.

      • IPv4 addresses are used for communication between worker nodes and the control plane.

      • You must select Terway as the network plug-in.

      • If you use the shared ENI mode of Terway, the ECS instance type must support IPv6 addresses. To add ECS instances of the specified type to the cluster, the number of IPv4 addresses supported by the ECS instance type must be the same as the number of IPv6 addresses. For more information about ECS instance types, see Overview of instance families.

      • You must use a VPC and ECS instances that support IPv4/IPv6 dual stack.

      • You must disable IPv4/IPv6 dual stack if you want to use Elastic Remote Direct Memory Access (eRDMA) in an cluster.

      VPC

      The VPC used by the cluster.

      Network Plug-in

      Select Terway.

      DataPath V2

      If you select this check box, the DataPath V2 acceleration mode is used. After you select the DataPath V2 acceleration mode, Terway adopts a traffic forwarding link that is different from the shared ENI mode for faster network communication. For more information about the features of DataPath V2, see Network acceleration.

      Support for NetworkPolicy

      If you select this check box, Kubernetes-native NetworkPolicies are supported.

      Note

      The feature of managing NetworkPolicies by using the console is in public preview. If you want to use the feature, log on to the Quota Center console and submit an application.

      Support for ENI Trunking

      If you select this check box, the Trunk ENI feature is enabled. You can specify a static IP address, a separate vSwitch, and a separate security group for each pod.

      Note

      • You can select the Support for ENI Trunking option for an ACK managed cluster without the need to submit an application. If you want to enable the Trunk ENI feature in an ACK dedicated cluster, log on to the Quota Center console and submit an application.

      • By default, the Trunk ENI feature is enabled for newly created ACK managed clusters that run Kubernetes 1.31 or later versions.

      vSwitch

      The CIDR block of the vSwitch used by the nodes in the cluster. We recommend that you select at least 3 vSwitches in different zones to ensure high availability.

      Pod vSwitch

      The CIDR block of the vSwitch used by the pod. The CIDR block can overlap with the vSwitch CIDR block of the node.

      Service CIDR

      The Service CIDR block cannot overlap with the node or pod CIDR block.

      IPv6 Service CIDR

      The IP address can be configured after IPv4/IPv6 dual stack is enabled.

    Introduction to Terway modes

    For more information about how Terway works, see the following section.

    Shared ENI mode and exclusive ENI mode

    When you assign an IP address to a pod, Terway operates in one of two modes: Shared ENI Mode or Exclusive ENI Mode.

    Important

    • In Terway v1.11.0 and later versions, you can select either shared ENI mode or exclusive ENI mode for a node pool in a cluster, and this selection is no longer available during cluster creation.

    • The primary ENI on the node is allocated to the node OS. The remaining ENIs are used by Terway to configure the pod network. Therefore, do not manually configure the ENIs. For more information about how to manually manage ENIs, see Configure an ENI filter.

    Item

    Shared ENI mode

    Exclusive ENI mode

    Pod IP address management

    ENI allocation mode

    Multiple pods share the same ENI.

    Each pod is assigned a separate ENI.

    Pod deployment density

    The density of pod deployment is high. Hundreds of pods can be deployed on a single node.

    The density of pod deployment is low. Single-digit pods can be deployed on a regular node.

    Network architecture

    		imageimage

    Data links

    When a pod accesses other pods or is accessed as a Service backend, traffic passes through the network protocol stack of the node.

    When a pod accesses a Service, traffic still passes through the protocol stack of the node OS. However, when a pod accesses other pods or is accessed as a Service backend, the attached ENI is directly used to bypass the node network protocol stack for higher performance.

    Use scenario

    Common Kubernetes scenarios.

    The network performance of this mode is comparable to that of traditional virtual machines, which is suitable for scenarios that require high network performance, such as applications that require high network throughput or low latency.

    Network acceleration

    DataPath V2 network acceleration is supported. For more information, see Network acceleration.

    Network acceleration is not supported. The exclusive ENI resources of a pod provide excellent network performance.

    Support for NetworkPolicy

    Kubernetes-native NetworkPolicies are supported and provide access control capabilities.

    The NetworkPolicy feature is not supported.

    Access control

    The Trunk ENI feature allows you to specify a static IP address, a separate vSwitch, and a separate security group for each pod. For more information, see Configure a static IP address, a separate vSwitch, and a separate security group for each pod.

    By default, you can configure a static IP address, a separate vSwitch, and a separate security group for a pod.

    Network acceleration

    If you use the shared ENI mode of Terway, you can enable the network acceleration mode. After you enable the acceleration mode, Terway uses a traffic forwarding path different from that of the regular shared ENI mode to achieve higher performance. DataPath V2 is supported in Terway. For more information about the features of DataPath V2, see the following description.

    Important

    • DataPath V2 is an upgraded version of the earlier IPVLAN and extended Berkeley Packet Filter (eBPF) acceleration mode. You can select only DataPath V2 when you create a cluster that is installed with Terway 1.8.0 or later.

    • The DataPath V2 and IPVLAN + eBPF acceleration modes apply only to node pools that run in shared ENI mode. They are not applicable to node pools that run in exclusive ENI mode.

    DataPath V2 feature

    Description

    Applicable Terway version

    Clusters that are installed with Terway 1.8.0 or later.

    Network architecture

    		image

    Accelerated data link

    • When a pod accesses a Service, eBPF is used to resolve the Service address to the address of a backend pod of the Service.

    • When a pod accesses another pod on a different node, eBPF is used to bypass the network protocol stacks of the nodes.

    • When a pod accesses another pod on the same node, the pod bypasses the node protocol stack. The traffic is forwarded within the node.

    Performance optimization

    • The network forwarding process of pods on the host is simplified. This way, the performance of pod networks is almost the same as that of the host network, and the latency is reduced by 30% compared with the traditional mode.

    • Service networks are implemented based on the eBPF technology instead of the kube-proxy mode. Traffic bypasses the host iptables or IP Virtual Server (IPVS), which greatly reduces the network latency. The impact on the network performance is reduced in large-scale clusters and the scalability is improved.

    • The network policies (NetworkPolicy) of pods are implemented based on the eBPF technology instead of iptables. This way, large numbers of iptables rules are no longer generated on the host and the impact of network policies on network performance is reduced.

    Usage method

    Create a cluster, set Network Plug-in to Terway, and select DataPath V2.

    Usage notes

    • Only Alibaba Cloud Linux OS images are supported.

    • The Sandboxed-Container runtime is not supported.

    • The implementation logic of network policies (NetworkPolicy) is different from that of the original Terway.

      • The CIDR selector has a lower priority than the pod selector. If the CIDR selector includes the pod CIDR block, you must add another pod selector.

      • The except keyword of the CIDR selector is not fully supported. We recommend that you do not use the except keyword.

      • If you use a network policy (NetworkPolicy) of the Egress type, you cannot access pods in the host network or the IP addresses of nodes in the cluster.

    • You may fail to access the Internet-facing SLB instance that is associated with a LoadBalancer Service from within the cluster due to loopback issues. For more information, see Why am I unable to access an SLB instance?

    The IPVLAN + eBPF acceleration mode is selected in an existing cluster. For more information about the features of IPVLAN + eBPF, see the following description.

    IPVLAN + eBPF

    Feature

    Description

    Applicable Terway version

    Clusters that are installed with Terway 1.7.0 or later.

    Network architecture

    		image

    Accelerated data link

    • When a pod accesses a Service, eBPF is used to resolve the Service address to the address of a backend pod of the Service in the network namespace of the pod.

    • When a pod accesses another pod on a different node, IPVLAN is used to bypass the network protocol stacks of the nodes.

    Usage method

    Create a cluster, set Network Plug-in to Terway, and select Pod IPVLAN.

    Access control

    The shared ENI mode of Terway supports the NetworkPolicy feature and the Support for ENI Trunking option. This allows you to facilitate the management of network traffic in a cluster. The exclusive ENI mode of Terway also supports specific traffic control capabilities.

    Support for NetworkPolicy

    • Node pools that run in exclusive ENI mode of Terway does not support the NetworkPolicy feature.

    • Node pools that run in shared ENI mode of Terway supports the Kubernetes-native NetworkPolicy feature. Network policies allow you to use user-defined rules to control network traffic between pods.

      Create a cluster, set Network Plug-in to Terway, and select Support for NetworkPolicy. This way, the cluster supports the NetworkPolicy feature. For more information, see Use network policies in ACK clusters.

      Note

      The feature of managing NetworkPolicies by using the console is in public preview. If you want to use the feature, log on to the Quota Center console and submit an application.

    Configure a static IP address, a separate vSwitch, and a separate security group for a pod

    • Node pools that run in exclusive ENI mode of Terway allows you to specify a static IP address, a separate vSwitch, and a separate security group for each pod. This allows you to manage and isolate user traffic, configure network policies, and manage IP addresses in a fine-grained manner.

    • The Trunk ENI feature is an option in node pools that run in shared ENI mode of Terway. The Trunk ENI feature allows you to specify a static IP address, a separate vSwitch, and a separate security group for each pod.

      Create a cluster, set Network Plug-in to Terway, and select Support for ENI Trunking. For more information, see Configure a static IP address, a separate vSwitch, and a separate security group for each pod.

      Note

      • You can select the Support for ENI Trunking option for an ACK managed cluster without the need to submit an application. If you want to enable the Trunk ENI feature in an ACK dedicated cluster, log on to the Quota Center console and submit an application.

      • By default, the Trunk ENI feature is enabled for newly created ACK managed clusters that run Kubernetes 1.31 or later versions.

      • After the Trunk ENI feature is enabled, the terway-eniip and terway-controlplane components are installed.