In most cases, pods in Kubernetes clusters use private IP addresses. However, in some scenarios, you may want to assign an exclusive public IP address to a pod to allow the pod to separately communicate with external networks without being affected by the traffic of other pods. You can use Terway or ack-extend-network-controller to directly associate an elastic IP address (EIP) with a pod. This way, the pod can communicate with the Internet over an exclusive data egress.
Prerequisites
A Container Service for Kubernetes (ACK) managed cluster or an ACK dedicated cluster is created. The cluster uses Terway as the network plug-in. For more information, see Create an ACK managed cluster and Create an ACK dedicated cluster. For more information about Terway, see Work with Terway.
Since November 2023, Terway has discontinued updates of the EIP feature. This feature was removed in v1.7.0. We recommend that you use the ack-extend-network-controller component to associate an exclusive EIP with a pod. For more information, see Migrate EIPs from Terway to ack-extend-network-controller.
Limits
Before you use EIPs, you must understand the limits of EIPs. For more information, see Limits.
Billing rules
You are not charged for enabling Terway or ack-extend-network-controller. However, you may be charged fees for the cloud resources that you use. For more information about the billing of Alibaba Cloud services used by ACK, see Cloud service billing.
Background information
In most cases, SNAT and EIPs are used to enable pods to access the Internet in Terway mode. Traffic from the Internet is forwarded by a LoadBalancer Service to backend pods. However, you may want to assign an exclusive public IP address to a pod in the following scenarios:
The pod exposes random ports to the Internet in scenarios in which game servers and audio conferences use UDP. For example, if Real Time Streaming Protocol (RTSP) is used, different ports are used for different clients.
The pod has to compete with other pods for a public IP address to access the Internet. In this case, the pod requires an exclusive public IP address.
Step 1: Configure the RAM permissions that are required for associating EIPs
ack-extend-network-controller
Grant RAM permissions to an ACK managed cluster or ACK dedicated cluster
Create a custom policy and add the following content to the policy. For more information, see the Step 1: Create a custom policy section of the "[Product Changes] Permissions of the worker RAM role of ACK managed clusters are revoked" topic.
{ "Effect": "Allow", "Action": [ "vpc:DescribeVSwitches", "vpc:AllocateEipAddress", "vpc:AllocateEipAddressPro", "vpc:DescribeEipAddresses", "vpc:AssociateEipAddress", "vpc:UnassociateEipAddress", "vpc:ReleaseEipAddress", "vpc:AddCommonBandwidthPackageIp", "vpc:RemoveCommonBandwidthPackageIp", "vpc:TagResources", "ecs:DescribeNetworkInterfaces" ], "Resource": [ "*" ], "Condition": {} }
Grant permissions to the worker RAM role of the cluster. For more information, see the Attach the custom policy to the worker RAM role section of the "[Product Changes] Permissions of the worker RAM role of ACK managed clusters are revoked" topic.
Grant RAM permissions to an ACK Serverless cluster
If your cluster is an ACK Serverless cluster, you need to generate an AccessKey pair for the RAM user. For more information, see Create a RAM user and Create custom policies.
Terway
Terway requires EIP management permissions to apply for and associate EIPs. Therefore, the custom policy also provides permissions related to EIPs.
Grant RAM permissions to an ACK managed cluster or an ACK Basic cluster created in June 2020 or later.
Create a custom policy and add the following content to the policy. For more information, see the Step 1: Create a custom policy section of the "[Product Changes] Permissions of the worker RAM role of ACK managed clusters are revoked" topic.
{ "Effect": "Allow", "Action": [ "vpc:DescribeVSwitches", "vpc:AllocateEipAddress", "vpc:DescribeEipAddresses", "vpc:AssociateEipAddress", "vpc:UnassociateEipAddress", "vpc:ReleaseEipAddress", "vpc:AddCommonBandwidthPackageIp", "vpc:RemoveCommonBandwidthPackageIp" ], "Resource": [ "*" ], "Condition": {} }
Grant permissions to the worker RAM role of the cluster. For more information, see the Step 2: Attach the custom policy to the worker RAM role section of the "[Product Changes] Permissions of the worker RAM role of ACK managed clusters are revoked" topic.
Grant RAM permissions to an ACK dedicated cluster or an ACK managed cluster created before June 2020.
Create a custom policy and add the following content to the policy. For more information, see the Step 1: Create a custom policy section of the "[Product Changes] Permissions of the worker RAM role of ACK managed clusters are revoked" topic.
{ "Effect": "Allow", "Action": [ "vpc:DescribeVSwitches", "vpc:AllocateEipAddress", "vpc:DescribeEipAddresses", "vpc:AssociateEipAddress", "vpc:UnassociateEipAddress", "vpc:ReleaseEipAddress", "vpc:AddCommonBandwidthPackageIp", "vpc:RemoveCommonBandwidthPackageIp" ], "Resource": [ "*" ], "Condition": {} }
On the Permissions tab of the AliyunCSManagedNetworkRole page, click Grant Permission. In the Grant Permission panel, click Policy and select the custom policy that you created in the previous step.
Click Grant permissions.
Click Close.
Step 2: Install or update the plug-in
You cannot use ack-extend-network-controller and Terway to associate EIPs with pods at the same time. Otherwise, an EIP may be associated with multiple pods, which causes EIP association failures. We recommend that you use ack-extend-network-controller.
Comparison of Terway and ack-extend-network-controller
Feature | ack-extend-network-controller | Terway |
Supported cluster types |
|
|
Static EIPs | Supported | Not supported |
Phase | ack-extend-network-controller associates EIPs with pods during the pod IP address allocation phase. | The Terway plug-in associates EIPs with pods when the plug-in applies configurations. |
Supported versions | v0.2.0 and later | v1.0.10.280-gdc2cb6c-aliyun or later versions that are earlier than v1.7.0 |
Install or update the plug-in
If your cluster uses ack-extend-network-controller, go to the Marketplace page in the ACK console, install ack-extend-network-controller, and then enable the EIP controller. If your cluster uses Terway, update Terway to v1.0.10.280-gdc2cb6c-aliyun
or later.
Install ack-extend-network-controller
Log on to the ACK console. In the left-side navigation pane, choose .
On the Marketplace page, search for
ack-extend-network-controller
and then click the plug-in.In the upper-right corner of the chart details page, click Deploy. The Deploy panel appears.
In the Basic Information step, select a cluster and a namespace, and click Next.
In the Parameters step, select a version, configure the parameters, and then click OK.
The following table describes the parameters.
Parameter
Type
Required
Description
clusterID
string
Yes
The ID of the cluster.
regionID
string
Yes
The ID of the region in which the cluster is deployed.
enableControllers
[]string
Yes
The controllers that you want to use. Specify
eip
to enable the EIP feature.networkController.vpcid
string
Yes
The ID of the virtual private cloud (VPC) in which the cluster is deployed.
networkController.eipController.maxConcurrentReconciles
int
No
The number of concurrent EIP controllers.
networkController.eipController.garbageCollectionPeriodInMinutes
int
No
The interval at which the EIP controller clears static EIPs.
customStatefulWorkloadKinds
[]string
No
The type of the custom stateful controller.
enableVirtualNode
bool
No
Specifies whether to support Elastic Container Instance-based pods on virtual nodes.
ImportantIf you set this parameter to true, the controller assigns EIPs to pods running on elastic container instances. The pod annotations you configure must not duplicate the EIP functions supported by Elastic Container Instance. Use this parameter with caution.
enableRRSA
bool
No
Specifies whether to support authentication through RAM Roles for Service Accounts (RRSA).
NoteThis parameter is supported in ack-extend-network-controller v0.10.0 or later.
Only the injection method of ack-pod-identity-webhook is supported for configuring RRSA. You must configure the corresponding labels for the kube-system namespace. For more information, see Use RRSA to authorize different pods to access different cloud services.
rrsaRoleName
string
No
The name of the RRSA role.
NoteThis parameter is supported in ack-extend-network-controller v0.10.0 or later.
credential.accessKey
string
No
The AccessKey ID of the account that is used to associate EIPs.
You do not need to configure this parameter if you grant permissions to the worker RAM role of the cluster in Step 1.
ImportantThis parameter stores sensitive information in a Kubernetes Secret. We recommend that you do not use this parameter.
credential.accessSecret
string
No
The AccessKey secret of the account that is used to associate EIPs.
You do not need to configure this parameter if you grant permissions to the worker RAM role of the cluster in Step 1.
ImportantThis parameter stores sensitive information in a Kubernetes Secret. We recommend that you do not use this parameter.
Sample code:
clusterID: "c11ba338192xxxxxxx" regionID: "cn-hangzhou" vpcID: "vpc-bp1rkq0zxxxxxx" enableControllers: - eip networkController: eipController: maxConcurrentReconciles: 1 garbageCollectionPeriodInMinutes: 1 customStatefulWorkloadKinds: - foo credential: accessKey: "" accessSecret: ""
For more information about how to update the version and parameters of ack-extend-network-controller, see App Marketplace.
Update Terway
If your cluster uses Terway, update the Terway plug-in (terway-eni or terway-eniip) to a version that supports the EIP feature. For more information about how to update Terway, see Manage components.
Terway must be updated to v1.0.10.280-gdc2cb6c-aliyun or a later version that is earlier than v1.7.0. For more information about Terway, see Terway.
Step 3: Enable the EIP feature
Annotations for enabling the EIP feature
ACK allows you to use annotations to enable the EIP feature. You can use the automatic EIP allocation feature or manually associate static EIPs with pods.
Automatically associate an EIP with a pod
If you use the automatic EIP allocation feature, the system may repeatedly apply for and release EIPs. For example, this issue occurs when the pod is recreated or the Container Network Interface (CNI) plug-in fails to configure the network. To prevent this issue, you can manually associate an EIP with the pod by specifying the ID of the EIP.
Pod Annotations | value |
network.alibabacloud.com/pod-with-eip k8s.aliyun.com/pod-with-eip | Specifies whether to automatically create an EIP and associate the EIP with the pod. Valid values:
|
k8s.aliyun.com/eci-with-eip (compatible) | |
network.alibabacloud.com/eip-bandwidth k8s.aliyun.com/eip-bandwidth | The maximum bandwidth of the EIP. Unit: Mbit/s. For more information, see Apply for an EIP. |
network.alibabacloud.com/eip-internet-charge-type k8s.aliyun.com/eip-internet-charge-type | The metering method of the EIP. Valid values:
For more information, see Apply for an EIP. Note
|
k8s.aliyun.com/eip-charge-type (compatible) | |
network.alibabacloud.com/eip-instance-charge-type k8s.aliyun.com/eip-instance-charge-type | The billing method of the EIP. Valid values:
For more information, see Apply for an EIP. Note Only ack-extend-network-controller supports this annotation. |
network.alibabacloud.com/eip-common-bandwidth-package-id k8s.aliyun.com/eip-common-bandwidth-package-id | The Internet Shared Bandwidth instance that you want to use. Note To use this annotation, update Terway to v1.2.3 or later. This annotation can be used with any version of the ack-extend-network-controller plug-in. |
network.alibabacloud.com/eip-isp k8s.aliyun.com/eip-isp | The line type of the EIP. Valid values:
For more information, see Apply for an EIP. Note To use this annotation, update Terway to v1.2.3 or later. This annotation can be used with any version of the ack-extend-network-controller plug-in. |
network.alibabacloud.com/eip-public-ip-address-pool-id k8s.aliyun.com/eip-public-ip-address-pool-id | The IP address pool. For more information, see Create and manage an IP address pool. Note Only ack-extend-network-controller supports this annotation. |
network.alibabacloud.com/eip-resource-group-id k8s.aliyun.com/eip-resource-group-id | The resource group to which the EIP belongs. For more information, see Apply for an EIP. Note Only ack-extend-network-controller v0.4.0 and later support this annotation. |
network.alibabacloud.com/eip-name k8s.aliyun.com/eip-name | The name of the EIP. For more information, see Apply for an EIP. Note Only ack-extend-network-controller v0.6.0 and later support this annotation. |
network.alibabacloud.com/eip-description k8s.aliyun.com/eip-description | The description of the EIP. For more information, see Apply for an EIP. Note Only ack-extend-network-controller v0.6.0 and later support this annotation. |
network.alibabacloud.com/eip-security-protection-types k8s.aliyun.com/eip-security-protection-types | The Anti-DDoS type of the EIP. Anti-DDoS-(Enhanced) is supported. Separate multiple types with commas ( Note Only ack-extend-network-controller v0.9.0 and later support this annotation. |
Specify the ID of an EIP
You can associate an existing EIP with a pod by specifying the ID of the EIP. The pod annotations do not modify the configurations of the EIP. The annotations only associate the EIP with the specified pod.
This feature is not applicable to controllers that manage multiple replicated pods. Make sure that the EIP is used by only one pod. We recommend that you use this feature only in StatefulSets.
Pod Annotations | value |
network.alibabacloud.com/pod-eip-instanceid k8s.aliyun.com/pod-eip-instanceid | The ID of the EIP, such as eip-bp14qxxxxxxx. Important Do not associate an EIP with a pod that uses a name different than the PodEIP resource. The EIP controller disassociates an EIP from a pod when the pod terminates. During the disassociation process, do not use the EIP in other pods. You can determine whether the EIP is disassociated from the pod by checking whether a PodEIP resource that uses the same name as the pod exists. |
k8s.aliyun.com/eci-eip-instanceid (compatible) |
Annotation for configuring EIP release policies
If you configure a pod to use a static EIP, the pod still uses the same EIP after the pod is recreated. You can use this method together with the automatic EIP allocation feature to associate static EIPs with the pods of StatefulSets.
Only ack-extend-network-controller provides this capability and you must use a stateful pod controller.
You can specify an EIP by its ID. The specified EIP will not be released.
Pod Annotation | value |
network.alibabacloud.com/pod-eip-release-strategy k8s.aliyun.com/pod-eip-release-strategy | The EIP release policy. Valid values:
You can directly configure the expiration time of the EIP. For example, if you set the expiration time of the EIP to 5m30s, the EIP that is associated with the pod is released 5.5 minutes after the pod is deleted. Time expressions written in Go are supported. |
Enable the EIP feature
Enable the EIP feature in a cluster that uses ack-extend-network-controller
ack-extend-network-controller needs to access OpenAPI Explorer to create resources. You need to first grant the required permissions to the worker role of the cluster in the RAM console. Then, install ack-extend-network-controller on the Marketplace page of the ACK console, and add annotations to create and associate an EIP with the specified pod.
Add annotations to create and associate an EIP with the specified pod.
You can add annotations to the configurations of a pod to create or associate an EIP with the pod. For more information about annotations, see the FAQ section of this topic.
Create an application. Then, create and associate an EIP with the specified pod.
Deployments
Create a Deployment based on the following sample code. The Deployment allocates an EIP to each pod. The bandwidth of each EIP is 5 Mbit/s.
apiVersion: apps/v1 kind: Deployment metadata: name: example labels: app: example spec: replicas: 1 selector: matchLabels: app: example template: metadata: labels: app: example annotations: k8s.aliyun.com/pod-with-eip: "true" k8s.aliyun.com/eip-bandwidth: "5" spec: readinessGates: - conditionType: "k8s.aliyun.com/eip" containers: - name: example image: nginx
After the pods are created, run the following command to query the
podeips.alibabacloud.com
resources that use the same names as the pods to track the EIP allocation information.kubectl get podeip -n <namespace> <podname> -o yaml
Expected results:
apiVersion: alibabacloud.com/v1beta1 kind: PodEIP metadata: creationTimestamp: "2023-12-15T04:25:37Z" finalizers: - podeip-controller.alibabacloud.com/finalizer generation: 1 name: example-xxx namespace: default resourceVersion: "222800" uid: 43xxx-f1xx-4xxx-b3xx-969faxxx spec: allocationID: eip-2ze2qe8zsxxx allocationType: releaseStrategy: Follow type: Auto status: eipAddress: 39.102.XX.XX internetChargeType: PayByTraffic isp: BGP networkInterfaceID: eni-2zeagv8f3xxxx podLastSeen: "2023-12-15T05:18:47Z" privateIPAddress: 192.168.XX.XX resourceGroupID: rg-acfmwxxxxxsq status: InUse
StatefulSets
Create a StatefulSet based on the following sample code. The StatefulSet creates two pods and allocates an EIP to each pod. The EIPs are automatically released 10 minutes after the pods are deleted.
apiVersion: apps/v1 kind: StatefulSet metadata: name: example labels: app: example spec: serviceName: "example" replicas: 2 selector: matchLabels: app: example template: metadata: labels: app: example annotations: k8s.aliyun.com/pod-with-eip: "true" k8s.aliyun.com/pod-eip-release-strategy: "10m" spec: containers: - name: example image: nginx
After the pods are created, run the following command to query the
podeips.alibabacloud.com
resources that use the same names as the pods to track the EIP allocation information.kubectl get podeip -n <namespace> -o yaml
Expected results:
apiVersion: v1 items: - apiVersion: alibabacloud.com/v1beta1 kind: PodEIP metadata: creationTimestamp: "2023-12-15T03:28:01Z" finalizers - podeip-controller.alibabacloud.com/finalizer generation: 1 name: example-0 namespace: default resourceVersion: "227221" uid: 79954xx-17xx-4dxx-b7xx-15b84xxx spec: allocationID: eip-2ze08metxxx allocationType: releaseAfter: 10m releaseStrategy: TTL type: Auto status: eipAddress: 39.105.XX.XX internetChargeType: PayByTraffic isp: BGP networkInterfaceID: eni-2ze4tkg4xxx podLastSeen: "2023-12-15T05:31:34Z" privateIPAddress: 192.168.XX.XX resourceGroupID: rg-acfmwxxx status: InUse - apiVersion: alibabacloud.com/v1beta1 kind: PodEIP metadata: creationTimestamp: "2023-12-15T03:28:03Z" finalizers: - podeip-controller.alibabacloud.com/finalizer generation: 1 name: example-1 namespace: default resourceVersion: "227222" uid: 1339xxxe7-97xx-46xx-9bxx-537690xxx spec: allocationID: eip-2zetwhffqxxx allocationType: releaseAfter: 10m releaseStrategy: TTL type: Auto status: eipAddress: 39.105.XX.XX internetChargeType: PayByTraffic isp: BGP networkInterfaceID: eni-2zeagv8f3wxxx podLastSeen: "2023-12-15T05:31:34Z" privateIPAddress: 192.168.XX.XX resourceGroupID: rg-acfmwqnwxxx status: InUse - apiVersion: alibabacloud.com/v1beta1 kind: PodEIP metadata: creationTimestamp: "2023-12-15T04:25:37Z" finalizers: - podeip-controller.alibabacloud.com/finalizer generation: 1 name: example-5bxxx-9xx namespace: default resourceVersion: "227220" uid: 43cdfxxx-f1xx-42xx-b3xx-969fxxx spec: allocationID: eip-2ze2qe8zsmnxxx allocationType: releaseStrategy: Follow type: Auto status: eipAddress: 39.102.XX.XX internetChargeType: PayByTraffic isp: BGP networkInterfaceID: eni-2zeagv8f3wxxx podLastSeen: "2023-12-15T05:31:34Z" privateIPAddress: 192.168.XX.XX publicIpAddressPoolID: pippool-2ze498cxxx resourceGroupID: rg-acfmwqnxxx status: InUse kind: List metadata: resourceVersion: ""
After the pods of the StatefulSet are deleted, the EIPs are retained for 10 minutes and then released. If you create a pod with the same name during this period, the pod uses the same EIP.
Verify the configuration.
After the status of a pod changes to Running, you can check the value of the k8s.aliyun.com/allocated-eipAddress annotation of the pod to identify the EIP used by the pod. Then, you can use the EIP to access the pod.
Enable the EIP feature in a cluster that uses Terway
Configure Terway to support EIPs.
Run the following command to modify the eni_conf ConfigMap of Terway:
kubectl edit cm eni-config -n kube-system
Add the following content to the eni_conf ConfigMap:
"enable_eip_pool": "true"
NoteIf you want to disassociate the existing EIP before you associate another EIP, add "allow_eip_rob": "true" to the eni_conf ConfigMap.
After you modify the ConfigMap, press the Esc key. Then, enter :wq! and press the Enter key to save the modified file and exit the edit mode.
Run the following command to redeploy Terway:
kubectl delete pod -n kube-system -l app=terway-eniip # If the terway-eni plug-in is installed, replace terway-eniip with terway-eni.
Add annotations to create and associate an EIP with a pod.
You can add annotations to the configurations of a pod to create or associate an EIP with the pod. For more information about the limits of EIPs, see Limits.
Add annotations based on the scenario.
k8s.aliyun.com/pod-with-eip: "true"
: the ID of an EIP.k8s.aliyun.com/eip-bandwidth: "5"
: the maximum bandwidth of the EIP. The default value is 5 Mbit/s, which equals the default value defined in the EIP service.An EIP cannot be associated with multiple pods. Therefore, you cannot use this annotation for pods that are created by Deployments or StatefulSets.
By default, if an EIP is already associated with a pod, you cannot create a new EIP for the pod. If you want to disassociate the existing EIP and create another EIP, add the "allow_eip_rob": "true" setting to the eni_conf ConfigMap as described in the preceding step.
You can specify an existing EIP only if the application runs in one replicated pod. For example, when you create a Deployment, make sure that the value of Replicas is 1 so that you can associate an existing EIP.
Verify the configuration.
After a pod enters the Running state, you can check the value of the network.alibabacloud.com/allocated-eipAddress annotation of the pod to identify the EIP used by the pod. Then, you can use the EIP to access the pod.
Automatically create and associate an EIP with a pod
Add the following annotations to automatically create and associate an EIP with a pod and specify the maximum bandwidth of the EIP:
The following sample YAML template provides an example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment-basic
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
annotations:
network.alibabacloud.com/pod-with-eip: "true" # Specifies that an EIP is automatically created and associated with each NGINX pod.
network.alibabacloud.com/eip-bandwidth: "5" # Specifies the maximum bandwidth of the EIP. The default value is 5 Mbit/s, which equals the default value defined in the EIP service.
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
Specify an existing EIP
Add the following annotation to specify the ID of an existing EIP and associate the EIP with a pod:
k8s.aliyun.com/pod-eip-instanceid: "<youreipInstanceId>"
Troubleshooting
Why does the status of a pod change to Ready?
After the controller allocates an EIP to a pod, the status of the pod may change to Ready before the pod is associated with the EIP. You can use one of the following methods to control when the status of the pod changes to Ready.
Configure readiness gates for the pod
Only ack-extend-network-controller supports pod readiness gates.
After you configure readiness gates for the pod, the controller sets pod conditions only after an EIP is associated with the pod. The status of the pod does not change to Ready before the pod is associated with an EIP.
kind: Pod
...
spec:
readinessGates:
- conditionType: "k8s.aliyun.com/eip"
status:
conditions:
- lastProbeTime: "2022-12-12T03:45:48Z"
lastTransitionTime: "2022-12-12T03:45:48Z"
reason: Associate eip succeed
status: "True"
type: k8s.aliyun.com/eip
...
Configure an init container for the pod
You can configure an init container for the pod and use the init container to check whether the pod is associated with an EIP. The following sample code shows how to configure an init container.
apiVersion: v1
kind: Pod
metadata:
name: example
annotations:
network.alibabacloud.com/pod-with-eip: "true"
spec:
containers:
- name: example
image: busybox:1.28
command: ['sh', '-c', 'echo The app is running! && sleep 3600']
initContainers:
- name: init
image: busybox:1.28
command: ['timeout', '-t' ,'60', 'sh','-c', "until grep -E '^k8s.aliyun.com\\/allocated-eipAddress=\\S?[0-9]+\\S?' /etc/podinfo/annotations; do echo waiting for annotations; sleep 2; done"]
volumeMounts:
- name: podinfo
mountPath: /etc/podinfo
volumes:
- name: podinfo
downwardAPI:
items:
- path: "labels"
fieldRef:
fieldPath: metadata.labels
- path: "annotations"
fieldRef:
fieldPath: metadata.annotations