Configure a Classic Load Balancer by using annotations in an ACK cluster - Container Service for Kubernetes

You can use annotations in a Service YAML file to configure resources at three levels: Classic Load Balancer (CLB) instances, listeners, and backend server groups. This enables a wide range of load balancing features.

Index

Category	Feature Classification	Configuration link
Annotation usage
Typical operations for a CLB instance	Create an SLB instance	Create an Internet-facing SLB instance Create an internal-facing SLB instance Create a pay-by-bandwidth SLB instance Specify primary and secondary zones for an SLB instance Create a pay-by-LCU SLB instance Create an IPv6 SLB instance
	Specify an existing instance	Use an existing SLB instance Use an existing SLB instance and forcibly overwrite existing listeners
	Configure an SLB instance	Specify the specification of an SLB instance Specify a vSwitch for an SLB instance Specify an IP address for an internal-facing SLB instance Add additional tags to an SLB instance Specify a name for an SLB instance Specify a resource group for an SLB instance Set a hostname for a service
	Enable instance protection	Enable deletion protection for an SLB instance Enable configuration read-only mode for an SLB instance Retain an auto-created SLB instance when you delete a service
Typical operations for a listener	Session persistence settings	Configure session persistence for a TCP listener Configure session persistence for HTTP and HTTPS listeners (insert cookie)
	Port and protocol configuration	Specify a redirection port for an SLB instance Create a listener with a health check Create a UDP listener Create an HTTP listener Create an HTTPS listener Configure additional domain name certificates for an HTTPS listener Configure both TCP and UDP protocols for a listener Configure Proxy Protocol for TCP and UDP listeners
	Advanced configuration	Set a scheduling algorithm for an SLB instance Configure an access control policy group for an SLB instance Configure a security policy for a listener Enable connection draining for a listener Configure additional request headers for a listener Set an idle connection timeout for a listener Specify a request timeout for a listener Specify a connection timeout for a listener Disable HTTP/2 for a listener
Typical operations for a backend server group	Configuration management	Use worker nodes with specified labels as backend servers Use the nodes where pods reside as backend servers Remove unschedulable nodes from the backend of a CLB instance Directly attach pod ENIs to the backend of a CLB instance Reuse an existing vServer group Set the weight of a service to receive traffic Ignore backend server weight updates
References

Annotation usage

Annotations are case-sensitive.
Before you use an annotation, check the supported Cloud Controller Manager (CCM) versions for this feature. To upgrade the CCM component, see Manage components. For the change history of the CCM component, see Cloud Controller Manager.
As of September 11, 2019, alicloud in the annotations field was updated to alibaba-cloud.
For example:
Before the update: service.beta.kubernetes.io/alicloud-loadbalancer-id
After the update: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id
The system remains compatible with the alicloud format. You do not need to make any changes.

Typical operations for a CLB instance

Create an Internet-facing SLB instance

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an internal-facing SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type

Description

Supported CCM versions

Specifies the network type of the SLB instance. Valid values:

internet: The service is accessed over the Internet. The Address Type of the CLB instance must be Internet.
intranet: The service is accessed over a private network. The Address Type of the CLB instance must be Internal.

Default value: internet

We recommend that you see Specify a vSwitch for an SLB instance to manually specify a vSwitch for the CLB instance. If you do not specify a vSwitch, the system automatically selects one based on the CCM component version:

Versions earlier than v2.13.0: The first vSwitch configured on the cluster's control plane is used by default.
v2.13.0 and later: One vSwitch is automatically selected from all available vSwitches.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Specify the specification of an SLB instance

Annotation: Multiple annotations are available, as described in the following table.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type

The billing method of the instance. Valid values:

PayBySpec: pay-by-specification.
PayByCLCU: pay-by-LCU.

Default value: PayBySpec

Important

For Cloud Controller Manager v2.5.0 and later, the default value is changed to PayByCLCU.
You cannot set a specification for a pay-by-LCU SLB instance. This means that the value PayByCLCU cannot be used together with the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec annotation.

v2.4.0 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec

The specification of the SLB instance. Use this parameter to create a CLB instance of a specific specification or update the specification of an existing CLB instance. Valid values:

slb.s1.small
slb.s2.small
slb.s2.medium
slb.s3.small
slb.s3.medium
slb.s3.large

Default value: slb.s1.small

For more information about the valid values of this parameter, see CreateLoadBalancer.

Important

If you modify the specification of a CLB instance in the CLB console (only pay-by-specification instances are supported), CCM may change the specification back to the original one. Proceed with caution.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type: "PayBySpec"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec: "slb.s1.small"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use an existing SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id

Description

Supported CCM versions

Important

To prevent unexpected behaviors such as cluster unavailability or traffic interruptions, do not reuse the API Server CLB instance or a CLB instance created by CCM. Manually create a new instance in the Classic Load Balancer (CLB) console.
Do not add or modify the reuse annotation for an existing LoadBalancer service. This may cause the reuse to fail or prevent the original SLB instance created by CCM from being released.

The ID of the SLB instance. Use this annotation to specify an existing CLB instance.

By default, using an existing SLB instance does not overwrite its listeners. To forcibly overwrite existing listeners, set service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners to "true". This configuration overwrites only the listeners of the current service and does not affect other listeners.
Note
By default, reusing an existing SLB instance does not overwrite existing listeners for the following two reasons:
- If services are bound to the listeners of the existing SLB instance, a forced overwrite may cause service interruptions.
- CCM supports only a limited number of backend configurations and cannot handle complex configurations. If you have complex backend configuration requirements, you can configure listeners in the console without overwriting them.
For these reasons, we recommend that you do not forcibly overwrite listeners. You can forcibly overwrite listeners if the listener ports of the existing SLB instance are no longer in use.
(For versions earlier than v2.10.0) When you use an existing SLB instance, you cannot add additional tags (annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags).

v1.9.3.81-gca19cd4-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: "${YOUR_LOADBALANCER_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use an existing SLB instance and forcibly overwrite existing listeners

Annotation: Multiple annotations are available, as described in the following table. Forcibly overwriting existing listeners deletes the existing listeners if a port conflict occurs.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id

Important

To prevent unexpected behaviors such as cluster unavailability or traffic interruptions, do not reuse the API Server CLB instance or a CLB instance created by CCM. Manually create a new instance in the Classic Load Balancer (CLB) console.

The ID of the SLB instance. Use this annotation to specify an existing CLB instance.

By default, using an existing SLB instance does not overwrite its listeners. To forcibly overwrite existing listeners, set service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners to "true". This configuration overwrites only the listeners of the current service and does not affect other listeners.
Note
By default, reusing an existing SLB instance does not overwrite existing listeners for the following two reasons:
- If services are bound to the listeners of the existing SLB instance, a forced overwrite may cause service interruptions.
- CCM supports only a limited number of backend configurations and cannot handle complex configurations. If you have complex backend configuration requirements, you can configure listeners in the console without overwriting them.
For these reasons, we recommend that you do not forcibly overwrite listeners. You can forcibly overwrite listeners if the listener ports of the existing SLB instance are no longer in use.
(For versions earlier than v2.10.0) When you use an existing SLB instance, you cannot add additional tags (annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags).

v1.9.3.81-gca19cd4-aliyun and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners

When you attach an existing Server Load Balancer, decide whether to sync the CLB listener configuration based on the Service.

"true": CCM automatically manages the CLB listeners associated with the service based on the service configuration (create, update, and delete). Listeners not managed by CCM are not affected.
"false": CCM does not perform any operations on the CLB listeners.

Default value: "false"

Important

When you reuse an existing CLB instance and set force-override to "true", do not reuse the same listener of the same CLB instance for multiple services. Otherwise, listener configuration conflicts may occur.

v1.9.3.81-gca19cd4-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: "${YOUR_LOADBALANCER_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: "true"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify primary and secondary zones for an SLB instance

Annotation: Multiple annotations are available, as described in the following table. The primary and secondary zones cannot be modified after the instance is created.

SLB instances in some regions do not support primary and secondary zones. The settings on the CLB instance creation page prevail.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-master-zoneid	The ID of the zone where the primary backend server is located.	v1.9.3.10-gfb99107-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-slave-zoneid	The ID of the zone where the secondary backend server is located.	v1.9.3.10-gfb99107-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-master-zoneid: "cn-hangzhou-k"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-slave-zoneid: "cn-hangzhou-j"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create a pay-by-bandwidth SLB instance

Annotation: Multiple annotations are available, as described in the following table. The following two annotations are required.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-charge-type

The billing method of the SLB instance. Valid values:

paybytraffic
paybybandwidth

Default value: paybytraffic

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth

The bandwidth of the SLB instance. This is the peak bandwidth. The default value is 50. This parameter applies only to Internet-facing SLB instances. For other limits, see Modify the billing method of an Internet-facing SLB instance.

v1.9.3.10-gfb99107-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-charge-type: "paybybandwidth"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth: "2"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify a vSwitch for an SLB instance

Annotation: Multiple annotations are available, as described in the following table. The following two annotations are required.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type

Specifies the network type of the SLB instance. Valid values:

internet: The service is accessed over the Internet. The Address Type of the CLB instance must be Internet.
intranet: The service is accessed over a private network. The Address Type of the CLB instance must be Internal.

Default value: internet

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id

The ID of the vSwitch to which the SLB instance belongs. The vSwitch must be in the same VPC as the Kubernetes cluster.

When you set this parameter, you must also set service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type to "intranet".

You can log on to the VPC console to query vSwitch IDs.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
   service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
   service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: "${YOUR_VSWITCH_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify an IP address for an internal-facing SLB instance

Annotation: Multiple annotations are available, as described in the following table. The following three annotations are required.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type	Specifies the network type of the SLB instance. Valid values: `internet`: The service is accessed over the Internet. This is the default value. The Address Type of the CLB instance must be Internet. `intranet`: The service is accessed over a private network. The Address Type of the CLB instance must be Internal. Default value: `internet`	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id	The ID of the vSwitch to which the SLB instance belongs. The vSwitch must be in the same VPC as the Kubernetes cluster. When you set this parameter, you must also set `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type` to intranet. You can log on to the VPC console to query vSwitch IDs.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip	The IP address of the internal-facing SLB instance. The IP address must be within the destination CIDR block of the vSwitch. Only IPv4 addresses are supported. This parameter must be used together with `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id`. The IP address cannot be changed after it is created.	v2.7.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: "${YOUR_VSWITCH_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip: "192.168.x.x"
  name: nginx
  namespace: default
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 80
      name: http
  selector:
    app: nginx

Add additional tags to an SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags

Description	Supported CCM versions
The list of tags to add. Separate multiple tags with commas (,), for example, `"k1=v1,k2=v2"`. For v2.10.0 and later, you can modify the tags of created instances and reused instances. Important After you add this annotation to a service to specify additional tags, any additional modifications to the tags of the corresponding SLB instance in the console may be overwritten.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags: "Key1=Value1,Key2=Value2" 
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an IPv6 SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version

Description

Supported CCM versions

The IP version of the SLB instance. The IP version cannot be changed after the instance is created. When you use this parameter, the kube-proxy mode of the cluster must be IPVS. Valid values:

ipv4: IPv4.
ipv6: IPv6. The generated IPv6 address can only be accessed in an IPv6 environment.

Default value: ipv4

v1.9.3.220-g24b1885-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: "ipv6"
  name: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Enable deletion protection for an SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-delete-protection

Description

Supported CCM versions

Deletion protection for the SLB instance. Valid values:

on
off

Default value: on

Important

For an SLB instance created for a LoadBalancer service, if you manually enable deletion protection in the CLB console, you can still delete the SLB instance associated with the service by running the kubectl delete svc {your-svc-name} command.

v1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-delete-protection: "on"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Enable configuration read-only mode for an SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-modification-protection

Description

Supported CCM versions

Configuration read-only mode for the SLB instance. Valid values:

ConsoleProtection
NonProtection

Default value: ConsoleProtection

v1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-modification-protection: "ConsoleProtection"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Specify a name for an SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name

Description	Supported CCM versions
The name of the SLB instance. The name must be 2 to 128 characters in length, start with a letter or a Chinese character, and can contain digits, periods (.), underscores (_), and hyphens (-).	v1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name: "your-svc-name"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Specify a resource group for an SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id

Description	Supported CCM versions
The ID of the resource group to which the SLB instance belongs. The resource group ID cannot be modified after it is specified. You can query resource group IDs on the Alibaba Cloud Resource Management platform.	v1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id: "rg-xxxx"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Set a hostname for a service

Annotation: Multiple annotations are available, as described in the following table.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Separate multiple values with commas (,), for example, https:443,http:80.

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname

Sets a hostname for the service. The hostname must follow DNS naming conventions.

Note the following items:

After you add this annotation, the EXTERNAL-IP of the service changes from the default CLB IP address to your_service_name. When you access the CLB IP address from within the cluster, traffic is routed through the CLB instance and then forwarded to the cluster.
After you add this annotation, if the listener protocol is TCP or UDP, a loopback access issue occurs when you access the CLB IP address from within the cluster. For more information, see A client fails to access an SLB instance.
Adding this annotation does not automatically bind the CLB instance to the domain name. To bind them, go to the domain name service page to purchase a domain name and bind it to the CLB instance. To purchase a domain name, see Purchase a domain name.

v2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname: "${your_service_hostname}"
  name: nginx-svc
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Expected output:

NAME         TYPE           CLUSTER-IP       EXTERNAL-IP            PORT(S)                      AGE
nginx-svc    loadBalancer   47.100.XX.XX     www.example.com        80:30248/TCP,443:32670/TCP   10s

Create a pay-by-LCU SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type

Description

Supported CCM versions

The billing method of the instance. Valid values:

PayBySpec: The default value. pay-by-specification.
PayByCLCU: pay-by-LCU.

Default value: PayBySpec

Important

For Cloud Controller Manager v2.5.0 and later, the default value is changed to PayByCLCU.
You cannot set a specification for a pay-by-LCU SLB instance. This means that the value PayByCLCU cannot be used together with the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec annotation.

v2.4.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type: "PayByCLCU"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Retain an auto-created SLB instance when you delete a service

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-lb-on-delete

Description

Supported CCM versions

When you delete a LoadBalancer service, the CLB instance created for the service is retained, and the kubernetes.do.not.delete and ack.aliyun.com tags are removed from the CLB instance. The existing servers in the vServer group are retained.

When this feature is enabled, a Warning event of the PreservedOnDelete type is generated during service synchronization. After you configure this annotation, check whether this event exists to confirm that the feature is enabled.

Valid values:

Not empty: The retention feature is enabled.
Empty or not set: The retention feature is not enabled.

Important

Perform this operation by deleting the service instead of changing the service type. Otherwise, the service may be incorrectly re-associated with the previously retained CLB instance.

v2.10.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-lb-on-delete: "true"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Typical operations for a listener

Configure session persistence for a TCP listener

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-persistence-timeout

Description

Supported CCM versions

The session persistence timeout period. This parameter applies only to TCP listeners. If multiple TCP listener ports are configured for the SLB instance, this configuration is applied to all TCP listeners by default.

Unit: seconds. Valid values: [0, 3600]. The default value is 0, which indicates that session persistence is disabled. For more information, see CreateLoadBalancerTCPListener.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-persistence-timeout: "1800"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Configure session persistence for HTTP and HTTPS listeners (insert cookie)

Annotation: Multiple annotations are available, as described in the following table. When you insert a cookie, the following four annotations are required.

This feature is supported only for SLB instances that use HTTP or HTTPS.
If multiple HTTP or HTTPS listener ports are configured, this session persistence configuration applies to all HTTP and HTTPS listeners by default.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session	Specifies whether to enable session persistence. This parameter applies only to HTTP and HTTPS listeners. Valid values: `on` `off` Default value: `off` For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type	The method to handle cookies. This parameter applies only to HTTP and HTTPS listeners. This parameter is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session` is set to `on`. Valid values: `insert`: insert cookie. `server`: rewrite cookie. For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie-timeout	The cookie timeout period. This parameter is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session` is set to `on` and `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type` is set to `insert`. Unit: seconds. Valid values: [1, 86400]. For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie	The name of the cookie configured on the server. The name must be 1 to 200 characters in length and can contain only ASCII letters and digits. It cannot contain commas (,), semicolons (;), or spaces. It cannot start with a dollar sign ($). This parameter is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session` is set to `on` and `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type` is set to `server`. For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type: "insert"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie-timeout: "1800"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure an access control policy group for an SLB instance

Annotation: Multiple annotations are available, as described in the following table. The following three annotations are required.

Before you use this annotation to create an SLB instance with access control, you must first create an access control policy group in the Classic Load Balancer (CLB) console and record its ID (acl-id).

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status	Specifies whether to enable access control. Valid values: `on` `off` Default value: `off`	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-id	The ID of the access control policy group bound to the listener. This parameter is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status` is set to `"on"`.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-type	The type of access control. Valid values: `white`: Only requests from the IP addresses or CIDR blocks specified in the selected access control policy group are forwarded. A whitelist is suitable for scenarios where only specific IP addresses can access an application. Setting a whitelist poses certain business risks. Once a whitelist is set, only the IP addresses in the whitelist can access the SLB listener. If you enable a whitelist but do not add any IP addresses to the access control policy group, the SLB listener forwards all requests. `black`: All requests from the IP addresses or CIDR blocks specified in the selected access control policy group are not forwarded. A blacklist is suitable for scenarios where an application restricts access from only specific IP addresses. If you enable a blacklist but do not add any IP addresses to the access control policy group, the SLB listener forwards all requests. This parameter is required when the AclStatus parameter is set to on.	v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-id: "${YOUR_ACL_ID}" # Multiple policy groups are not supported.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-type: "white"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify a redirection port for an SLB instance

Port forwarding redirects requests from an HTTP port to an HTTPS port.

Annotation: Multiple annotations are available, as described in the following table. The following three annotations are required.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of the certificate on Alibaba Cloud. Log on to the CLB console and view the certificate ID on the Certificate Management page. Note To create a certificate, see Select an Alibaba Cloud-issued certificate.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-forward-port	Forwards HTTP requests to a specified HTTPS port, for example, `80:443`.	v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443,http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-forward-port: "80:443"
  name: nginx
  namespace: default
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Set a scheduling algorithm for an SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler

Description

Supported CCM versions

The scheduling algorithm. Valid values:

wrr: weighted round-robin. Backend servers with higher weights receive more requests.
rr: round-robin. External requests are sequentially distributed to backend servers.
sch: source IP hash. Requests from the same source IP address are scheduled to the same backend server.
tch: four-element hash (source IP address + destination IP address + source port + destination port). Requests from the same stream are scheduled to the same backend server.

Default value: rr.

For more information about the valid values of this parameter, see the Scheduler field in the API documentation for creating the corresponding listener type, such as Create a TCP listener.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler: "wrr"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Create a UDP listener

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: UDP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an HTTP listener

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Health check: Health checks are disabled by default for HTTP listeners. To ensure the availability of backend services, we recommend configuring an HTTP or TCP health check for the listener port.
Intra-cluster access: When you access a Service from within the cluster using its CLB IP address, traffic may be short-circuited within the cluster, which can cause access issues. To force traffic to bypass the CLB, add an annotation as described in Set a Hostname for a Service. For more information, see Notes on Accessing the External IP of a LoadBalancer Service Within a Cluster.

Description	Supported CCM versions
Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an HTTPS listener

Annotation: Multiple annotations are available, as described in the following table.

HTTPS requests are decrypted at the CLB layer and then sent to the backend pods as HTTP requests.

Health check: Health checks are disabled by default for HTTPS listeners. To ensure the availability of backend services, we recommend configuring an HTTP or TCP health check for the listener port.
Intra-cluster access: When you access a Service from within the cluster using its CLB IP address, traffic may be short-circuited within the cluster, which can cause access issues. To force traffic to bypass the CLB, add an annotation as described in Set a Hostname for a Service. For more information, see Notes on Accessing the External IP of a LoadBalancer Service Within a Cluster.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Separate multiple values with commas (,), for example, https:443,http:80.

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id

The ID of the certificate on Alibaba Cloud.

Log on to the CLB console and view the certificate ID on the Certificate Management page.

Note

To create a certificate, see Select an Alibaba Cloud-issued certificate.

v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

SSL protocol error when accessing a CLB instance from within a cluster

Symptoms

After you create an HTTPS listener for a service, the service can be accessed from outside the cluster. However, if you use curl to access the HTTPS port of the CLB instance associated with the service from a node or pod within the cluster, the following error is returned:

curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Cause

This issue is caused by the IPVS rules on the node:

Service configuration: The HTTPS listener of the CLB instance supports only HTTP as the backend protocol. Therefore, the service can only forward traffic from port: 443 to targetPort: 80. ACK then creates an IPVS rule on the node to directly forward traffic from port 443 to backend port 80.
Layer 4 forwarding: IPVS operates at Layer 4 of the TCP/IP protocol stack. It only forwards TCP packets and does not parse application-layer protocols, such as TLS or HTTPS.
Protocol mismatch: The HTTPS request (TLS handshake data) initiated by the client (curl) is directly forwarded by IPVS to the backend HTTP port 80 of the service. Because this port is not configured for TLS, it cannot parse the TLS request. As a result, an HTTP 400 error is returned, and the client reports an SSL protocol error.

Solution

Add the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname annotation to the service. This annotation prevents IPVS rules from being generated on the node and forces traffic within the cluster to be routed through the CLB instance. This ensures that TLS is correctly handled on the CLB instance. For more information, see .

Configure additional domain name certificates for an HTTPS listener

Annotation: Multiple annotations are available, as described in the following table.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of the certificate on Alibaba Cloud. Log on to the CLB console and view the certificate ID on the Certificate Management page. Note To create a certificate, see Select an Alibaba Cloud-issued certificate.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-domain-extensions	Specifies the additional domain names and certificate IDs. The format is `{Domain Name}:{Certificate ID}`. Separate multiple values with commas (,). Log on to the CLB console and view the certificate ID on the Certificate Management page.	v2.13.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-domain-extensions: "${DOMAIN_1}:${CERT_ID_1},${DOMAIN_2}:${CERT_ID_2}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create a listener with a health check

Set a TCP health check

Annotation: Multiple annotations are available, as described in the following table. All of the following annotations are required.

Health checks are enabled by default for TCP ports.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch	Specifies whether to enable health checks for TCP and UDP listeners. Valid values: `on` `off` Default value: `on`	v2.6.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type	The health check type. Valid values: `tcp` `http` Default value: `tcp`. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout	The timeout period for a health check response. This parameter applies to TCP health checks. If a backend ECS instance does not respond within the specified timeout period, the health check fails. Unit: seconds. Valid values: [1, 300]. If the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout` is smaller than the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval`, the former is invalid and the timeout period is the value of the latter. The default value is 5. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks required to change the health check status of a backend server from `fail` to `success`. Valid values: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks required to change the health check status of a backend server from success to fail. Valid values: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval between two consecutive health checks. Unit: seconds. Valid values: [1, 50]. Default value: 2. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "tcp"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "8"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "3"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Set a UDP health check

Annotation: Multiple annotations are available, as described in the following table. All of the following annotations are required.

Health checks are enabled by default for UDP ports.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch	Specifies whether to enable health checks for TCP and UDP listeners. Valid values: `on` `off` Default value: `on`	v2.6.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout	The timeout period for a health check response. This parameter applies to TCP health checks. If a backend ECS instance does not respond within the specified timeout period, the health check fails. Unit: seconds. Valid values: [1, 300]. If the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout` is smaller than the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval`, the former is invalid and the timeout period is the value of the latter. The default value is 5. For more information, see CreateLoadBalancerUDPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks required to change the health check status of a backend server from fail to success. Valid values: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerUDPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks required to change the health check status of a backend server from success to fail. Valid values: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerUDPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval between two consecutive health checks. Unit: seconds. Valid values: [1, 50]. Default value: 2. For more information, see CreateLoadBalancerUDPListener.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "5"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "10"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "3"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "3"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: UDP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Disable health checks for TCP and UDP listeners

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch

Description

Supported CCM versions

Specifies whether to enable health checks for TCP and UDP listeners. Valid values:

on
off

Default value: on

v2.6.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "off" # Disable the health check.
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Set an HTTP health check

Annotation: Multiple annotations are available, as described in the following table.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag	Valid values: `on`: The default value for TCP listeners is `on` and cannot be changed. You do not need to change this annotation for TCP listeners. `off`: The default value for HTTP listeners is `off`. Default value: `off`	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type	The health check type. Valid values: `tcp` `http` Default value: `tcp` For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri	The URI for health checks. You do not need to configure this annotation for TCP health checks.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-httpcode	The HTTP status code that indicates a successful health check. Separate multiple status codes with commas (,). Valid values: `http_2xx` `http_3xx` `http_4xx` `http_5xx` Default value: `http_2xx` For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain	The domain name for health checks. Valid values: `$_ip`: The private IP address of the backend server. If you specify an IP address or do not specify this parameter, the SLB instance uses the private IP address of each backend server as the domain name for health checks. `domain`: The domain name must be 1 to 80 characters in length and can contain letters, digits, periods (.), and hyphens (-). For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout	The timeout period for a health check response. This parameter applies to HTTP health checks. If a backend ECS instance does not respond within the specified timeout period, the health check fails. Unit: seconds. Valid values: [1, 300]. If the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout` is smaller than the value of service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval, `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout` is invalid and the timeout period is the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval`. For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks required to change the health check status of a backend server from fail to success. Valid values: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks required to change the health check status of a backend server from success to fail. Valid values: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval between two consecutive health checks. Unit: seconds. Valid values: [1, 50]. Default value: 2. For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method	The health check method for an HTTP listener. Valid values: `head` `get`	v2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "http"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri: "/test/index.html"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout: "10"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "3"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    # Set the HTTP status code for the health check. This parameter is optional.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-httpcode: "http_4xx"
    # Set the domain name for the health check. This parameter is optional.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain: "www.aliyun.com"
    # Set the health check method. This parameter is optional.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method: "head"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Enable connection draining for a listener

Annotation: Multiple annotations are available, as described in the following table. All of the following annotations are required.

This feature is supported only for TCP and UDP.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain

Specifies whether to enable connection draining. Valid values:

on
off

v2.0.1 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout

The timeout period for connection draining. Unit: seconds. Valid values: [10, 900].

v2.0.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: "30"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure additional request headers for a listener

Annotation: Multiple annotations are available, as described in the following table.

This feature is supported only for HTTP or HTTPS.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-proto	Specifies whether to use the X-Forwarded-Proto header to retrieve the listener protocol of the CLB instance. Valid values: `on` `off` Default value: `off`	v2.1.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-slbport	Specifies whether to use the X-Forwarded-For-SLBPORT header to retrieve the listener port of the SLB instance. Valid values: `on` `off` Default value: `off`	v2.9.1 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-clientsrcport	Specifies whether to use the X-Forwarded-For-ClientSrcPort header to retrieve the port of the client that accesses the SLB instance. Valid values: `on` `off` Default value: `off`	v2.9.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-proto: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-slbport: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-clientsrcport: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Set an idle connection timeout for a listener

Annotation: Multiple annotations are available, as described in the following table.

This feature is supported only for HTTP or HTTPS.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Separate multiple values with commas (,), for example, https:443,http:80.

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout

The idle connection timeout period for the listener. Unit: seconds. Valid values: [1, 60].

Default value: 15

v2.1.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout: "30"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Disable HTTP/2 for a listener

Annotation: Multiple annotations are available, as described in the following table.

This feature is supported only for HTTPS.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of the certificate on Alibaba Cloud. Log on to the CLB console and view the certificate ID on the Certificate Management page. Note To create a certificate, see Select an Alibaba Cloud-issued certificate.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-http2-enabled	Specifies whether to enable HTTP/2. Valid values: `on` `off` Default value: `on`	v2.1.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-http2-enabled: "off"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Specify a request timeout for a listener

Annotation: Multiple annotations are available, as described in the following table.

This feature is supported only for HTTP or HTTPS.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Separate multiple values with commas (,), for example, https:443,http:80.

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-request-timeout

The request timeout period. Unit: seconds. Valid values: [1, 180]. Default value: 60

If a backend server does not respond within the timeout period, the SLB instance stops waiting and returns an HTTP 504 error code to the client.

v2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-request-timeout: "60"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Specify a connection timeout for a listener

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-established-timeout

This feature is supported only for TCP.

Description	Supported CCM versions
The connection timeout period. Unit: seconds. Valid values: [10, 900]. For more information, see CreateLoadBalancerTCPListener.	v2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-established-timeout: "60"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure a security policy for a listener

Annotation: Multiple annotations are available, as described in the following table.

This feature is supported only for HTTPS.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of the certificate on Alibaba Cloud. Log on to the CLB console and view the certificate ID on the Certificate Management page. Note To create a certificate, see Select an Alibaba Cloud-issued certificate.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy	A security policy includes the optional TLS versions and supported cipher suites for HTTPS. For more information, see CreateLoadBalancerHTTPSListener. Valid values: `tls_cipher_policy_1_0` `tls_cipher_policy_1_1` `tls_cipher_policy_1_2` `tls_cipher_policy_1_2_strict` `tls_cipher_policy_1_2_strict_with_1_3` Default value: `tls_cipher_policy_1_0`	v2.4.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443,http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy: "tls_cipher_policy_1_2"
  name: nginx
  namespace: default
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure both TCP and UDP protocols for a listener

Note

This feature requires Kubernetes v1.24 or later. For more information about how to upgrade a cluster, see Upgrade an ACK cluster.

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: udp
    port: 80
    protocol: UDP
    targetPort: 81
  selector:
    app: nginx
  sessionAffinity: None
  type: LoadBalancer

Configure Proxy Protocol for TCP and UDP listeners

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol

When you access a service using the CLB instance's IP address from within the cluster, traffic might be intercepted inside the cluster, leading to access exceptions. To force traffic routing through the CLB instance, add an annotation. For more information, see Set a hostname for a service and Notes on accessing the external IP address of a LoadBalancer service from within a cluster.

Description

Supported CCM versions

Configures Proxy Protocol for TCP and UDP listeners. After you configure Proxy Protocol, you can use it to carry the source IP address of the client to the backend server. Valid values:

on
off

Default value: off

Important

This feature does not support smooth online migration. Switching Proxy Protocol requires a service outage for the upgrade. Configure this parameter with caution.

v2.6.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Typical operations for a backend server group

Use worker nodes with specified labels as backend servers

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-backend-label

Description	Supported CCM versions
Specifies which worker nodes to attach to the backend of the CLB instance using labels. Separate multiple labels with commas (,), for example, `"k1=v1,k2=v2"`. Multiple labels have an AND relationship.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-backend-label: "failure-domain.beta.kubernetes.io/zone=ap-southeast-5a"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use the nodes where pods reside as backend servers

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler

By default, externalTrafficPolicy is set to Cluster mode, which attaches all nodes in the cluster to the backend server group. Local mode attaches only the nodes where the pods reside as backend servers.

Description

Supported CCM versions

The scheduling algorithm. Valid values:

wrr: weighted round-robin. Backend servers with higher weights receive more requests.
In Local mode, you must set the scheduling policy to weighted round-robin (wrr).
Note
For v1.9.3.164-g2105d2e-aliyun and later, if the external traffic policy is set to Local mode, the service automatically sets the weight of a node based on the number of pods on the node. For more information about how the weight is calculated, see How is the node weight automatically set in Local mode?.
rr: round-robin. External requests are sequentially distributed to backend servers.

Default value: rr

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler: "wrr"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Remove unschedulable nodes from the backend of a CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-remove-unscheduled-backend

Description

Supported CCM versions

Removes SchedulingDisabled nodes from the backend of the CLB instance. Valid values:

on: Nodes in the unschedulable state are removed from the backend server group of the CLB instance.
off: The kubectl cordon and kubectl drain commands set a node to the unschedulable state. In this case, nodes in the unschedulable state are not removed from the backend server group of the CLB instance.

Default value: off

v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-remove-unscheduled-backend: "on"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 30080
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Directly attach pod ENIs to the backend of a CLB instance

Annotation: service.beta.kubernetes.io/backend-type

Description

Default value

Supported CCM versions

The type of backend server for the CLB instance. Valid values:

eni: Use this configuration to attach pods to the backend of the CLB instance, which can improve network forwarding performance. This setting is effective only in Terway network mode.
You can also manually change eni in service.beta.kubernetes.io/backend-type: "eni" to ecs to attach ECS instances to the backend of the CLB instance.
ecs: Attach ECS instances to the backend of the CLB instance.

Flannel network mode: ecs
Terway network mode:
- Terway clusters created before August 10, 2020: ecs
- Terway clusters created after August 10, 2020: eni

v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/backend-type: "eni"
  name: nginx
spec:
  ports:
  - name: http
    port: 30080
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Reuse an existing vServer group

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vgroup-port

You can reuse an existing vServer group. This is effective only when you reuse an existing CLB instance. For a usage example, see Deploy services across clusters by reusing an SLB instance.

Set the weight of a service to receive traffic

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-weight

When multiple services reuse the same CLB instance, you can use this annotation to set the weight of the current service to receive traffic. This annotation is effective only when you reuse an existing vServer group. For a usage example, see Deploy services across clusters by reusing an SLB instance.

Ignore backend server weight updates

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ignore-weight-update

Description

Supported CCM versions

Skips updating the weights of backend servers in the vServer group during service synchronization. This configuration is suitable for scenarios where you need to manually manage backend server weights using a mechanism other than CCM. Valid values:

on
off

Default value: off

v2.11.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ignore-weight-update: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer