How to configure a Classic Load Balancer (CLB) instance for an ACK cluster by using annotations - Container Service for Kubernetes

You can use annotations in a Service YAML file to configure Classic Load Balancer (CLB) instances, listeners, and backend server groups to implement a wide range of load balancing features.

Index

Category	Feature category	Configuration link
Annotation usage notes
Typical operations for CLB instances	Create an SLB instance	Create an Internet-facing SLB instance Create an internal-facing SLB instance Create a pay-by-bandwidth SLB instance Specify primary and secondary zones when you create an SLB instance Create a pay-by-usage SLB instance Create an IPv6 SLB instance
	Specify an existing instance	Use an existing SLB instance Use an existing SLB instance and forcibly overwrite existing listeners
	Configure an SLB instance	Specify the specification of an SLB instance Specify a vSwitch for an SLB instance Specify an IP address for an internal-facing SLB instance Add additional tags to an SLB instance Specify the name of an SLB instance Specify the resource group of an SLB instance Set a hostname for a service
	Enable instance protection	Enable deletion protection for an SLB instance Enable configuration read-only mode for an SLB instance Retain an auto-created SLB instance when you delete a service
Typical operations for listeners	Session persistence settings	Configure the session persistence timeout for a TCP listener Configure session persistence (insert cookie) for HTTP and HTTPS listeners
	Port and protocol configuration	Specify a redirection port for an SLB instance Create a listener with a health check Create a UDP listener Create an HTTP listener Create an HTTPS listener Configure additional domain name certificates for an HTTPS listener Configure TCP and UDP protocols for a listener at the same time Configure the Proxy Protocol for TCP and UDP listeners
	Advanced configuration	Set the scheduling algorithm for an SLB instance Configure an access control policy group for an SLB instance Configure a security policy for a listener Enable connection draining for a listener Configure additional request headers for a listener Set the idle connection timeout for a listener Specify the request timeout for a listener Specify the connection timeout for a listener Disable the HTTP/2 feature for a listener
Typical operations for backend server groups	Configuration management	Use worker nodes with a specific label as backend servers Use the nodes where pods reside as backend servers Remove unschedulable nodes from the backend server group of a CLB instance Directly attach pod ENIs to the backend of a CLB instance Reuse an existing vServer group Set the weight of a service to receive traffic Ignore backend server weight updates
References

Annotation usage notes

Annotations are case-sensitive.
Before you use an annotation, verify the supported Cloud Controller Manager (CCM) versions for the feature. To upgrade the CCM component, see Manage components. For more information about the change history of the CCM component, see Cloud Controller Manager.
Starting from September 11, 2019, alibaba-cloud is used in the annotations field instead of alicloud.
For example:
Before the update: service.beta.kubernetes.io/alicloud-loadbalancer-id
After the update: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id
The system remains compatible with the alicloud format. No changes are required.

Typical operations for CLB instances

Create an Internet-facing SLB instance

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an internal-facing SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type

Description

Supported CCM versions

Specifies the network type of the SLB instance. Values:

internet: The service is accessed over the Internet. The Address Type of the CLB instance must be Public Network.
intranet: The service is accessed over a private network. The Address Type of the CLB instance must be Private Network.

Default value: internet

See Specify a vSwitch for an SLB instance to manually specify a vSwitch for the CLB instance. If you do not specify a vSwitch, the system automatically selects one based on the CCM version:

Versions earlier than v2.13.0: The first vSwitch configured on the cluster control plane is used by default.
v2.13.0 and later: One vSwitch is automatically selected from all available vSwitches.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Specify the specification of an SLB instance

Annotation: Multiple annotations are available, as described in the following table.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type

The billing method of the instance. Values:

PayBySpec: pay-by-specification.
PayByCLCU: pay-by-usage.

Default value: PayBySpec

Important

For Cloud Controller Manager v2.5.0 and later, the default value is changed to PayByCLCU.
Pay-by-usage SLB instances do not support specifications. Therefore, the value PayByCLCU cannot be used together with the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec annotation.

v2.4.0 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec

The specification of the SLB instance. Use this parameter to create a CLB instance of a specific specification or update the specification of an existing CLB instance. Values:

slb.s1.small
slb.s2.small
slb.s2.medium
slb.s3.small
slb.s3.medium
slb.s3.large

Default value: slb.s1.small

For more information about the values of this parameter, see CreateLoadBalancer.

Important

If you modify the specification of a pay-by-specification CLB instance in the CLB console, CCM may change the specification back to the original one. Proceed with caution.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type: "PayBySpec"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec: "slb.s1.small"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use an existing SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id

Description

Supported CCM versions

Important

To prevent unexpected behavior such as cluster unavailability or traffic interruption, do not reuse the API Server CLB instance or a CLB instance created by CCM. Manually create a new instance in the Classic Load Balancer (CLB) console.

The ID of the SLB instance. Use this annotation to specify an existing CLB instance.

By default, using an existing SLB instance does not overwrite its listeners. To forcibly overwrite existing listeners, set service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners to "true". This configuration overwrites only the listeners of the current service and does not affect other listeners.
Note
Using an existing SLB instance does not overwrite its listeners by default for the following reasons:
- If the listeners of the existing SLB instance are bound to a service, forcibly overwriting the listeners may cause service interruptions.
- CCM supports only limited backend configurations and cannot process complex configurations. If you have complex backend configuration requirements, you can configure listeners in the console without overwriting them.
For these reasons, we recommend that you do not forcibly overwrite listeners. You can forcibly overwrite a listener if its port is no longer in use.
For CCM versions earlier than v2.10.0, you cannot add additional tags (annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags) when you use an existing SLB instance.

v1.9.3.81-gca19cd4-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: "${YOUR_LOADBALANCER_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use an existing SLB instance and forcibly overwrite existing listeners

Annotation: Multiple annotations are available, as described in the following table. Forcibly overwriting existing listeners deletes them if a port conflict occurs.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id

Important

The ID of the SLB instance. Use this annotation to specify an existing CLB instance.

By default, using an existing SLB instance does not overwrite its listeners. To forcibly overwrite existing listeners, set service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners to "true". This configuration overwrites only the listeners of the current service and does not affect other listeners.
Note
Using an existing SLB instance does not overwrite its listeners by default for the following reasons:
- If the listeners of the existing SLB instance are bound to a service, forcibly overwriting the listeners may cause service interruptions.
- CCM supports only limited backend configurations and cannot process complex configurations. If you have complex backend configuration requirements, you can configure listeners in the console without overwriting them.
For these reasons, we recommend that you do not forcibly overwrite listeners. You can forcibly overwrite a listener if its port is no longer in use.
For CCM versions earlier than v2.10.0, you cannot add additional tags (annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags) when you use an existing SLB instance.

v1.9.3.81-gca19cd4-aliyun and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners

When you bind an existing SLB instance, this annotation determines whether to synchronize the CLB listener configuration based on the service.

"true": CCM automatically manages the CLB listeners associated with the service (create, update, and delete) based on the service configuration. Listeners not managed by CCM are not affected.
"false": CCM does not process CLB listeners.

Default value: "false"

Important

When you reuse an existing CLB instance and set force-override to "true", do not reuse the same listener of the CLB instance for multiple services. Otherwise, listener configuration conflicts may occur.

v1.9.3.81-gca19cd4-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: "${YOUR_LOADBALANCER_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: "true"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify primary and secondary zones when you create an SLB instance

Annotation: Multiple annotations are available, as described in the following table. The primary and secondary zones cannot be modified after the instance is created.

SLB instances in some regions do not support primary and secondary zones. The settings on the CLB instance creation page take precedence.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-master-zoneid	The ID of the zone where the primary backend server is deployed.	v1.9.3.10-gfb99107-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-slave-zoneid	The ID of the zone where the secondary backend server is deployed.	v1.9.3.10-gfb99107-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-master-zoneid: "cn-hangzhou-k"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-slave-zoneid: "cn-hangzhou-j"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create a pay-by-bandwidth SLB instance

Annotation: Multiple annotations are available, as described in the following table. The following two annotations are required.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-charge-type

The billing method of the SLB instance. Values:

paybytraffic
paybybandwidth

Default value: paybytraffic

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth

The bandwidth of the SLB instance. This is the peak bandwidth. Default value: 50. This parameter applies only to Internet-facing SLB instances. For more information about other limits, see Modify the billing method of an Internet-facing SLB instance.

v1.9.3.10-gfb99107-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-charge-type: "paybybandwidth"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth: "2"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify a vSwitch for an SLB instance

Annotation: Multiple annotations are available, as described in the following table. The following two annotations are required.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type

Specifies the network type of the SLB instance. Values:

internet: The service is accessed over the Internet. The Address Type of the CLB instance must be Public Network.
intranet: The service is accessed over a private network. The Address Type of the CLB instance must be Private Network.

Default value: internet

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id

The ID of the vSwitch to which the SLB instance belongs. The vSwitch must be in the same VPC as the Kubernetes cluster.

When you set this parameter, you must also set service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type to "intranet".

Log on to the VPC console to query the vSwitch ID.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
   service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
   service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: "${YOUR_VSWITCH_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify an IP address for an internal-facing SLB instance

Annotation: Multiple annotations are available, as described in the following table. The following three annotations are required.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type	Specifies the network type of the SLB instance. Values: `internet`: The service is accessed over the Internet. This is the default value. The Address Type of the CLB instance must be Public Network. `intranet`: The service is accessed over a private network. The Address Type of the CLB instance must be Private Network. Default value: `internet`	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id	The ID of the vSwitch to which the SLB instance belongs. The vSwitch must be in the same VPC as the Kubernetes cluster. When you set this parameter, you must also set `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type` to intranet. Log on to the VPC console to query the vSwitch ID.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip	The IP address of the internal-facing SLB instance. The IP address must be within the destination CIDR block of the vSwitch. Only IPv4 addresses are supported. This parameter must be used together with `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id`. An IP address cannot be changed after creation.	v2.7.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: "${YOUR_VSWITCH_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip: "192.168.x.x"
  name: nginx
  namespace: default
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 80
      name: http
  selector:
    app: nginx

Add additional tags to an SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags

Description	Supported CCM versions
A list of tags to add. Separate multiple tags with commas (,), for example, `"k1=v1,k2=v2"`. For CCM v2.10.0 and later, you can modify the tags of created instances and reused instances. Important After you add this annotation to a service to specify additional tags, any modifications made to the tags of the corresponding SLB instance in the console may be overwritten.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags: "Key1=Value1,Key2=Value2" 
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an IPv6 SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version

Description

Supported CCM versions

The IP version of the SLB instance. The IP version cannot be changed after the instance is created. When you use this feature, the kube-proxy mode of the cluster must be IPVS. Values:

ipv4: IPv4.
ipv6: IPv6. The generated IPv6 address can be accessed only in an IPv6 environment.

Default value: ipv4

v1.9.3.220-g24b1885-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: "ipv6"
  name: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Enable deletion protection for an SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-delete-protection

Description

Supported CCM versions

The deletion protection feature of the SLB instance. Values:

on
off

Default value: on

Important

For an SLB instance created for a LoadBalancer service, even if you manually enable deletion protection in the CLB console, the SLB instance associated with the service can still be deleted by running the kubectl delete svc {your-svc-name} command.

v1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-delete-protection: "on"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Enable configuration read-only mode for an SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-modification-protection

Description

Supported CCM versions

The configuration read-only mode of the SLB instance. Values:

ConsoleProtection
NonProtection

Default value: ConsoleProtection

v1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-modification-protection: "ConsoleProtection"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Specify the name of an SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name

Description	Supported CCM versions
The name of the SLB instance. The name must be 2 to 128 characters in length, start with a letter or a Chinese character, and can contain digits, periods (.), underscores (_), and hyphens (-).	v1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name: "your-svc-name"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Specify the resource group of an SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id

Description	Supported CCM versions
The ID of the resource group to which the SLB instance belongs. The resource group ID cannot be changed after it is specified. You can query the resource group ID in the Alibaba Cloud Resource Management platform.	v1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id: "rg-xxxx"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Set a hostname for a service

Annotation: Multiple annotations are available, as described in the following table.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Separate multiple values with commas (,), for example, https:443,http:80.

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname

Sets a hostname for the service. The hostname must follow DNS naming conventions.

Note the following items:

After you add this annotation, the EXTERNAL-IP of the service changes from the default CLB IP address to your_service_name. When you access the CLB IP address from within the cluster, traffic is routed through the CLB instance before being forwarded to the cluster.
After you add this annotation, if the listener uses the TCP or UDP protocol, a loopback access issue occurs when you access the CLB IP address from within the cluster. For more information, see A client fails to access a CLB instance.
Adding this annotation does not automatically bind the CLB instance to a domain name. To bind them, go to the domain name service page to purchase a domain name and bind it to the CLB instance. To purchase a domain name, see Purchase a domain name.

v2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname: "${your_service_hostname}"
  name: nginx-svc
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Expected output:

NAME         TYPE           CLUSTER-IP       EXTERNAL-IP            PORT(S)                      AGE
nginx-svc    loadBalancer   47.100.XX.XX     www.example.com        80:30248/TCP,443:32670/TCP   10s

Create a pay-by-usage SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type

Description

Supported CCM versions

The billing method of the instance. Values:

PayBySpec: The default value. pay-by-specification.
PayByCLCU: pay-by-usage.

Default value: PayBySpec

Important

For Cloud Controller Manager v2.5.0 and later, the default value is changed to PayByCLCU.
Pay-by-usage SLB instances do not support specifications. Therefore, the value PayByCLCU cannot be used together with the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec annotation.

v2.4.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type: "PayByCLCU"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Retain an auto-created SLB instance when you delete a service

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-lb-on-delete

Description

Supported CCM versions

When you delete a LoadBalancer service, this feature retains the CLB instance created by the service and removes the kubernetes.do.not.delete and ack.aliyun.com tags from the CLB instance. The existing servers in the vServer group are retained.

When this feature is enabled, a Warning event of the PreservedOnDelete type is generated during service synchronization. After you configure this annotation, we recommend that you check whether this event exists to confirm that the feature is enabled.

Values:

Not empty: The retention feature is enabled.
Empty or not set: The retention feature is not enabled.

Important

Perform this operation by deleting the service instead of changing the service type. Otherwise, the service may be incorrectly re-associated with the previously retained CLB instance.

v2.10.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-lb-on-delete: "true"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Common listener operations

Configure the session persistence timeout for a TCP listener

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-persistence-timeout

Description

Supported CCM versions

The session persistence timeout. This applies only to TCP listeners. If multiple TCP listener ports are configured for the SLB instance, this configuration applies to all TCP listener ports by default.

Unit: seconds. Value range: [0, 3600]. Default value: 0, which indicates that session persistence is disabled. For more information, see CreateLoadBalancerTCPListener.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-persistence-timeout: "1800"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Configure session persistence (insert cookie) for HTTP and HTTPS listeners

Annotation: Multiple annotations are available, as described in the following table. When you insert a cookie, the following four annotations are required.

Support is available only for SLB instances that use the HTTP and HTTPS protocols.
If multiple HTTP or HTTPS listener ports are configured, this session persistence configuration applies to all HTTP and HTTPS listener ports by default.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session	Specifies whether to enable session persistence. This takes effect only for HTTP and HTTPS listeners. Values: `on` `off` Default value: `off` For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type	The method used to process cookies. This takes effect only for HTTP and HTTPS listeners. This parameter is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session` is set to `on`. Values: `insert`: Inserts a cookie. `server`: Rewrites a cookie. For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie-timeout	The cookie timeout period. This parameter is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session` is set to `on` and `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type` is set to `insert`. Unit: seconds. Value range: [1, 86400]. For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie	The name of the cookie configured on the server. The name must be 1 to 200 characters in length and can contain only ASCII letters and digits. It cannot contain commas (,), semicolons (;), or spaces. It cannot start with a dollar sign ($). This parameter is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session` is set to `on` and `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type` is set to `server`. For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type: "insert"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie-timeout: "1800"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure an access control policy group for an SLB instance

Annotation: Multiple annotations are available, as described in the following table. The following three annotations are required.

Before you use this annotation to configure access control for an SLB instance, you must first create an access control policy group in the Classic Load Balancer (CLB) console and record its ID (acl-id).

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status	Specifies whether to enable the access control feature. Values: `on` `off` Default value: `off`	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-id	The ID of the access control policy group bound to the listener. This parameter is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status` is set to `"on"`.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-type	The access control type. Values: `white`: a whitelist. Only requests from the IP addresses or CIDR blocks specified in the selected access control policy group are forwarded. Whitelists are suitable for scenarios where applications allow access only from specific IP addresses. Setting a whitelist may pose business risks. After a whitelist is set, only IP addresses in the whitelist can access the SLB listener. If you enable a whitelist but do not add any IP addresses to the access control policy group, the SLB listener forwards all requests. `black`: a blacklist. All requests from the IP addresses or CIDR blocks specified in the selected access control policy group are not forwarded. Blacklists are suitable for scenarios where applications restrict access from specific IP addresses. If you enable a blacklist but do not add any IP addresses to the access control policy group, the SLB listener forwards all requests. This parameter is required when the AclStatus parameter is set to on.	v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-id: "${YOUR_ACL_ID}" # Multiple policy groups are not supported.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-type: "white"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify a redirection port for an SLB instance

Port forwarding refers to forwarding requests from an HTTP port to an HTTPS port.

Annotation: Multiple annotations are available, as described in the following table. The following three annotations are required.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of the certificate on Alibaba Cloud. Log on to the CLB console and view the certificate ID on the Certificate Management page. Note To create a certificate, see Select an Alibaba Cloud-issued certificate.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-forward-port	Forwards HTTP requests to a specified HTTPS port, for example, `80:443`.	v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443,http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-forward-port: "80:443"
  name: nginx
  namespace: default
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Set the scheduling algorithm for an SLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler

Description

Supported CCM versions

The scheduling algorithm. Values:

wrr: weighted round-robin. Backend servers with higher weights receive more requests.
rr: round-robin. External requests are sequentially distributed to backend servers.
sch: source IP hash. Requests from the same source IP address are scheduled to the same backend server.
tch: four-tuple hash. Requests from the same stream (identified by source IP, destination IP, source port, and destination port) are scheduled to the same backend server.

Default value: rr.

For more information about the values of this parameter, see the Scheduler field in the API documentation for creating the corresponding listener type, such as Create a TCP listener.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler: "wrr"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Create a UDP listener

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: UDP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an HTTP listener

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Health check: The health check feature is disabled for HTTP listeners by default. To ensure the availability of backend services, configure an HTTP or TCP health check for the listener port.
Access from within the cluster: When you access a service from within the cluster using its CLB IP address, the traffic may be intercepted within the cluster, which can cause the access to fail. See Set a hostname for a Service to add an annotation that forces traffic to bypass the CLB. For more information, see Considerations for accessing the external IP of a LoadBalancer Service from within a cluster.

Description	Supported CCM versions
Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an HTTPS listener

Annotation: Multiple annotations are available, as described in the following table.

HTTPS requests are decrypted at the CLB layer and then sent to backend pods as HTTP requests.

Health check: The health check feature is disabled for HTTPS listeners by default. To ensure the availability of backend services, configure an HTTP or TCP health check for the listener port.
Access from within the cluster: When you access a service from within the cluster using its CLB IP address, the traffic may be intercepted within the cluster, which can cause the access to fail. See Set a hostname for a Service to add an annotation that forces traffic to bypass the CLB. For more information, see Considerations for accessing the external IP of a LoadBalancer Service from within a cluster.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Separate multiple values with commas (,), for example, https:443,http:80.

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id

The ID of the certificate on Alibaba Cloud.

Log on to the CLB console and view the certificate ID on the Certificate Management page.

Note

To create a certificate, see Select an Alibaba Cloud-issued certificate.

v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

SSL protocol error occurs when accessing the CLB instance from within the cluster

Symptoms

After you create an HTTPS listener for a service, you can access the service from outside the cluster. However, an error is returned when you use curl to access the HTTPS port of the CLB instance from a node or pod within the cluster:

curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Cause

This issue is caused by the IPVS rules on the node:

Service configuration: Because the HTTPS listener of the CLB instance supports only the HTTP protocol on its backend services, the service can only forward traffic from port: 443 to targetPort: 80. As a result, ACK creates an IPVS rule on the node to directly forward traffic destined for port 443 to the backend port 80.
Layer 4 forwarding: IPVS operates at Layer 4 of the TCP/IP protocol stack. It only forwards TCP data packets and does not parse application-layer protocols such as TLS or HTTPS.
Protocol mismatch: The HTTPS request (TLS handshake data) initiated by the client (curl) is directly forwarded by IPVS to the backend service's HTTP port 80. Because this port is not configured for TLS, it cannot parse the TLS request and returns an HTTP 400 error. The client then reports an SSL protocol error.

Solution

Add the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname annotation to the service. This annotation prevents IPVS rules from being generated on the node and forces intra-cluster traffic to be routed through the CLB instance. This ensures that the CLB instance correctly handles TLS. For more information, see .

Configure additional domain name certificates for an HTTPS listener

Annotation: Multiple annotations are available, as described in the following table.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of the certificate on Alibaba Cloud. Log on to the CLB console and view the certificate ID on the Certificate Management page. Note To create a certificate, see Select an Alibaba Cloud-issued certificate.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-domain-extensions	Specifies the additional domain name and certificate ID in the `{domain name}:{certificate ID}` format. Separate multiple values with commas (,). Log on to the CLB console and view the certificate ID on the Certificate Management page.	v2.13.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-domain-extensions: "${DOMAIN_1}:${CERT_ID_1},${DOMAIN_2}:${CERT_ID_2}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create a listener with a health check

Set up a TCP health check

Annotation: Multiple annotations are available, as described in the following table. All of the following annotations are required.

The health check feature is enabled for TCP ports by default.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch	Specifies whether to enable health checks for TCP and UDP listeners. Values: `on` `off` Default value: `on`	v2.6.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type	The health check type. Values: `tcp` `http` Default value: `tcp`. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout	The timeout period for a health check response. This applies to the TCP mode. If a backend ECS instance does not respond within the specified timeout period, the health check fails. Unit: seconds. Value range: [1, 300]. If the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout` is smaller than the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval`, the former is invalid and the timeout period is the value of the latter. Default value: 5. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks required to change the health check status of a backend server from `fail` to `success`. Value range: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks required to change the health check status of a backend server from success to fail. Value range: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval between two consecutive health checks. Unit: seconds. Value range: [1, 50]. Default value: 2. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "tcp"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "8"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "3"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Set up a UDP health check

Annotation: Multiple annotations are available, as described in the following table. All of the following annotations are required.

The health check feature is enabled for UDP ports by default.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch	Specifies whether to enable health checks for TCP and UDP listeners. Values: `on` `off` Default value: `on`	v2.6.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout	The timeout period for a health check response. This applies to the TCP mode. If a backend ECS instance does not respond within the specified timeout period, the health check fails. Unit: seconds. Value range: [1, 300]. If the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout` is smaller than the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval`, the former is invalid and the timeout period is the value of the latter. Default value: 5. For more information, see CreateLoadBalancerUDPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks required to change the health check status of a backend server from fail to success. Value range: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerUDPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks required to change the health check status of a backend server from success to fail. Value range: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerUDPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval between two consecutive health checks. Unit: seconds. Value range: [1, 50]. Default value: 2. For more information, see CreateLoadBalancerUDPListener.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "5"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "10"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "3"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "3"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: UDP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Disable health checks for TCP and UDP listeners

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch

Description

Supported CCM versions

Specifies whether to enable health checks for TCP and UDP listeners. Values:

on
off

Default value: on

v2.6.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "off" # Disable the health check
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Set up an HTTP health check

Annotation: Multiple annotations are available, as described in the following table.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag	Values: `on`: The default value for TCP listeners is `on` and cannot be changed. You do not need to change this annotation for TCP. `off`: The default value for HTTP listeners is `off`. Default value: `off`	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type	The health check type. Values: `tcp` `http` Default value: `tcp` For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri	The URI for health checks. You do not need to configure this annotation when the health check type is TCP.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-httpcode	The normal HTTP status codes for a health check. Separate multiple status codes with commas (,). Values: `http_2xx` `http_3xx` `http_4xx` `http_5xx` Default value: `http_2xx` For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain	The domain name for health checks. Values: `$_ip`: The private IP address of the backend server. If you specify an IP address or do not specify this parameter, the SLB instance uses the private IP address of each backend server as the domain name for health checks. `domain`: The domain name must be 1 to 80 characters in length and can contain letters, digits, periods (.), and hyphens (-). For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout	The timeout period for a health check response. This applies to the HTTP mode. If a backend ECS instance does not respond within the specified timeout period, the health check fails. Unit: seconds. Value range: [1, 300]. If the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout` is smaller than the value of service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval, `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout` is invalid and the timeout period is the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval`. For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks required to change the health check status of a backend server from fail to success. Value range: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks required to change the health check status of a backend server from success to fail. Value range: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval between two consecutive health checks. Unit: seconds. Value range: [1, 50]. Default value: 2. For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method	The health check method for an HTTP listener. Values: `head` `get`	v2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "http"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri: "/test/index.html"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout: "10"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "3"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    # Set the health check HTTP status code. This is optional.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-httpcode: "http_4xx"
    # Set the health check domain name. This is optional.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain: "www.aliyun.com"
    # Set the health check method. This is optional.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method: "head"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Enable connection draining for a listener

Annotation: Multiple annotations are available, as described in the following table. All of the following annotations are required.

This feature is supported only for the TCP and UDP protocols.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain

Specifies whether to enable connection draining. Values:

on
off

v2.0.1 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout

Sets the connection draining timeout. Unit: seconds. Value range: [10, 900].

v2.0.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: "30"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure additional request headers for a listener

Annotation: Multiple annotations are available, as described in the following table.

This feature is supported only for the HTTP and HTTPS protocols.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-proto	Specifies whether to use the X-Forwarded-Proto header to retrieve the listener protocol of the CLB instance. Values: `on` `off` Default value: `off`	v2.1.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-slbport	Specifies whether to use the X-Forwarded-For-SLBPORT header to retrieve the listener port of the SLB instance. Values: `on` `off` Default value: `off`	v2.9.1 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-clientsrcport	Specifies whether to use the X-Forwarded-For-ClientSrcPort header to retrieve the port of the client that accesses the SLB instance. Values: `on` `off` Default value: `off`	v2.9.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-proto: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-slbport: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-clientsrcport: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Set the idle connection timeout for a listener

Annotation: Multiple annotations are available, as described in the following table.

This feature is supported only for the HTTP and HTTPS protocols.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Separate multiple values with commas (,), for example, https:443,http:80.

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout

Sets the idle connection timeout for the listener. Unit: seconds. Value range: [1, 60].

Default value: 15

v2.1.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout: "30"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Disable the HTTP/2 feature for a listener

Annotation: Multiple annotations are available, as described in the following table.

This feature is supported only for the HTTPS protocol.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of the certificate on Alibaba Cloud. Log on to the CLB console and view the certificate ID on the Certificate Management page. Note To create a certificate, see Select an Alibaba Cloud-issued certificate.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-http2-enabled	Specifies whether to enable the HTTP/2 feature. Values: `on` `off` Default value: `on`	v2.1.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-http2-enabled: "off"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Specify the request timeout for a listener

Annotation: Multiple annotations are available, as described in the following table.

This feature is supported only for the HTTP and HTTPS protocols.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Separate multiple values with commas (,), for example, https:443,http:80.

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-request-timeout

Specifies the request timeout period. Unit: seconds. Value range: [1, 180]. Default value: 60

If the backend server does not respond within the timeout period, the SLB instance stops waiting and returns an HTTP 504 error to the client.

v2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-request-timeout: "60"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Specify the connection timeout for a listener

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-established-timeout

This feature is supported only for the TCP protocol.

Description	Supported CCM versions
The connection timeout period. Unit: seconds. Value range: [10, 900]. For more information, see CreateLoadBalancerTCPListener.	v2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-established-timeout: "60"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure a security policy for a listener

Annotation: Multiple annotations are available, as described in the following table.

This feature is supported only for the HTTPS protocol.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,), for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of the certificate on Alibaba Cloud. Log on to the CLB console and view the certificate ID on the Certificate Management page. Note To create a certificate, see Select an Alibaba Cloud-issued certificate.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy	A security policy includes the optional TLS versions and the supported cipher suites for HTTPS. For more information, see CreateLoadBalancerHTTPSListener. Values: `tls_cipher_policy_1_0` `tls_cipher_policy_1_1` `tls_cipher_policy_1_2` `tls_cipher_policy_1_2_strict` `tls_cipher_policy_1_2_strict_with_1_3` Default value: `tls_cipher_policy_1_0`	v2.4.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443,http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy: "tls_cipher_policy_1_2"
  name: nginx
  namespace: default
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure TCP and UDP protocols for a listener at the same time

Note

This feature requires a Kubernetes cluster that runs v1.24 or later. For more information, see Upgrade an ACK cluster.

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: udp
    port: 80
    protocol: UDP
    targetPort: 81
  selector:
    app: nginx
  sessionAffinity: None
  type: LoadBalancer

Configure the Proxy Protocol for TCP and UDP listeners

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol

When you access a service from within the cluster using the CLB IP address, traffic may be intercepted inside the cluster, which can cause access exceptions. To resolve this, see Set a hostname for a service to add an annotation that forces traffic to be routed through the CLB instance. For more information, see Notes on accessing the external IP address of a LoadBalancer service from within a cluster.

Description

Supported CCM versions

Configures the Proxy Protocol for TCP and UDP listeners. After the Proxy Protocol is configured, it can be used to carry the source IP address of the client to the backend server. Values:

on
off

Default value: off

Important

This feature does not support smooth online migration. Switching the Proxy Protocol requires service downtime for the upgrade. Configure this with caution.

v2.6.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Typical operations for backend server groups

Use worker nodes with a specific label as backend servers

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-backend-label

Description	Supported CCM versions
Specifies the worker nodes to attach to the backend of the CLB instance based on labels. Separate multiple labels with commas (,), for example, `"k1=v1,k2=v2"`. The AND operator is used between multiple labels.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-backend-label: "failure-domain.beta.kubernetes.io/zone=ap-southeast-5a"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use the nodes where pods reside as backend servers

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler

By default, externalTrafficPolicy is set to Cluster. In this mode, all nodes in the cluster are attached to the backend server group. In Local mode, only the nodes where the pods reside are used as backend servers.

Description

Supported CCM versions

The scheduling algorithm. Values:

wrr: weighted round-robin. Backend servers with higher weights receive more requests.
In Local mode, you must set the scheduling algorithm to weighted round-robin (wrr).
Note
For CCM v1.9.3.164-g2105d2e-aliyun and later, if the external traffic policy is set to Local mode, the service automatically sets the weight of a node based on the number of pods on the node. For more information about how the weight is calculated, see How are node weights automatically set in Local mode?.
rr: round-robin. External requests are sequentially distributed to backend servers.

Default value: rr

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler: "wrr"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Remove unschedulable nodes from the backend server group of a CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-remove-unscheduled-backend

Description

Supported CCM versions

Removes SchedulingDisabled nodes from the backend of the CLB instance. Values:

on: Removes nodes in the unschedulable state from the backend server group of the CLB instance.
off: The kubectl cordon and kubectl drain commands set nodes to the unschedulable state. Nodes in the unschedulable state are not removed from the backend server group of the CLB instance.

Default value: off

v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-remove-unscheduled-backend: "on"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 30080
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Directly attach pod ENIs to the backend of a CLB instance

Annotation: service.beta.kubernetes.io/backend-type

Description

Default value

Supported CCM versions

The type of backend server for the CLB instance. Values:

eni: Attaches pods to the backend of the CLB instance. This improves network forwarding performance. This value is valid only in Terway network mode.
You can also manually change the value of service.beta.kubernetes.io/backend-type: "eni" from eni to ecs to attach ECS instances to the backend of the CLB instance.
ecs: Attaches ECS instances to the backend of the CLB instance.

Flannel network mode: ecs
Terway network mode:
- Terway clusters created before August 10, 2020: ecs
- Terway clusters created after August 10, 2020: eni

v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/backend-type: "eni"
  name: nginx
spec:
  ports:
  - name: http
    port: 30080
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Reuse an existing vServer group

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vgroup-port

You can reuse an existing vServer group. This is applicable only when you reuse an existing CLB instance. For an example, see Deploy services across clusters by reusing an SLB instance.

Set the weight of a service to receive traffic

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-weight

If multiple services reuse the same CLB instance, you can use this annotation to set the weight for the current service. This annotation is applicable only when you reuse an existing vServer group. For an example, see Deploy services across clusters by reusing an SLB instance.

Ignore backend server weight updates

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ignore-weight-update

Description

Supported CCM versions

During service synchronization, this skips updating the weights of backend servers in the vServer group. This configuration is suitable for scenarios where you need to manually manage backend server weights using a mechanism other than CCM. Values:

on
off

Default value: off

v2.11.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ignore-weight-update: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer