Use annotations to configure CLB instances in ACK clusters - Container Service for Kubernetes

To use various load balancing features, you can add annotations to the YAML file of a Service to configure Classic Load Balancer (CLB) instances, listeners, and backend server groups.

Category	Hyperlink
Annotation usage notes
Common operations to manage CLB instances	Create an Internet-facing CLB instance Create an internal-facing CLB instance Create a CLB instance that uses the specified specifications Use an existing CLB instance Use an existing CLB instance and forcefully overwrite the listeners of the CLB instance Specify the primary and secondary zones when you create a CLB instance Create a pay-by-bandwidth CLB instance Specify a vSwitch for a CLB instance Specify an IP address for an internal-facing CLB instance Add additional tags to a CLB instance Create an IPv6 CLB instance Enable deletion protection for a CLB instance Enable the configuration read-only mode for a CLB instance Specify the name of a CLB instance Specify the resource group to which a CLB instance belongs Specify a hostname for a Service Create a pay-by-LCU CLB instance
Common operations to manage listeners	Specify the session persistence period for a TCP-based CLB instance Enable session persistence for an HTTP or HTTPS listener by inserting a cookie Configure an ACL for a CLB instance Configure port forwarding for a CLB instance Specify the scheduling algorithm for a CLB instance Create a UDP listener Create an HTTP listener Create an HTTPS listener Create a listener that enables health checks Configure connection draining for a listener Configure additional request headers for a listener Specify the timeout period of idle connections for a listener Disable HTTP/2 for a listener Specify the timeout period of requests for a listener Specify the timeout period of connections for a listener Configure security policies for a listener Configure both TCP and UDP for a listener Configure proxy protocol for TCP and UDP listeners
Common operations to manage backend server groups	Add worker nodes that have specified labels as the backend servers of a CLB instance Add the nodes on which pods run as the backend servers of a CLB instance Remove unschedulable nodes from the backend server group of a CLB instance Add pods that are assigned ENIs as the backend servers of a CLB instance Reuse an existing vServer group Specify weights for Services to enable weighted round robin
References

Annotation usage notes

The values of annotations are case-sensitive.
Check the versions of the cloud controller manager (CCM) component supported by this feature before you use the annotations. For more information about how to update the CCM component, see Manage components. For more information about the release notes for the CCM component, see Cloud Controller Manager.
Starting from September 11, 2019, the keyword alicloud in annotations is changed to alibaba-cloud.
Example:
Before the change: service.beta.kubernetes.io/alicloud-loadbalancer-id
After the change: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id
The system still supports annotations that use the alicloud keyword. You do not need to modify the existing annotations.

Common operations to manage CLB instances

Create an Internet-facing CLB instance

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an internal-facing CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type

Description

Default value

Supported CCM version

Creates an internal-facing CLB instance. Valid values:

internet: uses an Internet-facing CLB instance to expose the Service. This is the default value. The address type of the CLB instance must be set to Internet.
intranet: uses an internal-facing CLB instance to expose the Service. The address type of the CLB instance must be set to intranet.

internet

CCM 1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create a CLB instance that uses the specified specifications

The following annotations are used.

Annotation

Description

Default value

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type

The billing method. Valid values:

PayBySpec: The CLB instance is billed based on its specification. This is the default value.
PayByCLCU: The CLB instance is billed based on Loadbalancer Capacity Units (LCUs).

Important

You cannot change the specification of a pay-by-LCU CLB instance. You cannot specify PayByCLCU and add the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec annotation at the same time.

PayBySpec

CCM 2.4.0 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec

The specifications of the CLB instance. You can use this annotation to create a CLB instance with specified specifications, or update the specifications of an existing CLB instance. Valid values:

slb.s1.small
slb.s2.small
slb.s2.medium
slb.s3.small
slb.s3.medium
slb.s3.large

For more information, see CreateLoadBalancer.

Important

If you use the CLB console to update the specifications of a CLB instance, the CCM may restore the CLB instance to the original specifications. Proceed with caution. You can use the CLB console to modify the specifications of only CLB instances that use the pay-by-specification billing method.

slb.s1.small

CCM 1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type: "PayBySpec"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec: "slb.s1.small"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use an existing CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id

Description

Default value

Supported CCM version

The ID of the CLB instance that you want to use. You can use this annotation to specify an existing CLB instance.

By default, if you use an existing CLB instance, the listeners of the CLB instance are not overwritten. To overwrite the existing listeners, set the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners to annotation to "true". This configuration only overwrites the listeners of the current Service and does not affect other listeners.
Note
The CCM does not overwrite the listeners of an existing CLB instance due to the following reasons:
- If the listeners of the CLB instance are associated with applications, service interruptions may occur after the listeners are overwritten.
- The CCM supports limited backend configurations and cannot handle complex configurations. To use complex backend configurations, you can create listeners in the console. The listeners do not overwrite the existing ones.
Therefore, we recommend that you do not overwrite the listeners of an existing CLB instance. You can forcibly overwrite the listeners if the ports on which these listeners listen are no longer used.
(Earlier than V2.10.0) You cannot add additional tags such as annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags if you use an existing CLB instance.

No default value

CCM 1.9.3.81-gca19cd4-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: "${YOUR_LOADBALANCER_ID}" # You can configure only one access control list (ACL). 
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use an existing CLB instance and forcefully overwrite the listeners of the CLB instance

The following annotations are used. Use the following annotations to forcefully overwrite the listeners of a CLB instance. If a listener port conflict occurs, the original listener is deleted.

Annotation

Description

Default value

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id

The ID of the CLB instance that you want to use. You can use this annotation to specify an existing CLB instance.

By default, if you use an existing CLB instance, the listeners of the CLB instance are not overwritten. To overwrite the existing listeners, set the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners annotation to "true". This configuration only overwrites the listeners of the current Service and does not affect other listeners.
Note
The CCM does not overwrite the listeners of an existing CLB instance due to the following reasons:
- If the listeners of the CLB instance are associated with applications, service interruptions may occur after the listeners are overwritten.
- The CCM supports limited backend configurations and cannot handle complex configurations. To use complex backend configurations, you can create listeners in the console. The listeners do not overwrite the existing ones.
Therefore, we recommend that you do not overwrite the listeners of an existing CLB instance. You can forcibly overwrite the listeners if the ports on which these listeners listen are no longer used.
(Earlier than V2.10.0) You cannot add additional tags such as annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags if you use an existing CLB instance.

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners

Specifies whether to overwrite the listeners of an existing CLB instance. Valid values:

"true": overwrites the listeners. This configuration only overwrites the listeners of the current Service and does not affect other listeners.
"false": does not overwrite the listeners.

Important

If you use an existing CLB instance and set the force-override annotation to "true", do not associate a listener of the CLB instance with multiple Services. Otherwise, a configuration conflict occurs when these Services overwrite the listener.

"false"

CCM 1.9.3.81-gca19cd4-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: "${YOUR_LOADBALANCER_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: "true"
    
    
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify the primary and secondary zones when you create a CLB instance

The following annotations are used. After you specify the primary and secondary zones for a CLB instance, the zones cannot be changed.

CLB instances in some regions do not support the primary and secondary zones. The actual information on the CLB instance creation page shall prevail.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-master-zoneid	The ID of the zone in which the primary backend server is deployed.	No default value	CCM 1.9.3.10-gfb99107-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-slave-zoneid	The ID of the zone in which the secondary backend server is deployed.	No default value	CCM 1.9.3.10-gfb99107-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-master-zoneid: "cn-hangzhou-k"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-slave-zoneid: "cn-hangzhou-j"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create a pay-by-bandwidth CLB instance

The following annotations are used. Both annotations are required.

Annotation

Description

Default value

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-charge-type

The billing method of the CLB instance. Valid values:

paybytraffic
paybybandwidth

paybytraffic

CCM 1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth

The maximum bandwidth of the CLB instance. This annotation applies only to Internet-facing CLB instances. For more information about the limits on the billing methods for Internet-facing CLB instances, see ModifyLoadBalancerInstanceSpec.

CCM 1.9.3.10-gfb99107-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-charge-type: "paybybandwidth"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth: "2"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify a vSwitch for a CLB instance

The following annotations are used. Both annotations are required.

Annotation

Description

Default value

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type

To create an internal-facing CLB instance, set the value to intranet. Valid values:

internet: uses an Internet-facing CLB instance to expose the Service. This is the default value. The address type of the CLB instance must be set to Internet.
intranet: uses an internal-facing CLB instance to expose the Service. The address type of the CLB instance must be set to intranet.

internet

CCM 1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id

The ID of the vSwitch in which the CLB instance resides. The vSwitch and the cluster must be deployed in the same virtual private cloud (VPC).

When you use this annotation, you must set the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type annotation to "intranet".

You can log on to the VPC console to query vSwitch IDs.

No default value

CCM 1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
   service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
   service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: "${YOUR_VSWITCH_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify an IP address for an internal-facing CLB instance

The following annotations are used. These annotations are required.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type	To create an internal-facing CLB instance, set the value to intranet. Valid values: `internet`: uses an Internet-facing CLB instance to expose the Service. This is the default value. The address type of the CLB instance must be set to Internet. `intranet`: uses an internal-facing CLB instance to expose the Service. The address type of the CLB instance must be set to intranet.	internet	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id	The ID of the vSwitch in which the CLB instance resides. The vSwitch and the cluster must be deployed in the same VPC. When you use this annotation, you must set the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type` annotation to intranet. You can log on to the VPC console to query vSwitch IDs.	No default value	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip	The IP address of the internal-facing CLB instance. The IP address must fall within the CIDR block of the vSwitch. Only IPv4 addresses are supported. This annotation must be used together with the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id` annotation. You cannot change the IP address after it is specified.	No default value	CCM 2.7.0 and later

apiVersion: v1
kind: Service
metadata:
 annotations:
  service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
  service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: "${YOUR_VSWITCH_ID}"
  service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip: "192.168.x.x"
 name: nginx
 namespace: default
spec:
 type: LoadBalancer
 ports:
 - port: 80
   targetPort: 80
   name: http
 selector:
   app: nginx

Add additional tags to a CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags

Description	Default value	Supported CCM version
The tags that you want to add to the CLB instance. Separate multiple tags with commas (,). Example: `"k1=v1,k2=v2"`. CCM 2.10.0 and later versions allow you to modify the tags of existing or reused instances. Important After this annotation is added to a Service to specify additional tags, the operations performed on the CLB instance tags in the console may be overwritten.	No default value	CCM 1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags: "Key1=Value1,Key2=Value2" 
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an IPv6 CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version

Description

Default value

Supported CCM version

The IP version of the CLB instance. You cannot change the IP version cannot be changed after it is specified. To use this annotation, make sure that the kube-proxy mode of the cluster is set to IPVS. Valid values:

ipv4
ipv6 The IPv6 address can be used only in an IPv6 network.

ipv4

CCM 1.9.3.220-g24b1885-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: "ipv6"
  name: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Enable deletion protection for a CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-delete-protection

Description

Default value

Supported CCM version

Specifies whether to enable deletion protection for the CLB instance. Valid values:

on
off

Important

If you manually enable deletion protection in the CLB console for a CLB instance that is created for a LoadBalancer Service, you can run the kubectl delete svc {your-svc-name} command to delete the CLB instance.

CCM 1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-delete-protection: "on"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Enable the configuration read-only mode for a CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-modification-protection

Description

Default value

Supported CCM version

Specifies whether to enable the configuration read-only mode for the CLB instance. Valid values:

ConsoleProtection
NonProtection

ConsoleProtection

CCM 1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-modification-protection: "ConsoleProtection"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Specify the name of a CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name

Description	Default value	Supported CCM version
The name of the CLB instance. The name must be 2 to 128 characters in length, and can contain letters, digits, periods (.), underscores (_), and hyphens (-). The name must start with a letter.	No default value	CCM 1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name: "your-svc-name"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Specify a resource group to which a CLB instance belongs

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id

Description	Default value	Supported CCM version
The ID of the resource group to which the CLB instance belongs. The resource group cannot be changed after it is specified. You can log on to the Resource Management console to query resource group IDs.	No default value	CCM 1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id: "rg-xxxx"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Specify a hostname for a Service

The following annotations are used.

Annotation

Description

Default value

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Separate multiple values with commas (,). Example: https:443,http:80.

No default value

CCM 1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname

Specifies a hostname for the Service. The hostname must comply with the naming conventions of domain names.

Take note of the following items:

After you add this annotation, the external IP address of the Service changes from the default CLB IP address to your_service_name. If you access the CLB IP address from within the cluster, requests are first forwarded to the corresponding CLB instance.
After you add this annotation, if the listener protocol is TCP or UDP, a loop occurs when requests from within the cluster are forwarded to the CLB IP address. For more information, see Why am I unable to access an SLB instance?
This annotation does not automatically associate a domain name with the CLB instance. You can log on to the Domain Names console, purchase a domain name, and then associate the domain name with the CLB instance. For more information about how to purchase domain names, see the Purchase a domain name section of the "What is Domain Names?" topic.

No default value

CCM 2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname: "${your_service_hostname}"
  name: nginx-svc
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Expected output:

NAME         TYPE           CLUSTER-IP       EXTERNAL-IP            PORT(S)                      AGE
nginx-svc    loadBalancer   47.100.XX.XX     www.example.com        80:30248/TCP,443:32670/TCP   10s

Create a pay-by-LCU CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type

Description

Default value

Supported CCM version

The billing method of the CLB instance. Valid values:

PayBySpec: The CLB instance is billed based on its specification. This is the default value.
PayByCLCU: The CLB instance is billed based on Loadbalancer Capacity Units (LCUs).

Important

You cannot change the specification of a pay-by-LCU CLB instance. You cannot specify PayByCLCU and add the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec annotation at the same time.

PayBySpec

CCM 2.4.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type: "PayByCLCU"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Retain automatically created CLB instances when you delete a Service

Annotation:service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-lb-on-delete

Description

Default value

Supported CCM version

When you delete a LoadBalancer Service, the CLB instances created by using the Service are retained, and thekubernetes.do.not.deleteandack.aliyun.comtags of the CLB instances and server groups are removed. Existing servers in the server group are retained.

If this feature is enabled, a Warning event of thePreservedOnDeletetype is generated during the synchronization of the Service. After you configure the annotation, we recommend that you check whether the event exists to confirm that this feature is enabled.

Valid values:

Not empty: enables the retention feature.
If the value is empty or not set, the retention feature is not enabled.

Important

Perform this operation by deleting the Service instead of modifying the Service type. Otherwise, the Service may be incorrectly re-associated with the retained CLB instances.

No impact on workloads

CCM 2.10.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-lb-on-delete: "true"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Common operations to manage listeners

Specify the session persistence period for a TCP-based CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-persistence-timeout

Description

Default value

Supported CCM version

The session persistence period. Unit: seconds. This annotation applies only to TCP listeners. If a CLB instance has multiple TCP listeners, the configuration applies to all TCP listeners.

Unit: seconds. Valid values: 0 to 3600. Default value: 0. By default, session persistence is disabled. For more information, see CreateLoadBalancerTCPListener.

CCM 1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-persistence-timeout: "1800"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Enable session persistence for an HTTP or HTTPS listener by inserting a cookie

The following annotations are used. The following annotations are required if you insert a cookie to enable session persistence.

The following annotations apply only to HTTP-based and HTTPS-based CLB instances.
If a CLB instance has multiple HTTP or HTTPS listeners, the configuration applies to all HTTP or HTTPS listeners.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session	Specifies whether to enable session persistence. This annotation applies only to HTTP and HTTPS listeners. Valid values: `on`: enables session persistence. `off`: does not enable session persistence. For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	off	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type	The method that is used to process the cookie. This annotation applies only to HTTP and HTTPS listeners. This annotation is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session` is set to `on`. Valid values: `insert`: inserts the cookie. `server`: rewrites the cookie. For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	No default value	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie-timeout	The timeout period of the cookie. This annotation is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session` is set to `on` and `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type` is set to `insert`. Unit: seconds. Valid values: 1 to 86400. For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	No default value	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie	The name of a cookie configured on a backend server. The cookie must be 1 to 200 characters in length, and can contain only ASCII letters and digits. It cannot contain commas (,), semicolons (;), or space characters. It cannot start with a dollar sign ($). This annotation is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session` is set to `on` and `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type` is set to `server`. For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	No default value	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,). Example: `https:443,http:80`.	No default value	CCM 1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type: "insert"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie-timeout: "1800"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure an ACL for a CLB instance

The following annotations are used. These annotations are required.

Before you use this annotation to create a CLB instance that is configured with access control, you need to create an ACL in the CLB console and record the ID of the ACL.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status	Specifies whether to enable access control. Valid values: `on`: enables access control. `off`: does not enable access control.	off	CCM 1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-id	The ID of the ACL that you want to associate with a listener. This annotation is required when the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status` annotation is set to `"on"`.	No default value	CCM 1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-type	The type of the ACL. Valid values: `white`: a whitelist. Only requests from the IP addresses or CIDR blocks in the ACL are forwarded. You can use an IP address whitelist in scenarios in which you want to allow access from only specified IP addresses to an application. Risks may arise if a whitelist is improperly configured. After a whitelist is set, the CLB instance forwards only requests from the IP addresses in the whitelist. If you enable a whitelist but do not add an IP address to the ACL, all requests are forwarded. `black`: a blacklist. All requests from the IP addresses or CIDR blocks in the ACL are not forwarded. You can use an IP address blacklist in scenarios in which you want to deny access from only specified IP addresses to an application. If you enable an IP address blacklist but do not add an IP address to the blacklist, all requests are forwarded. This annotation is required when the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status annotation is set to on.	No default value	CCM 1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-id: "${YOUR_ACL_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-type: "white"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Configure port forwarding for a CLB instance

Port forwarding allows a CLB instance to forward requests from an HTTP port to an HTTPS port.

The following annotations are used. These annotations are required.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,). Example: `https:443,http:80`.	No default value	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of a certificate for the CLB instance. You can log on to the CLB console and view certificate IDs on the Certificates page. Note For more information about how to create a certificate, see Use a certificate from Alibaba Cloud SSL Certificates Service.	No default value	CCM 1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-forward-port	Forwards requests from an HTTP port to the specified HTTPS port. Example: `80:443`.	No default value	CCM 1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443,http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-forward-port: "80:443"
  name: nginx
  namespace: default
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Specify the scheduling algorithm for a CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler

Description

Default value

Supported CCM version

The scheduling algorithm. Valid values:

wrr: the weighted round-robin algorithm. Backend servers with higher weights receive more requests than backend servers with lower weights.
rr: the round-robin algorithm. Requests are distributed to backend servers in sequence.
sch: consistent hashing based on the source IP. Requests from the same source IP address are distributed to the same backend server.
tch: consistent hashing based on four-tuple (source IP + destination IP + source port + destination port). Flows that have the same four-tuple are scheduled to the same backend server.

For more information, see the Scheduler field in the API documentation of CLB for creating listeners of the corresponding type, such as CreateLoadBalancerTCPListener.

CCM 1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler: "wrr"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Create a UDP listener

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: UDP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an HTTP listener

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Description	Default value	Supported CCM version
Separate multiple values with commas (,). Example: `https:443,http:80`.	No default value	CCM 1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an HTTPS listener

The following annotations are used.

The HTTPS listeners of the CLB instance decrypt HTTPS requests into HTTP requests and forward the requests to pods on the backend servers.

Annotation

Description

Default value

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Separate multiple values with commas (,). Example: https:443,http:80.

No default value

CCM 1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id

The ID of a certificate for the CLB instance.

You can log on to the CLB console and view certificate IDs on the Certificates page.

Note

For more information about how to create a certificate, see Use a certificate from Alibaba Cloud SSL Certificates Service.

No default value

CCM 1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create a listener that enables health checks

Enable TCP health checks

The following annotations are used. These annotations are required.

By default, health checks are enabled for TCP ports.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch	Specifies whether to enable health checks for a TCP or UDP listener. Valid values: `on`: enables health checks. `off`: does not enable health checks.	on	CCM 2.6.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type	The type of the health check. Valid values: `tcp` `http` For more information, see CreateLoadBalancerTCPListener.	tcp	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout	The timeout period of a health check. This annotation applies to TCP health checks. If a backend server, such as an Elastic Compute Service (ECS) instance, does not return a health check response within the specified timeout period, the server fails the health check. Unit: seconds. Valid values: 1 to 300. If the value of the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout` annotation is smaller than that of the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval` annotation, the first annotation does not take effect. The timeout period is set to the value of the second annotation. For more information, see CreateLoadBalancerTCPListener.	5	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks that must occur before an unhealthy backend server is declared healthy (from `fail` to `success`). Valid values: 2 to 10. For more information, see CreateLoadBalancerTCPListener.	3	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks that must occur before a healthy backend server is declared unhealthy. Valid values: 2 to 10. For more information, see CreateLoadBalancerTCPListener.	3	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval at which health checks are performed. Unit: seconds. Valid values: 1 to 50. For more information, see CreateLoadBalancerTCPListener.	2	CCM 1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "tcp"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "8"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "3"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Enable UDP health checks

The following annotations are used. These annotations are required.

By default, health checks are enabled for UDP ports.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch	Specifies whether to enable health checks for a TCP or UDP listener. Valid values: `on` `off`	on	CCM 2.6.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout	The timeout period of a health check. This annotation applies to TCP health checks. If a backend server, such as an ECS instance, does not return a health check response within the specified timeout period, the server fails the health check. Unit: seconds. Valid values: 1 to 300. If the value of the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout` annotation is smaller than that of the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval` annotation, the first annotation does not take effect. The timeout period is set to the value of the second annotation. For more information, see CreateLoadBalancerUDPListener.	5	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks that must occur before an unhealthy backend server is declared healthy. Valid values: 2 to 10. For more information, see CreateLoadBalancerUDPListener.	3	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks that must occur before a healthy backend server is declared unhealthy. Valid values: 2 to 10. For more information, see CreateLoadBalancerUDPListener.	3	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval at which health checks are performed. Unit: seconds. Valid values: 1 to 50. For more information, see CreateLoadBalancerUDPListener.	2	CCM 1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "5"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "10"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "3"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "3"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: UDP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Disable health checks for a TCP or UDP listener

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch

Description

Default value

Supported CCM version

Specifies whether to enable health checks for a TCP or UDP listener. Valid values:

on: enables health checks.
off: does not enable health checks.

CCM 2.6.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "off" # Disable health checks.
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Enable HTTP health checks

The following annotations are used.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag	Valid values: `on`: The value is set to `on` for TCP listeners by default and cannot be changed. This annotation is not required if you use TCP listeners. `off`: The value is set to `off` for HTTP listeners by default.	off	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type	The type of the health check. Valid values: `tcp` `http` For more information, see CreateLoadBalancerHTTPListener.	tcp	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri	The Uniform Resource Identifier (URI) that you want to use for health checks. This annotation is not required when the type of the health check is set to TCP.	No default value	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-httpcode	The HTTP status code for a successful health check. Multiple HTTP status codes are separated by commas (,). Valid values: `http_2xx` `http_3xx` `http_4xx` `http_5xx` For more information, see CreateLoadBalancerHTTPListener.	http_2xx	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain	The domain name that is used for health checks. Valid values: `$_ip`: The private IP address of the backend server. If an IP address is specified, or this parameter is not specified, the CLB instance uses the private IP address of each backend server as the domain name for health checks. `domain`: The domain name must be 1 to 80 characters in length and can contain letters, digits, periods (.), and hyphens (-). For more information, see CreateLoadBalancerHTTPListener.	No default value	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout	The timeout period of a health check. This annotation applies to HTTP health checks. If a backend server, such as an ECS instance, does not return a health check response within the specified timeout period, the server fails the health check. Unit: seconds. Valid values: 1 to 300. If the value of the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout` annotation is smaller than that of the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval` annotation, the first annotation does not take effect. The timeout period is set to the value of the second annotation. For more information, see CreateLoadBalancerHTTPListener.	No default value	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks that must occur before an unhealthy backend server is declared healthy. Valid values: 2 to 10. For more information, see CreateLoadBalancerHTTPListener.	3	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks that must occur before a healthy backend server is declared unhealthy. Valid values: 2 to 10. For more information, see CreateLoadBalancerHTTPListener.	3	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval at which health checks are performed. Unit: seconds. Valid values: 1 to 50. For more information, see CreateLoadBalancerHTTPListener.	2	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,). Example: `https:443,http:80`.	No default value	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method	The health check method used by HTTP listeners. Valid values: `head` `get`	No default value	CCM 2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "http"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri: "/test/index.html"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout: "10"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "3"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    # Specify the HTTP status codes for health check. This annotation is optional.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-httpcode: "http_4xx"
    # Specify the domain name for health check. This annotation is optional.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain: "www.aliyun.com"
    # Specify the health check method. This annotation is optional.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method: "head"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure connection draining for a listener

The following annotations are used. These annotations are required.

Only TCP and UDP listeners are supported.

Annotation

Description

Default value

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain

Specifies whether to enable connection draining. Valid values:

on: enables connection draining.
off: does not enable connection draining.

No default value

CCM 2.0.1 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout

The timeout period of connection draining. Unit: seconds. Valid values: 10 to 900.

No default value

CCM 2.0.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: "30"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure additional request headers for a listener

The following annotations are used.

Only HTTP and HTTPS listeners are supported.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,). Example: `https:443,http:80`.	No default value	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-proto	Specifies whether to use the X-Forwarded-Proto header to obtain the listener protocol of the CLB instance. Valid values: `on`: uses the X-Forwarded-Proto header to obtain the listener protocol of the CLB instance. `off`: does not use the X-Forwarded-Proto header to obtain the listener protocol of the CLB instance.	off	CCM 2.1.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-slbport	Specifies whether to use the XForwardedFor_SLBPORT header to obtain the listener ports of the CLB instance. Valid values: `on`: uses the XForwardedFor_SLBPORT header to obtain the listener ports of the CLB instance. `off`: does not use the XForwardedFor_SLBPORT header to obtain the listener ports of the CLB instance.	off	CCM 2.9.1 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-clientsrcport	Specifies whether to use the XForwardedFor_ClientSrcPort header to obtain the listener ports of the CLB instance. Valid values: `on`: uses the XForwardedFor_ClientSrcPort header to obtain the listener ports of the CLB instance. `off`: does not use the XForwardedFor_ClientSrcPort header to obtain the listener ports of the CLB instance.	off	CCM 2.9.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-proto: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-slbport: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-clientsrcport: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Specify the timeout period of idle connections for a listener

The following annotations are used.

Only HTTP and HTTPS listeners are supported.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,). Example: `https:443,http:80`.	No default value	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout	Specifies the timeout period of idle connections for a listener. Unit: seconds. Valid values: 1 to 60.	15	CCM 2.1.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout: "30"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Disable HTTP/2 for a listener.

The following annotations are used.

Only HTTPS listeners are supported.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,). Example: `https:443,http:80`.	No default value	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of a certificate for the CLB instance. You can log on to the CLB console and view certificate IDs on the Certificates page. Note For more information about how to create a certificate, see Use a certificate from Alibaba Cloud SSL Certificates Service.	No default value	CCM 1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-http2-enabled	Specifies whether to enable HTTP/2. Valid values: `on`: enables HTTP/2. `off`: does not enable HTTP/2.	on	CCM 2.1.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-http2-enabled: "off"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Specify the timeout period of requests for a listener.

The following annotations are used.

Only HTTP and HTTPS listeners are supported.

Annotation

Description

Default value

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Separate multiple values with commas (,). Example: https:443,http:80.

No default value

CCM 1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-request-timeout

The timeout period of requests. Unit: seconds. Valid values: 1 to 180.

If the backend server fails to respond within the timeout period, the CLB instance stops waiting and returns an HTTP 504 error code to the client.

CCM 2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-request-timeout: "60"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Specify the timeout period of connections for a listener.

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-established-timeout

Only TCP listeners are supported.

Description	Default value	Supported CCM version
The timeout period of connections. Unit: seconds. Valid values: 10 to 900. For more information, see CreateLoadBalancerTCPListener.	No default value	CCM 2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-established-timeout: "60"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure security policies for a listener

The following annotations are used.

Only HTTPS listeners are supported.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with commas (,). Example: `https:443,http:80`.	No default value	CCM 1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of a certificate for the CLB instance. You can log on to the CLB console and view certificate IDs on the Certificates page. Note For more information about how to create a certificate, see Use a certificate from Alibaba Cloud SSL Certificates Service.	No default value	CCM 1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy	The TLS security policy. Each security policy contains TLS protocol versions and cipher suites available for HTTPS. For more information, see CreateLoadBalancerHTTPSListener. Valid values: `tls_cipher_policy_1_0` `tls_cipher_policy_1_1` `tls_cipher_policy_1_2` `tls_cipher_policy_1_2_strict` `tls_cipher_policy_1_2_strict_with_1_3`	tls_cipher_policy_1_0	CCM 2.4.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443,http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy: "tls_cipher_policy_1_2"
  name: nginx
  namespace: default
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure both TCP and UDP for a listener

Note

Only clusters that run Kubernetes 1.24 and later support this feature. For more information about how to update a cluster, see Manually upgrade ACK clusters.

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: udp
    port: 80
    protocol: UDP
    targetPort: 81
  selector:
    app: nginx
  sessionAffinity: None
  type: LoadBalancer

Configure proxy protocol for TCP and UDP listeners

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol

Description

Default value

Supported CCM version

Specifies whether to enable proxy protocol for a TCP or UDP listener. After proxy protocol is configured, client IP addresses can be passed to backend servers. Valid values:

on: enables Proxy Protocol.
off: does not enable Proxy Protocol.

Important

Enabling this feature may cause service interruptions. You must stop and update your applications before you enable proxy protocol. Proceed with caution.

off

CCM 2.6.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Common operations to manage backend server groups

Add worker nodes that have specified labels as the backend servers of a CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-backend-label

Description	Default value	Supported CCM version
Specifies the labels of worker nodes that you want to add as the backend servers of the CLB instance. Separate multiple labels with commas (,). Example: `"k1=v1,k2=v2"`. A node must have all the specified labels before you can add the node as a backend server.	No default value	CCM 1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-backend-label: "failure-domain.beta.kubernetes.io/zone=ap-southeast-5a"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Add the nodes on which pods run as the backend servers of a CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler

By default, the externalTrafficPolicy parameter is set to Cluster for a Service. In Cluster mode, all nodes in the cluster are added as the backend servers of the CLB instance. In Local mode, only nodes on which the pods are deployed are added as the backend servers of the CLB instance.

Description

Default value

Supported CCM version

The scheduling algorithm. Valid values:

wrr: backend servers with higher weights receive more requests than backend servers with lower weights.
In Local mode, you must set the scheduling algorithm to weighted round-robin (WRR).
Note
For CCM V1.9.3.164-g2105d2e-aliyun and later, node weights are calculated based on the number of pods that run on each node for Services whose externalTrafficPolicy parameter is set to Local. For more information about node weight calculation, see How does the CCM calculate node weights in Local mode? section of the "Service FAQ" topic.
rr: uses the round-robin algorithm. This way, requests are forwarded to backend servers in sequence.

CCM 1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler: "wrr"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Remove unschedulable nodes from the backend server group of a CLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-remove-unscheduled-backend

Description

Default value

Supported CCM version

Specifies whether to remove unschedulable nodes from the backend server group of the CLB instance. Valid values:

on: removes unschedulable nodes from the backend server group of the CLB instance.
off: the kubectl cordon and kubectl drain commands are run to set nodes to the unschedulable state. In this scenario, unschedulable nodes are not removed from the backend server group of the CLB instance.

off

CCM 1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-remove-unscheduled-backend: "on"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 30080
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Add pods that are assigned ENIs as the backend servers of a CLB instance

Annotation: service.beta.kubernetes.io/backend-type

Description

Default value

Supported CCM version

The type of resources that are added as the backend servers of the CLB instance. Valid values:

eni: adds pods as the backend servers of the CLB instance. This improves the network forwarding performance. This annotation takes effect only in Terway mode.
You can also manually set the service.beta.kubernetes.io/backend-type annotation from "eni" to "ecs" and then add ECS instances as the backend servers of the CLB instance.
ecs: adds ECS instances as the backend servers of the CLB instance.

Flannel mode: ecs.
Terway mode:
- Terway clusters that are created before August 10, 2020: ecs
- Terway clusters that are created after August 10, 2020: eni

CCM 1.9.3.164-g2105d2e-aliyun and later

  apiVersion: v1
  kind: Service
  metadata:
    annotations:
      service.beta.kubernetes.io/backend-type: "eni"
    name: nginx
  spec:
    ports:
    - name: http
      port: 30080
      protocol: TCP
      targetPort: 80
    selector:
      app: nginx
    type: LoadBalancer

Reuse an existing vServer group

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vgroup-port

This annotation can be used to reuse a vServer group. This annotation takes effect only when you use an existing CLB instance. For more information, see Use the CCM to deploy services across clusters.

Specify weights for Services to enable weighted round robin

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-weight

When multiple Services use the same CLB instance, you can use this annotation to specify the percentage of traffic distributed to the current Service. This annotation takes effect only when an existing vServer group is reused. For more information, see Use the CCM to deploy services across clusters.

Table of contents

Annotation usage notes

Common operations to manage CLB instances

Create an Internet-facing CLB instance

Create an internal-facing CLB instance

Create a CLB instance that uses the specified specifications

Use an existing CLB instance

Use an existing CLB instance and forcefully overwrite the listeners of the CLB instance

Specify the primary and secondary zones when you create a CLB instance

Create a pay-by-bandwidth CLB instance

Specify a vSwitch for a CLB instance

Specify an IP address for an internal-facing CLB instance

Add additional tags to a CLB instance

Create an IPv6 CLB instance

Enable deletion protection for a CLB instance

Enable the configuration read-only mode for a CLB instance

Specify the name of a CLB instance

Specify a resource group to which a CLB instance belongs

Specify a hostname for a Service

Create a pay-by-LCU CLB instance

Retain automatically created CLB instances when you delete a Service

Common operations to manage listeners

Specify the session persistence period for a TCP-based CLB instance

Enable session persistence for an HTTP or HTTPS listener by inserting a cookie

Configure an ACL for a CLB instance

Configure port forwarding for a CLB instance

Specify the scheduling algorithm for a CLB instance

Create a UDP listener

Create an HTTP listener

Create an HTTPS listener

Create a listener that enables health checks

Enable TCP health checks

Enable UDP health checks

Disable health checks for a TCP or UDP listener

Enable HTTP health checks

Configure connection draining for a listener

Configure additional request headers for a listener

Specify the timeout period of idle connections for a listener

Disable HTTP/2 for a listener.

Specify the timeout period of requests for a listener.

Specify the timeout period of connections for a listener.

Configure security policies for a listener

Configure both TCP and UDP for a listener

Configure proxy protocol for TCP and UDP listeners

Common operations to manage backend server groups

Add worker nodes that have specified labels as the backend servers of a CLB instance

Add the nodes on which pods run as the backend servers of a CLB instance

Remove unschedulable nodes from the backend server group of a CLB instance

Add pods that are assigned ENIs as the backend servers of a CLB instance

Reuse an existing vServer group

Specify weights for Services to enable weighted round robin