Use annotations supported by ACK clusters to configure NLB instances - Container Service for Kubernetes

You can add annotations to the YAML file of a Service to configure Network Load Balancer (NLB) instances to use various load balancing features. NLB instances are next-generation Layer 4 load balancers developed by Alibaba Cloud for the Internet of Everything (IoE). NLB instances provides ultrahigh performance and auto scaling. This topic describes how to use annotations to configure NLB instances, listeners, and backend server groups.

Category	Hyperlink
Annotation usage notes
Common operations to manage NLB instances	Create an Internet-facing NLB instance
	Create an internal-facing NLB instance
	Specify the name of an NLB instance
	Specify the resource group to which an NLB instance belongs
	Create a dual-stack NLB instance
	Add additional tags to an NLB instance
	Use an existing NLB instance
	Associate an EIP bandwidth plan with an NLB instance
	Enable deletion protection for an NLB instance
	Enable the configuration read-only mode for an NLB instance
	Specify the network type of an IPv6 NLB instance
Common operations to manage listeners	Configure security groups for a listener
	Configure both TCP and UDP for a listener
	Create TCP listeners
	Create a UDP listener
	Create TCP/SSL listeners
	Enable mutual authentication
	Configure a TLS security policy
	Configure proxy protocol
	Configure proxy protocol with additional information
	Specify the maximum number of connections that can be created per second
	Specify the timeout period of idle connections for the listeners
Common operations to manage backend server groups	Specify the scheduling algorithm for an NLB instance
	Configure connection draining
	Configure client IP preservation
	Configure TCP health checks
	Configure HTTP health checks
	Specify the backend server group type
	Reuse an existing vServer group
	Specify weights for Services to enable weighted round robin

Usage notes

The Kubernetes version of your cluster must be 1.24 or later and the cloud controller manager (CCM) must be of V2.5.0 or later. For more information about how to update a cluster, see Manually upgrade ACK clusters. For more information about how to update a component, see Manage components.
To configure an NLB instance for a Service, the spec.loadBalancerClass parameter of the Service must be set to alibabacloud.com/nlb. Otherwise, a Classic Load Balancer (CLB) instance is created by default.
You cannot modify the spec.loadBalancerClass parameter of a Service to change the type of a load balancer instance between CLB and NLB after the Service is created.
You cannot use the Container Service for Kubernetes (ACK) console to manage NLB instances. To manage NLB instances, use kubectl.

Common operations to manage NLB instances

Create an Internet-facing NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps

Description

Default value

Supported CCM version

The zones in which the NLB instance is deployed.

You can log on to the NLB console to view the regions and zones that support NLB. You must select at least two zones for each NLB instance. Separate multiple zones with commas (,). Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.

No default value

CCM 2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create an internal-facing NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type

Description

Default value

Supported CCM version

Creates an internal-facing NLB instance. You can change the value of the annotation to change the network type of an NLB instance. Valid values:

internet: creates an Internet-facing NLB instance.
intranet: creates an internal-facing NLB instance.

internet

CCM 2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify the name of an NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name

Description	Default value	Supported CCM version
The name of an NLB instance. The name must be 2 to 128 characters in length, and can contain letters, digits, periods (.), underscores (_), and hyphens (-). The name must start with a letter.	No default value	CCM 2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name: "${your-nlb-name}" # The name of the NLB instance. 
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify the resource group to which an NLB instance belongs

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id

Description

Default value

Supported CCM version

The ID of the resource group to which the NLB instance belongs. The resource group cannot be changed after it is specified.

You can log on to the Resource Management console to query resource group IDs.

No default value

CCM 2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id:  "${your-resource-group-id}" # The ID of the resource group. 
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create a dual-stack NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version

Description

Default value

Supported CCM version

The IP version of an NLB instance. The IP version cannot be changed after it is specified. To use this annotation, make sure that the kube-proxy mode of the cluster is set to IPVS. Valid values:

ipv4: IPv4.
DualStack: IPv4/IPv6 dual-stack.
- IPv6 must be enabled for the vSwitches specified in the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps annotation.
- The IPv6 address can be used only in an IPv6 network.

ipv4

CCM 2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: "DualStack"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Add additional tags to an NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags

Description	Default value	Supported CCM version
The additional tags that you want to add. Separate multiple tags with commas (,). Example: `k1=v1,k2=v2`.	No default value	CCM 2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags: "Key1=Value1,Key2=Value2"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Use an existing NLB instance

The following annotations are used.

Annotation

Description

Default value

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id

The ID of an existing NLB instance that you want to use.

No default value

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners

The annotation specifies whether to overwrite the existing listeners of the NLB instance. Valid values:

true: the CCM creates, updates, or deletes listeners for the NLB instance based on the configurations of the Service.
false: the CCM does not make any changes to the listeners of the NLB instance.

false

CCM 2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: "${your-nlb-id}" # The ID of the existing NLB instance that you want to use. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: "true"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Associate an EIP bandwidth plan with an NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth-package-id

Description

Default value

Supported CCM version

The ID of the Elastic IP Address (EIP) bandwidth plan with which you want to associate the NLB instance.

You can log on to the Virtual Private Cloud (VPC) console to query the IDs of EIP bandwidth plans.

No default value

CCM 2.9.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth-package-id: "cbwp-xxxxxxxxxx" 
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Enable deletion protection for an NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-modification-protection

Description

Default value

Supported CCM version

Specifies whether to enable deletion protection for an NLB instance. Valid values:

on
off

Important

If you manually enable deletion protection in the NLB console for an NLB instance that is created for a LoadBalancer Service, you can run the kubectl delete svc {your-svc-name} command to delete the NLB instance.

CCM 2.9.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-delete-protection: "on"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Enable the configuration read-only mode for an NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-modification-protection

Description

Default value

Supported CCM version

Specifies whether to enable the configuration read-only mode for the NLB instance. Valid values:

ConsoleProtection
NonProtection

ConsoleProtection

CCM 2.9.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-modification-protection: "ConsoleProtection"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify the network type of an IPv6 NLB instance

The following annotations are used.

Annotation

Description

Default value

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version

The IP version of an NLB instance. The IP version cannot be changed after it is specified. To use this annotation, make sure that the kube-proxy mode of the cluster is set to IPVS. Valid values:

ipv4: IPv4.
DualStack: IPv4/IPv6 dual-stack.
- IPv6 must be enabled for the vSwitches specified in the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps annotation.
- The IPv6 address can be used only in an IPv6 network.

ipv4

CCM 2.5.0 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ipv6-address-type

Specifies the network type of the IPv6 NLB instance. Valid values:

intranet: a private IPv6 address.
internet: a public IPv6 address.

Note

To use a public IPv6 address, an IPv6 gateway must be created in the VPC in which the NLB instance resides. For more information, see Create and manage an IPv6 gateway.

intranet

CCM 2.9.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: "DualStack"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ipv6-address-type: internet # Specify a public IPv6 address.
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Common operations to manage listeners

Configure security groups for listeners

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-security-group-ids

Description	Default value	Supported CCM version
The IDs of the security groups configured for listeners. Separate multiple IDs with commas (,). Example: `sg-aaaaa,sg-bbbbb`.	No default value	CCM 2.6.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-security-group-ids: "sg-aaaaa,sg-bbbbb" # Separate multiple security group IDs with commas (,). 
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure both TCP and UDP for a listener

Description	Default value	Supported CCM version
Only clusters that run Kubernetes 1.24 and later support this feature. For more information about how to update an ACK cluster, see Manually upgrade ACK clusters.	No default value	N/A

apiVersion: v1
kind: Service
metadata:
  annotations:
      service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: udp
    port: 80
    protocol: UDP
    targetPort: 81
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create TCP listeners

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create a UDP listener

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: udp
    port: 80
    protocol: UDP
    targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create TCP/SSL listeners

The following annotations are used.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	The protocol types of the listeners. Separate multiple listener types with commas (,). Example: `TCP:80,TCPSSL:443`.	No default value	CCM 2.5.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The server certificate ID. You can log on to the Certificate Management Service console to view SSL certificate IDs.	No default value	CCM 2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "tcpssl:443"
    # If the cluster is deployed in a region in the Chinese mainland, the SSL certificate ID appended with region information is ${your-cert-id}-cn-hangzhou. 
    # If the cluster is deployed in a region outside the Chinese mainland, the SSL certificate ID appended with region information is ${your-cert-id}-ap-southeast-1. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${The appended certificate ID}"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Enable mutual authentication

The following annotations are used.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	The protocol types of the listeners. Separate multiple listener types with commas (,). Example: `TCP:80,TCPSSL:443`.	No default value	CCM 2.5.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The server certificate ID. You can log on to the Certificate Management Service console to view SSL certificate IDs.	No default value
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cacert-id	The certificate authority (CA) certificate ID. You can log on to the Certificate Management Service console to view CA certificate IDs.	No default value
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cacert	Specifies whether to enable mutual authentication. Valid values: `on`: enables mutual authentication. `off`: disables mutual authentication.	No default value

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "tcpssl:443"   
    # If the cluster is deployed in a region in the Chinese mainland, the SSL certificate ID appended with region information is ${your-cert-id}-cn-hangzhou. 
    # If the cluster is deployed in a region outside the Chinese mainland, the SSL certificate ID appended with region information is ${your-cert-id}-ap-southeast-1. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${The appended certificate ID}" 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cacert-id: "${your-cacert-id}"  # The CA certificate ID. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cacert: "on"
name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure a TLS security policy

The following annotations are used.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	The protocol types of the listeners. Separate multiple listener types with commas (,). Example: `TCP:80,TCPSSL:443`.	No default value	CCM 2.5.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The server certificate ID. You can log on to the Certificate Management Service console to view SSL certificate IDs.	No default value
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy	The ID of the security policy. System and custom security policies are supported. Valid values: `tls_cipher_policy_1_0` `tls_cipher_policy_1_1` `tls_cipher_policy_1_2` `tls_cipher_policy_1_2_strict` `tls_cipher_policy_1_2_strict_with_1_3`	tls_cipher_policy_1_0

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "tcpssl:443"
    # If the cluster is deployed in a region in the Chinese mainland, the SSL certificate ID appended with region information is ${your-cert-id}-cn-hangzhou. 
    # If the cluster is deployed in a region outside the Chinese mainland, the SSL certificate ID appended with region information is ${your-cert-id}-ap-southeast-1. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${The appended certificate ID}" 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy: "tls_cipher_policy_1_0"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure proxy protocol

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol

Description

Default value

Supported CCM version

Specifies whether to enable proxy protocol to pass client IP addresses to backend servers. Valid values:

on:: enables proxy protocol.
off: disables proxy protocol.

Important

Before you enable proxy protocol, check whether proxy protocol V2 is enabled for the backend service. If proxy protocol V2 is disabled, the client IP addresses cannot be passed to backend servers. Configure with caution.

off

CCM 2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol: "on"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure proxy protocol with additional information

The following annotations are used.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol	Specifies whether to enable proxy protocol to pass client IP addresses to backend servers. Valid values: `on:`: enables proxy protocol. `off`: disables proxy protocol. Important Before you enable proxy protocol, check whether proxy protocol V2 is enabled for the backend service. If proxy protocol V2 is disabled, the client IP addresses cannot be passed to backend servers. Configure with caution.	off	CCM 2.5.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-vpc-id-enabled	Specifies whether the proxy protocol is used to pass the VpcId parameter to backend servers. Valid values: `on:`: uses the proxy protocol to pass the VpcId parameter to backend servers. `off`: does not use the proxy protocol to pass the VpcId parameter to backend servers.	off	CCM 2.9.1 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-ep-id-enabled	Indicates whether the proxy protocol is used to pass the PrivateLinkEpId parameter to backend servers. Valid values: `on:`: enables client IP preservation. `off`: disables client IP preservation.	off	CCM 2.9.1 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-eps-id-enabled	Specifies whether the proxy protocol is used to pass the PrivateLinkEpsId parameter to backend servers. Valid values: `on:`: uses proxy protocol to pass the PrivateLinkEpsId parameter to backend servers. `off`: does not use proxy protocol to pass the PrivateLinkEpsId parameter to backend servers.	off	CCM 2.9.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-ep-id-enabled: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-eps-id-enabled: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-vpc-id-enabled: "on"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify the maximum number of connections that can be created per second

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cps

Description	Default value	Supported CCM version
The maximum number of connections that can be created per second on the NLB instance. Valid values: 0 to 1000000. A value of 0 indicates that the number of connections that can be created per second is unlimited.	No default value	CCM 2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cps: "100"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify the timeout period of idle connections for the listeners

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout

Description	Default value	Supported CCM version
The timeout period of idle connections. Unit: seconds. Valid values: 10 to 900.	900	CCM 2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout: "60"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Common operations to manage backend server groups

Specify the scheduling algorithm for an NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler

Description

Default value

Supported CCM version

The scheduling algorithm. Valid values:

wrr: the weighted round-robin (WRR) algorithm is used. Backend servers with higher weights receive more requests than backend servers with lower weights. This is the default value.
rr: the round-robin algorithm is used. Requests are forwarded to backend servers in sequence.
sch: the source IP hashing algorithm. Requests from the same source IP address are forwarded to the same backend server.
tch: the four-element hashing algorithm is used. Consistent hashing is based on the following factors: source IP address, destination IP address, source port, and destination port. Requests that contain the same information based on the four factors are forwarded to the same backend server.
wlc: the weighted least connections algorithm distributes requests based on server weights and the number of connections to backend servers. If multiple backend servers have the same weight, the backend server that has the least connections receives more requests.

For more information about scheduling algorithms, see the Scheduler field in CreateServerGroup.

wrr

CCM 2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler: "sch"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure connection draining

The following annotations are used.

Annotation

Description

Default value

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain

Specifies whether to enable connection draining. Valid values:

on:: enables connection draining.
off: disables connection draining.

off

CCM 2.5.0 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout

The timeout period of connection draining. Unit: seconds. Valid values: 10 to 900.

No default value

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: "30"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure client IP preservation

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-client-ip

Description

Default value

Supported CCM version

Specifies whether to enable client IP preservation. Valid values:

on:: enables client IP preservation.
off: disables client IP preservation.

off

CCM 2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-client-ip: "on"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure TCP health checks

The following annotations are used. To configure TCP health checks, all annotations in the following table are required. By default, health checks are enabled for TCP ports.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag	Specifies whether to enable health checks. Valid values: `on:`: enables health checks. `off`: disables health checks.	on
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type	The protocol that you want to use for health checks. Valid values: `tcp` `http`	tcp
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-port	The backend server port that is used for health checks. Valid values: 0 to 65535. Default value: 0. A value of 0 indicates that the health check port specified on a backend server is used.	0
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout	The timeout period of health checks. Unit: seconds. Valid values: 1 to 300.	5
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks that must occur before an unhealthy backend server is declared healthy. Valid values: 2 to 10.	2
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks that must occur before a healthy backend server is declared unhealthy. Valid values: 2 to 10.	2
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval at which health checks are performed. Unit: seconds. Valid values: 1 to 50.	10

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "tcp"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "8"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "5"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure HTTP health checks

The following annotations are used. To configure TCP health checks, all annotations in the following table are required. By default, health checks are enabled for TCP ports.

Annotation	Description	Default value	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag	Specifies whether to enable health checks. Valid values: `on:`: enables health checks. `off`: disables health checks.	on	CCM 2.5.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type	The protocol that you want to use for health checks. Valid values: `tcp` `http`	tcp
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri	The path to which health check requests are sent. The path must be 1 to 80 characters in length, and can contain only letters, digits, and special characters. The path must start with a forward slash (/). For more information, see CreateServerGroup. Note This annotation takes effect only when the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type` annotation is set to `HTTP`.	No default value
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain	The domain name that is used for health checks. Valid values: $_ip: private server IP domain: domain name The domain name must be 1 to 80 characters in length, and can contain letters, digits, hyphens (-), and periods (.). Note This annotation takes effect only when the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type` annotation is set to `HTTP`.	No default value
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-port	The backend server port that is used for health checks. Valid values: 0 to 65535. Default value: 0. A value of 0 indicates that the health check port specified on a backend server is used.	0
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout	The timeout period of health checks. Unit: seconds. Valid values: 1 to 300.	5
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks that must occur before an unhealthy backend server is declared healthy. Valid values: 2 to 10.	2
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks that must occur before a healthy backend server is declared unhealthy. Valid values: 2 to 10.	2
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval at which health checks are performed. Unit: seconds. Valid values: 1 to 50.	10
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method	The HTTP method that is used for health checks. Valid values: `GET` `HEAD` Note This annotation takes effect only when the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type` annotation is set to `HTTP`.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "http"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri: "/test/index.html"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain: "www.test.com"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "10"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "5"
    # Optional. Specify the health check method. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method: "head"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify the backend server group type

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-server-group-type

Description

Default value

Supported CCM version

Specifies the backend server group type. Valid values:

Ip: allows you to add backend servers by specifying IP addresses.
Instance: allows you to add Elastic Compute Service (ECS) instances and elastic network interfaces (ENIs) as backend servers. This is the default value.

For more information about NLB server groups, see Overview of NLB server groups.

Instance

CCM 2.8.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # Example: cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-server-group-type: "Ip"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Reuse an existing vServer group

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vgroup-port

This annotation can be used to reuse a vServer group. This annotation takes effect only when you use an existing NLB instance. For more information, see Reuse an existing SLB instance to expose Services in different ACK clusters.

Specify weights for Services to enable weighted round robin

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-weight

When multiple Services use the same NLB instance, you can use this annotation to specify the percentage of traffic distributed to the current Service. This annotation takes effect only when an existing vServer group is reused. For more information, see Reuse an existing SLB instance to expose Services in different ACK clusters.

Container Service for Kubernetes:Use annotations to configure NLB instances

Table of contents

Usage notes

Common operations to manage NLB instances

Create an Internet-facing NLB instance

Create an internal-facing NLB instance

Specify the name of an NLB instance

Specify the resource group to which an NLB instance belongs

Create a dual-stack NLB instance

Add additional tags to an NLB instance

Use an existing NLB instance

Associate an EIP bandwidth plan with an NLB instance

Enable deletion protection for an NLB instance

Enable the configuration read-only mode for an NLB instance

Specify the network type of an IPv6 NLB instance

Common operations to manage listeners

Configure security groups for listeners

Configure both TCP and UDP for a listener

Create TCP listeners

Create a UDP listener

Create TCP/SSL listeners

Enable mutual authentication

Configure a TLS security policy

Configure proxy protocol

Configure proxy protocol with additional information

Specify the maximum number of connections that can be created per second

Specify the timeout period of idle connections for the listeners

Common operations to manage backend server groups

Specify the scheduling algorithm for an NLB instance

Configure connection draining

Configure client IP preservation

Configure TCP health checks

Configure HTTP health checks

Specify the backend server group type

Reuse an existing vServer group

Specify weights for Services to enable weighted round robin