Considerations for configuring a LoadBalancer Service and the policies used by the CCM to update SLB resources - Container Service for Kubernetes

If you specify Type=LoadBalancer for a Service, the cloud controller manager (CCM) of Container Service for Kubernetes (ACK) creates and configures Server Load Balancer (SLB) resources for the Service, including an SLB instance, listeners, and backend server groups. The supported types of SLB instances are Classic Load Balancer (CLB) and Network Load Balancer (NLB). This topic describes the considerations for configuring a LoadBalancer Service and the policies that are used by the CCM to update SLB resources.

Usage notes

Reuse of SLB instances

You can reuse only the SLB instances that are created by using the SLB console. You cannot reuse the SLB instances that are automatically created by the CCM.
If you want to reuse an internal-facing SLB instance in an ACK cluster, the instance must be in the same virtual private cloud (VPC) as the ACK cluster. Only NLB instances can be reused for cross-VPC scenarios.
The network type of the SLB instance to be reused must be the same as the access type of the Service. If you specify service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "internet" to enable Internet access for the Service, the network type of the SLB instance must be Internet-facing. If you specify service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet" to enable internal access for the SLB instance, the network type of the SLB instance must be internal-facing.
A listening port of an SLB instance cannot be used by multiple Services at a time.
To reuse an existing SLB instance across clusters, you must make sure that the Services exposed by the SLB instance have different names, unless they are deployed in different namespaces.

Load balancing management by the CCM

The CCM creates and configures SLB resources for only Services with the Type=LoadBalancer setting.

Important

If you change the setting for a Service from Type=LoadBalancer to Type!=LoadBalancer, the CCM deletes the configurations of the SLB instance created for the Service. As a result, the Service cannot be accessed by using the SLB instance.

The CCM uses a declarative API and automatically updates the configurations of an SLB instance to match the configurations of the exposed Service when specific conditions are met. If you set service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: to true, the SLB configurations that you modify in the SLB console may be overwritten by the CCM.

Important

For an SLB instance created and managed by the CCM, we recommend that you do not modify the configurations of the instance in the SLB console. Otherwise, the CCM may overwrite the configurations and the relevant Service may become inaccessible.

Considerations for managing large-scale SLB instances with CCM

CCM has certain limitations in handling Service events. In large-scale clusters, especially those with a large number of nodes or numerous LoadBalancer Services, the following situations can lead to delays in operations related to SLB instances and modifications to server group endpoints performed by CCM:

Creation or deletion of a large number of Services.
Concurrent changes to numerous Service endpoints.
Addition and removal of nodes when many LoadBalancer Services exist.

When you perform bulk changes to meet business requirements, it is essential to conduct capacity assessments and stress testing in advance. This can help prevent business disruptions that may arise from delays in CCM processing.

What do I do if I need to replace an SLB instance for a Service?

For a LoadBalancer Service, you cannot change the SLB instance specified or created for the Service. To replace an SLB instance in this case, delete and recreate the Service.

Quotas

VPC

A node in a cluster is mapped to a route entry in a route table. By default, each route table for a VPC can contain up to 200 entries. If the number of nodes in a cluster exceeds 200, apply for a quota increase in the log on to the Quota Center console and submit an application
For more information about the limits and quotas related to VPC, see Limits and quotas.
To query the VPC resource quotas, go to the Quota Management page in the VPC console.

SLB

The CCM creates SLB instances for Services with the Type=LoadBalancer setting. By default, you can have a maximum of 60 SLB instances within your Alibaba Cloud account. To create more SLB instances, apply for a quota increase in the log on to the Quota Center console and submit an application.
The CCM automatically adds Elastic Compute Service (ECS) instances to the backend server groups of an SLB instance based on the Service configurations.
- By default, an ECS instance can be added to at most 50 backend server groups. To add the ECS instance to more backend server groups, apply for a quota increase in the log on to the Quota Center console and submit an application.
- By default, you can add at most 200 backend servers to an SLB instance. To add more backend servers to an SLB instance, apply for a quota increase in the log on to the Quota Center console and submit an application.
The CCM automatically creates listeners that use Service ports for SLB instances. By default, each SLB instance supports at most 50 listeners. To increase the number of listeners supported by each SLB instance, apply for a quota increase in the log on to the Quota Center console and submit an application.
For more information about the limits on SLB, see Limits on CLB and Limits on NLB.
To query the SLB resource quotas, go to the Quota Center page in the SLB console.

Policies used to update SLB resources

ACK allows you to specify an existing SLB instance for a Service. You can also use the CCM to automatically create an SLB instance for the Service. The two methods use different policies to update SLB resources, as described in the following table.

Resource object	Existing SLB instance	SLB instance created and managed by the CCM
SLB instance	Use the following annotation to specify an existing SLB instance for a Service: `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id`. The CCM uses the specified SLB instance to enable load balancing and automatically creates vServer groups for the instance. You can use other annotations to configure the SLB instance. If the Service is deleted, the CCM does not delete the SLB instance that is specified in the preceding annotation.	The CCM automatically creates, configures, and manages SLB resources based on the Service configuration, including an SLB instance, listeners, and vServer groups. If the Service is deleted, the CCM deletes the created SLB instance.
Listener	Use the following annotation to configure listeners: `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners`. If you set the annotation to `false`, the CCM does not configure or manage listeners for the SLB instance. If you set the annotation to `true`, the CCM configures and manages listeners for the SLB instance based on the Service configuration. If the SLB instance has existing listeners, the CCM creates new listeners to replace the existing ones.	The CCM configures listeners for the SLB instance based on the Service configuration.
Backend server group	When the endpoints of the Service change or the cluster nodes change, the CCM automatically updates the vServer groups of the SLB instance created for the Service. If the cluster uses the Terway network plug-in, the CCM adds pod IP addresses instead of ECS nodes to SLB instances as backend servers by default. If the cluster uses the Flannel network plug-in, the policies for updating backend server groups vary based on the mode of the Service. If `spec.externalTrafficPolicy = Cluster` is specified for the Service, the CCM adds all cluster nodes to the SLB instance as backend servers by default. If node labels are specified in the Service configuration, the CCM adds only cluster nodes that have the specified labels to the SLB instance. Important The number of SLB instances to which an ECS instance can be added is limited. If the Service is in Cluster mode, the quota is consumed at a high rate. When the quota is used up, Service reconciliation fails. To fix this issue, set the Local mode for the Service. If `spec.externalTrafficPolicy = Local` is specified for the Service, the CCM adds only the nodes where the backend pods of the Service are deployed to the SLB instance as backend servers. This can reduce the consumption rate of SLB quotas. In addition, source IP addresses can also be preserved in Layer 4 load balancing. If the cluster uses the Flannel network plug-in, the CCM does not add master nodes to the SLB instances used by the cluster. If the cluster uses the Flannel network plug-in, the CCM does not remove a node after you run the `kubectl drain` command on the node. In addition, the CCM does not remove a node after you run the `kubectl cordon` command to mark the node as unschedulable. To remove such a node, set the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-remove-unscheduled-backend` annotation to `on`.

Enable deletion protection for Services

You can enable deletion protection for Services involving critical business or sensitive data to avoid maintenance costs associated with accidental deletions. After you enable this feature, you must manually disable it before deleting any resources. For more information about how to enable Service Deletion Protection, see Enable deletion protection for a Service.