You can use multilevel resource scheduling to optimize resource scheduling for workloads that are deployed on the cloud and in on-premises networks in a hybrid cloud environment. Multilevel resource scheduling can scale resources based on traffic fluctuations and help ensure the high availability of your applications. This topic describes how to use ack-co-scheduler to achieve multilevel resource scheduling.
Prerequisites
A registered cluster is created and an external Kubernetes cluster is connected to the registered cluster. For more information, see Create a registered cluster.
The following table describes the versions of the system components that are required.
Component
Required version
Kubernetes
1.18.8 and later
Helm
3.0 and later
Docker
19.03.5
OS
CentOS 7.6, CentOS 7.7, Ubuntu 16.04 and 18.04, and Alibaba Cloud Linux.
Background information
Multilevel resource scheduling is an extension of priority-based resource scheduling for hybrid cloud scenarios. Multilevel resource scheduling supports the idc type resources for pod scheduling. Container Service for Kubernetes (ACK) allows you to configure priority-based resource scheduling. When you deploy or scale out an application, you can use the ResourcePolicy resource to specify the nodes to which the pods are scheduled and the sequence in which the nodes are selected for pod scheduling. When you scale in the application, pods are deleted from nodes in the reverse sequence. For more information, see Configure priority-based resource scheduling.
Limits
Priority-based resource scheduling is mutually exclusive with the pod deletion cost feature. For more information about the pod deletion cost feature, see Pod deletion cost.
You cannot use priority-based resource scheduling and Elastic Container Instance-based scheduling at the same time. For more information about Elastic Container Instance-based scheduling, see Use Elastic Container Instance-based scheduling.
After you modify a ResourcePolicy, only the pods that are scheduled based on the modified ResourcePolicy are deleted based on the modified ResourcePolicy. The pods that are scheduled based on the original ResourcePolicy are deleted based on the original ResourcePolicy.
If you use this feature together with elastic node pools, invalid nodes may be added to the elastic node pools. Make sure that the elastic node pools are included in units. Do not specify the max parameter for the units.
Do not modify the ResourcePolicy before all pods that match the ResourcePolicy are deleted.
How to configure priority-based resource scheduling
Create a ResourcePolicy based on the following content:
apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
name: xxx
namespace: xxx
spec:
selector:
key1: value1
strategy: prefer
units:
- resource: idc
max: 3
- resource: ecs
nodeSelector:
key2: value2
- resource: ecs
nodeSelector:
key3: value3
- resource: eciselector: the selector that is used to select pods in a namespace. The ResourcePolicy is applied to the selected pods. In this example, pods that have thekey1=value1labelare selected.strategy: the scheduling policy. Set the value toprefer.units: the schedulable units. In a scale-out activity, pods are scheduled to nodes based on the priorities of the nodes that are listed underunitsin descending order. In a scale-in activity, pods are deleted from the nodes based on the priorities of the nodes in ascending order.resource: the type of elastic resource. The following types are supported:eci,ecs, andidc.nodeSelector: the selector that is used to select nodes with a specifiednodelabel. This parameter takes effect only if the resource parameter is set toecs.: the maximum number of instances that can be deployed by using the resources.
Install components
Install the following components before you use multilevel resource scheduling.
ack-virtual-node: optional. To use ECI, you need to install the ack-virtual-node component in the registered cluster.
ack-co-scheduler: You can use this component to create ResourcePolicy CustomResources (CRs) to use the multilevel resource scheduling feature.
Use onectl
Install onectl on your on-premises machine. For more information, see Use onectl to manage registered clusters.
Run the following command to install the ack-virtual-node and ack-co-scheduler components:
onectl addon install ack-virtual-node onectl addon install ack-co-schedulerExpected output:
Addon ack-virtual-node, version **** installed. Addon ack-co-scheduler, version **** installed.
Use the console
Log on to the ACK console and click Clusters in the left-side navigation pane.
On the Clusters page, click the name of the cluster that you want to manage and choose in the left-side navigation pane.
On the Add-ons page, search for ack-virtual-node and ack-co-scheduler in the search box, and click Install in the lower-right corner of each card.
In the message that appears, click OK.
Scenario 1: Priority-based multilevel scheduling
Create a ResourcePolicy with the following template:
apiVersion: scheduling.alibabacloud.com/v1alpha1 kind: ResourcePolicy metadata: name: cost-balance-policy spec: selector: app: nginx strategy: prefer units: - resource: idc - resource: ecs nodeSelector: alibabacloud.com/nodepool-id=np7b30b52e99194275a3940936c19ea42b - resource: eciCreate a Deployment that provisions two pods based on the following content:
apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: name: nginx annotations: addannotion: "true" labels: app: nginx # The pod label must be the same as the one that you specified for the selector in the ResourcePolicy. spec: schedulerName: ack-co-scheduler containers: - name: nginx image: nginx resources: requests: cpu: 200m limits: cpu: 1Run the following command to deploy an NGINX application:
kubectl apply -f nginx.yamlRun the following command to scale out pods:
kubectl scale deployment nginx --replicas 10Expected output:
deployment.apps/nginx scaledRun the following command to query the status of the pods:
kubectl get pod -o wideExpected output:
Control the number of replicated pods for the NGINX application and then check the pod scheduling result. The result shows that pods are scheduled based on the ResourcePolicy. Different types of resources are prioritized for pod scheduling in the following sequence: idc resources, ecs resources, and eci resources.
Scenario 2: Resource scheduling in hybrid cloud environments
Create a ResourcePolicy with the following template:
apiVersion: scheduling.alibabacloud.com/v1alpha1 kind: ResourcePolicy metadata: name: load-balance-policy spec: selector: app: nginx strategy: prefer units: - resource: idc max: 2 - resource: ecs nodeSelector: alibabacloud.com/nodepool-id=np7b30b52e99194275a3940936c19ea42b max: 4 - resource: eciCreate a Deployment that provisions two pods based on the following content:
apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: name: nginx annotations: addannotion: "true" labels: app: nginx # The pod label must be the same as the one that you specified for the selector in the ResourcePolicy. spec: schedulerName: ack-co-scheduler containers: - name: nginx image: nginx resources: requests: cpu: 200m limits: cpu: 1Run the following command to deploy an NGINX application:
kubectl apply -f nginx.yamlRun the following command to scale out pods:
kubectl scale deployment nginx --replicas 10Expected output:
deployment.apps/nginx scaledRun the following command to query the status of the pods:
kubectl get pod -o wideExpected output:
Control the number of replicated pods for the NGINX application and then check the pod scheduling result. The result shows that pods are scheduled based on the ResourcePolicy. Two pods use idc resources, four pods use ecs resources, and the other pods use eci resources.