how to configure Elastic Container Instance-based scheduling in ACK - Container Service for Kubernetes

When you deploy Services in a Container Service for Kubernetes (ACK) cluster, you can configure tolerations and node affinity to enable the scheduler to use only Elastic Compute Service (ECS) instances or elastic container instances, or allow the scheduler to automatically apply for elastic container instances when ECS instances are insufficient. You can configure different scheduling policies to scale resources in different scenarios.

Terms

Taint: By default, the virtual-kubelet.io/provider=alibabacloud:NoSchedule taint is added to all virtual nodes in an ACK cluster in case elastic container instances are used without your approval.
Toleration: Tolerations apply to pods. Tolerations allow the scheduler to schedule pods to nodes with matching taints. In an ACK cluster, you need to configure the following toleration to tolerate the virtual-kubelet.io/provider=alibabacloud:NoSchedule taint. This way, pods can be scheduled to elastic container instances.
```
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: alibabacloud
        effect: NoSchedule
```
Node affinity: Node affinity is a property of pods that attracts pods to a set of nodes either as a preference or a hard requirement.

Prerequisites

The ack-virtual-node component is installed in an ACK Pro cluster. For more information, see Use Elastic Container Instance in ACK clusters.
The virtual node-based pod scheduling policy is enabled for the ACK Pro cluster. For more information, see Enable the virtual node-based pod scheduling policy for an ACK cluster.

Procedure

The following sections introduce how to use taints, tolerations, and node affinity to schedule pods based on the following policies:

Use only elastic container instances: The ECS instances in the cluster are not used.
Preferably use ECS instances: Elastic container instances are used only when the ECS instances in the cluster are insufficient.
Use only ECS instances: Only the ECS instances in the cluster are used.

Use only elastic container instances

Click to view details

apiVersion: apps/v1
kind: Deployment
metadata:
  name: eci-only
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: alibabacloud
        effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: type
                operator: In
                values:
                - virtual-kubelet
      containers:
      - name: my-container
        image: nginx

Preferably use ECS instances

Show YAML content

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ecs-prefer
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: alibabacloud
        effect: NoSchedule
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: type
                operator: NotIn
                values:
                - virtual-kubelet
      containers:
      - name: my-container
        image: nginx

If you want to prioritize an ECS node pool that consists of nodes with the label_1=key_1 label during pod scheduling and configure the scheduler to schedule pods to virtual nodes when the node pool cannot provide sufficient ECS nodes, use the following deployment.

Click to view details

apiVersion: apps/v1
kind: Deployment
metadata:
  name: some-ecs-prefer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: alibabacloud
        effect: NoSchedule
      affinity:
        nodeAffinity:
# Schedule pods only to nodes with the label_1:key_1 or type:virtual-kubelet label. 
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: label_1
                operator: In
                values:
                - key_1
            - matchExpressions:
              - key: type
                operator: In
                values:
                - virtual-kubelet
# Configure the scheduler to preferably schedule pods to nodes with the label_1:key_1 label. 
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - key: label_1
                operator: In
                values:
                - key_1
          - weight: 1
            preference:
              matchExpressions:
              - key: type
                operator: In
                values:
                - virtual-kubelet
      containers:
      - name: my-container
        image: nginx

Note

The logical relation among multiple node affinity rules in nodeSelectorTerms is OR. A pod is scheduled to a node if the pod matches one of the rules.
The logical relation among multiple expressions in the matchExpressions field of nodeSelectorTerms is AND. A pod is scheduled to a node only if the pod matches all expressions.
The node affinity setting preferredDuringSchedulingIgnoredDuringExecution cannot guarantee that pods are scheduled to elastic container instances only when ECS instances are insufficient. Pods may be scheduled to elastic container instances even if there are sufficient ECS instances.

Use only ECS instances

To avoid using elastic container instances which are pricy, the following taint is added to virtual nodes by default.

      virtual-kubelet.io/provider=alibabacloud:NoSchedule

If you do not configure a toleration to tolerate the taint, pods are scheduled only to ECS instances.

Click to view details

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ecs-only
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-container
        image: nginx