Alibaba Cloud Service Mesh:Deploy ASM gateway pods to dedicated nodes

How it works

Three Kubernetes scheduling mechanisms work together to pin gateway pods to dedicated nodes:

Taints and tolerations work together to prevent the system from scheduling pods to inappropriate nodes. One or more taints can be applied to a node.

• If a pod does not tolerate a specific taint, a node with the matching taint does not accept the pod.

• If a pod tolerates a specific taint, the pod can but is not required to be scheduled to a node with the matching taint.

With this setup, non-gateway pods stay off the dedicated nodes (enforced by the taint), and gateway pods land only on those nodes (enforced by node affinity).

Prerequisites

• A cluster added to the ASM instance

• An ingress gateway deployed

Step 1: Label the target node

Labels identify which nodes are reserved for gateway pods.

1. List all nodes in the cluster:

      kubectl get nodes

   The output is similar to:

      NAME    STATUS   ROLES    AGE   VERSION
      node1   Ready    <none>   30d   v1.24.6
      node2   Ready    <none>   30d   v1.24.6
      node3   Ready    <none>   30d   v1.24.6

2. Add a label to the node that will host the gateway pods:

      # Syntax
      kubectl label nodes <node-name> <label-key>=<label-value>
   
      # Example
      kubectl label nodes node1 mykey4pod=asmgateway

   The output is similar to:

      node/node1 labeled

Step 2: Taint the node

A taint prevents pods without a matching toleration from scheduling onto the node. This keeps application workloads off the dedicated gateway node.

Run the following command to taint the node:

kubectl taint nodes node1 mykey=myvalue:NoSchedule

The output is similar to:

node/node1 tainted

The NoSchedule effect means only pods with a toleration matching key mykey, value myvalue, and effect NoSchedule can be scheduled onto node1.

Step 3: Configure node affinity and tolerations on the ASM gateway

1. Log in to the ASM console.

2. In the left-side navigation pane, choose Service Mesh > Mesh Management.

3. On the Mesh Management page, click the name of the ASM instance.

4. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.

5. On the Ingress Gateway page, find the target gateway and click YAML.

6. In the Edit dialog box, add the following configuration to the spec field, then click OK.

   Note

   Tip: To match a taint regardless of its value, use operator: "Exists" instead of operator: "Equal" and omit the value field.

   Field	Value	Description
   nodeAffinity	requiredDuringSchedulingIgnoredDuringExecution	The scheduler must place gateway pods on nodes labeled mykey4pod=asmgateway. If no matching node is available, the pods remain unscheduled.
   tolerations.operator	Equal	Matches the taint from Step 2, allowing the gateway pods to schedule onto the tainted node.

      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: mykey4pod
                operator: In
                values:
                - asmgateway
      tolerations:
      - key: "mykey"
        operator: "Equal"
        value: "myvalue"
        effect: "NoSchedule"

Verify the result

After the gateway pods restart, confirm they are running on the expected node.

kubectl get pods -n istio-system -o wide | grep <gateway-name>

ingressgateway-xxxx-yyyy   1/1   Running   0   5m   10.0.1.12   node1   <none>   <none>

What's next

• To schedule gateway pods across multiple dedicated nodes for higher availability, label and taint additional nodes, then add their labels to the matchExpressions values list.

• For more information about Kubernetes scheduling, see Taints and Tolerations and Assigning Pods to Nodes in the Kubernetes documentation.