All Products
Search
Document Center

Container Service for Kubernetes:Manage north-south traffic

Last Updated:Nov 14, 2024

Multi-cluster gateways provided by Distributed Cloud Container Platform for Kubernetes (ACK One) can manage north-south traffic in multi-cluster deployments based on MSE Ingresses. This allows you to use features such as active zone-redundancy, traffic load balancing, and header-based traffic routing. This topic describes how to use multi-cluster gateways to manage north-south traffic.

Background information

Benefits of multi-cluster gateways

Ingresses are commonly used in the container service sector to manage north-south traffic of Services deployed in Kubernetes clusters. Ingresses are scoped to clusters. You cannot use Ingresses to manage the traffic of multi-cluster applications. ACK One allows you to use MSE Ingresses as global Ingresses to centrally manage the north-south traffic of multi-cluster applications in a region. These Ingresses provide powerful traffic management capabilities to help you implement active zone-redundancy, traffic load balancing, and header-based traffic routing with low costs. In addition, multi-cluster gateways provide easy-to-use Ingress APIs to narrow the technical gap for beginners.

Important

Fees are charged when you use multi-cluster gateways. For more information about the billing of multi-cluster gateways, see Billing overview.

MSE references

  • An MseIngressConfig is a CustomResourceDefinition (CRD) that is provided by Microservices Engine (MSE) Ingress Controller. MseIngressConfigs are used to manage the lifecycles of MSE cloud-native gateways and configure Ingress listening options and global settings. For more information about how to use MseIngressConfigs, see Configure an MseIngressConfig.

  • MSE Ingresses support core and commonly used annotations of Nginx-Ingress. MSE Ingresses also provide other annotations to cover the disadvantages of Nginx-Ingress. For more information, see Annotations supported by MSE Ingress gateways.

Prerequisites

Step 1: Create a multi-cluster gateway on an ACK One Fleet instance

Create a multi-cluster gateway on the ACK One Fleet instance and attach associated clusters to the multi-cluster gateway. By default, the multi-cluster gateway is deployed across zones to ensure high availability.

Use the console

  1. Log on to the ACK One console. In the left-side navigation pane, choose Fleet > Multi-cluster Gateways.

  2. In the upper-right corner of the Multi-cluster Gateway page, click Create Gateway.

  3. In the panel that appears, modify the YAML file that is used to create the multi-cluster gateway based on your business requirements and click Create.

Use the CLI

  1. Obtain and record the vSwitch ID of the ACK One Fleet instance.

    1. Run the following command to query the vSwitch ID:

    aliyun adcp DescribeHubClusterDetails --ClusterId <your_fleet_clusterid>
    1. Record the vSwitch ID in the VSwitches field of the output.

  2. Create a file named mseingressconfig.yaml and add the following content to the file.

    Replace ${vsw-id1} with the vSwitch ID that you recorded. You can add an annotation to the gateway configuration file to specify the associated clusters that you want to add to the gateway.

    apiVersion: mse.alibabacloud.com/v1alpha1
    kind: MseIngressConfig
    metadata:
      name: ackone-gateway
      # Connect associated clusters to the MSE gateway. 
      #annotations:
      #  mse.alibabacloud.com/remote-clusters: ${cluster1},${cluster2}
    spec:
      common:
        instance:
          replicas: 3
          spec: 2c4g
        network:
          # You can configure both an Internet-facing Server load Balancer (SLB) instance and an internal-facing SLB instance. If no SLB instance is specified, an Internet-facing SLB instance is used by default. 
          #publicSLBSpec: slb.s2.small
          #privateSLBSpec: slb.s2.small
          vSwitches:
          - ${vsw-id1}
      ingress:
        local:
          ingressClass: mse
      name: mse-ingress
  3. Run the following command to create a gateway named mse-ingress on the ACK One Fleet instance:

    kubectl apply -f mseingressconfig.yaml
  4. Run the following command to check whether the gateway is created:

    kubectl get mseingressconfig ackone-gateway

    Expected output:

    NAME             STATUS      AGE
    ackone-gateway   Listening   3m15s

    The output indicates that the gateway is in the Listening state. This means that the cloud-native gateway is created and running. The gateway listens on Ingresses whose IngressClasses are mse.

    The status of a gateway created from an MseIngressConfig changes in the following order: Pending, Running, and Listening. State description:

    • Pending: The cloud-native gateway is being created. This process may take about 3 minutes.

    • Running: The cloud-native gateway is created and running.

    • Listening: The cloud-native gateway is running and listens on Ingresses.

    • Failed: The cloud-native gateway is invalid. You can check the message in the Status field to troubleshoot the issue.

Step 2: Attach associated clusters to the multi-cluster gateway

Use the console

  1. Log on to the ACK One console. In the left-side navigation pane, choose Fleet > Multi-cluster Gateways.

  2. In the upper part of the Multi-cluster Gateway page, select the multi-cluster gateway that you want to manage from the Select a gateway drop-down list and click Modify in the upper-right corner.

  3. In the ModifyGateway panel, modify the MseIngresConfig.yaml file. Modify the cluster IDs in the annotations parameter and click Update.

    Sample code:

    annotations:
      mse.alibabacloud.com/remote-clusters: ${cluster1-id},${cluster2-id}
    • ${cluster1-id} and ${cluster2-id} are the IDs of the associated clusters. Separate multiple cluster IDs with commas (,). You can modify the cluster IDs to add or remove associated clusters.

    • If you do not add associated clusters to a multi-cluster gateway when you create the multi-cluster gateway, the MseIngresConfig.yaml file does not have the annotations parameter. To add specified associated clusters to the multi-cluster gateway, you must add the preceding code to the metadata object in the MseIngresConfig.yaml file and then modify the cluster IDs.

Use the CLI

  1. You can modify the corresponding annotation in the mseingressconfig of the ACK One Fleet instance to add or remove associated clusters. Replace ${cluster1-id} and ${cluster2-id} with the IDs of the associated clusters to be attached. Separate cluster IDs with commas (,).

    annotations:
      mse.alibabacloud.com/remote-clusters: ${cluster1-id},${cluster2-id}
  2. Run the following command to check whether the associated clusters are attached to the multi-cluster gateway:

    kubectl get mseingressconfig ackone-gateway -ojsonpath="{.status.remoteClusters}"

    Expected output:

    [{"clusterId":"c7fb82****"},{"clusterId":"cd3007****"}]

    The output indicates the IDs of the associated clusters and no Failed information is returned. This means that the associated clusters are attached to the multi-cluster gateway.

Step 3: Use GitOps to deploy a sample application

  1. Use GitOps to deploy a sample application to the associated clusters. For more information, see Getting started with GitOps.

    1. Create a GitOps application for each of the associated clusters. In this example, the web-demo-cluster1 and web-demo-cluster2 applications are created.

    2. Source:

      • Set Repository URL to https://github.com/AliyunContainerService/gitops-demo.git.

      • Set Revision to HEAD.

      • Set Path to manifests/helm/web-demo.

    3. Specify the associated clusters as DESTINATION and set namespace to web-demo.

    4. The names of the environment variables in Helm Values Files are envName and the values are cluster1 and cluster2.

    The following code block shows the YAML content of the Deployment and Service.

    View the YAML content of the Deployment and Service

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: web-demo
        app.kubernetes.io/instance: web-demo-cluster1
      name: web-demo
      namespace: web-demo
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: web-demo
      template:
        metadata:
          labels:
            app: web-demo
        spec:
          containers:
          - name: web-demo
            image: acr-multiple-clusters-registry.cn-hangzhou.cr.aliyuncs.com/ack-multiple-clusters/web-demo:0.4.0
            env:
            - name: ENV_NAME
              value: cluster1
    ---
    apiVersion: v1       
    kind: Service
    metadata:
      name: service1
      namespace: web-demo
    spec:
      ports:
      - port: 80
        protocol: TCP
        targetPort: 8080
      selector:
        app: web-demo
      sessionAffinity: None
      type: ClusterIP

Step 4: Create an MSE Ingress to manage traffic in the associated clusters

You can set the IngressClass of an Ingress to MSE Ingress to create an MSE Ingress and then use traffic management capabilities with different annotations. MSE Ingresses support annotations for NGINX Ingresses. MSE Ingresses also provide additional annotations to allow you to use traffic governance capabilities that are not supported by NGINX Ingresses. For more information about the annotations supported by MSE Ingresses, see Annotations supported by MSE Ingress gateways. The following examples describe the use scenarios of multi-cluster traffic management.

Important

The Ingress objects and Service objects in the Deployment of the sample application must belong to the same namespace.

Example 1: Use load balancing to distribute traffic to all backend pods by default

Create an Ingress object on the ACK One Fleet instance to distribute traffic to backend pods whose names are the same as the names of the associated clusters. Set the traffic ratio to the ratio of pods in Cluster 1 to pods in Cluster 2. For example, if the ratio of pods in Cluster 1 to pods in Cluster 2 is 9:1, set the traffic ratio to 9:1. In this example, the ratio of pods in Cluster 1 to pods in Cluster 2 is 1:1. The following figure shows the topology.image.png

  1. Create a file named ingress-demo.yaml and copy the following content to the file.

    In the YAML file of the following Ingress object, use /svc1 below the domain name example.com to expose the backend Service service1.

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: web-demo
      namespace: web-demo
    spec:
      ingressClassName: mse
      rules:
      - host: example.com
        http:
          paths:
          - path: /svc1
            pathType: Exact
            backend:
              service:
                name: service1
                port: 
                  number: 80
  2. Run the following command to deploy the Ingress in the ACK One Fleet instance:

    kubectl apply -f ingress-demo.yaml
  3. Run the following command to query the public IP address of the multi-cluster gateway:

    kubectl get ingress web-demo -nargocd -ojsonpath="{.status.loadBalancer}"
  4. Run the following command to query the traffic routing information.

    Replace XX.XX.XX.XX with the public IP address of the multi-cluster gateway that you obtained in the preceding step.

    for i in {1..50}; do curl -H "host: example.com" XX.XX.XX.XX/svc1; sleep 1;  done

    Expected output:

    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster2 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 

    The output indicates that traffic is distributed to both clusters.

Example 2: Distribute traffic to only the specified cluster

Create an Ingress object on the ACK One Fleet instance to distribute traffic only to the backend pod of Cluster 1. The following figure shows the topology.image.png

  1. Create a file named ingress-demo-cluster-one.yaml and add the following content to the file.

    Add the mse.ingress.kubernetes.io/service-subset and mse.ingress.kubernetes.io/subset-labels annotations to the YAML file of the Ingress object to use /service1 below the domain name example.com to expose the backend Service service1. For more information about the annotations supported by MSE Ingresses, see Annotations supported by MSE Ingress gateways.

    • mse.ingress.kubernetes.io/service-subset: The name of the subset of the Service. We recommend that you use a name related to the cluster.

    • mse.ingress.kubernetes.io/subset-labels: The ID of the associated cluster.

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        annotations:
          mse.ingress.kubernetes.io/service-subset: cluster-demo-1
          mse.ingress.kubernetes.io/subset-labels: |
            topology.istio.io/cluster ${cluster1-id}
        name: web-demo-cluster-one
        namespace: web-demo
      spec:
        ingressClassName: mse
        rules:
        - host: example.com
          http:
            paths:
            - path: /service1
              pathType: Exact
              backend:
                service:
                  name: service1
                  port: 
                    number: 80
  2. Run the following command to deploy the Ingress in the ACK One Fleet instance:

    kubectl apply -f ingress-demo-cluster-one.yaml
  3. Run the following command to query the public IP address of the multi-cluster gateway:

    kubectl get ingress web-demo -nargocd -ojsonpath="{.status.loadBalancer}"
  4. Run the following command to query the traffic routing information.

    Replace XX.XX.XX.XX with the public IP address of the multi-cluster gateway that you obtained in the preceding step.

    for i in {1..50}; do curl -H "host: example.com" XX.XX.XX.XX/service1; sleep 1;  done

    Expected output:

    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    ...

    The output indicates that traffic is distributed to Cluster 1.

Example 3: Distribute traffic that matches the header to the specified cluster

To distribute traffic that matches a header to the backend pod of the specified cluster, you need to create an Ingress object named Example 1 or Example 2 on the ACK One Fleet instance. Then, create the following Ingress object. The Ingress object cannot be used separately. The following figure shows the topology.

Important

When you configure header-based traffic scheduling, you need to create an Ingress that is configured with the canary annotation and a header match policy, and create another Ingress without the canary annotation. Both Ingresses are configured with the same host and path. This way, the Ingress without the canary annotation can route traffic to the Service in another cluster. This is because Ingresses that use header-based traffic scheduling cannot be used separately. They must be used together with Ingresses that do not use header-based traffic scheduling so that unmatched traffic can be routed to another cluster.

image.png

  1. Create a file named ingress-demo-header.yaml and add the following content to the file.

    In the YAML files of the following Ingress objects, use /service1 below the domain name example.com to expose the backend Service service1.

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/service-subset: cluster-demo-2
        mse.ingress.kubernetes.io/subset-labels: |
          topology.istio.io/cluster c15d48ca9d1fd43f9bbb89c56a474843c
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "stage"
        nginx.ingress.kubernetes.io/canary-by-header-value: "gray"
      name: web-demo-cluster-second
      namespace: web-demo
    spec:
      ingressClassName: mse
      rules:
      - host: example.com
        http:
          paths:
          - path: /service1
            pathType: Exact
            backend:
              service:
                name: service1
                port: 
                  number: 80
  2. Run the following command to deploy the Ingress in the ACK One Fleet instance:

    kubectl apply -f ingress-demo-header.yaml
  3. Run the following command to query the public IP address of the multi-cluster gateway:

    kubectl get ingress web-demo -nargocd -ojsonpath="{.status.loadBalancer}"
  4. Run the following command to query the traffic routing information.

    Replace XX.XX.XX.XX with the public IP address of the multi-cluster gateway that you obtained in the preceding step.

    for i in {1..50}; do curl -H "host: example.com" -H "stage: gray" xx.xx.xx.xx/service1; sleep 1;  done

    Expected output:

    This is env cluster2 !
    Config file is 
    This is env cluster2 !
    Config file is 
    This is env cluster2 !
    Config file is 
    This is env cluster2 !
    Config file is 
    This is env cluster2 !
    Config file is 
    ...

    The output indicates that traffic with the stage: gray header is distributed to Cluster 2.

Example 4: Use cross-cluster disaster recovery for multi-cluster applications

Multi-cluster gateways provide the cross-cluster disaster recovery feature for multi-cluster applications. You can directly use this feature without configuration. For example, the preceding multi-cluster gateway manages the traffic of two associated clusters. If the Service in one of the clusters is down or deleted, traffic is automatically failed over to the other cluster. In Example 1, Example 2, and Example 3, when the Service in one of the clusters is down, traffic is failed over to the other cluster.

In the following section, Example 3 is used to demonstrate how disaster recovery is implemented. Traffic that carries the stage: gray header is routed to Cluster 2. When the number of pods created by the Deployment in Cluster 2 is scaled to 0, traffic is failed over to Cluster 1. The following figure shows the topology.image.png

  1. Run the following command to query the public IP address of the multi-cluster gateway:

    kubectl get ingress web-demo -nargocd -ojsonpath="{.status.loadBalancer}"
  2. Run the following command to query the traffic routing information.

    Replace XX.XX.XX.XX with the public IP address of the multi-cluster gateway that you obtained in the preceding step.

    for i in {1..50}; do curl -H "host: example.com" -H "stage: gray" XX.XX.XX.XX/service1; sleep 1;  done

    Expected output:

    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    ...

    The output indicates that traffic is automatically failed over to Cluster 1.

Example 5: Distribute traffic based on weights

In Example 1, you can modify the pod ratio to adjust the proportion of traffic distributed to each cluster. This example demonstrates how to use annotations to distribute traffic to different clusters based on weights. You can use this method to perform canary releases. Create the following Ingress objects on the ACK One Fleet instance. The following figure shows the topology.

Important

When you configure weight-based traffic scheduling, you need to create an Ingress that is configured with the canary annotation and a header match policy, and create another Ingress without the canary annotation. Both Ingresses are configured with the same host and path. This way, the Ingress without the canary annotation can route traffic to the Service in another cluster. This is because Ingresses that use weight-based traffic scheduling cannot be used separately. They must be used together with Ingresses that do not use weight-based traffic scheduling so that traffic can be routed to another cluster.

image.png

  1. Create a file named ingress-weight.yaml and add the following content to the file.

    In the YAML files of the following Ingress objects, replace ${cluster1-id} with the ID of the associated cluster. Add annotations to use /svc1-w below the domain name example.com to expose the backend Service service1.

    • mse.ingress.kubernetes.io/service-subset: The name of the subset of the Service. We recommend that you use a name related to the cluster.

    • mse.ingress.kubernetes.io/subset-labels: Specify the ID of the associated cluster.

    • nginx.ingress.kubernetes.io/canary: Set the value to "true" to enable canary releases.

    • nginx.ingress.kubernetes.io/canary-weight: Specify the percentage of traffic distributed to the cluster in a range of 0 to 100.

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        annotations:
          mse.ingress.kubernetes.io/service-subset: cluster-demo-1
          mse.ingress.kubernetes.io/subset-labels: |
            topology.istio.io/cluster ${cluster1-id}
        name: web-demo-weight
        namespace: web-demo
      spec:
        ingressClassName: mse
        rules:
        - host: example.com
          http:
            paths:
            - path: /svc1-w
              pathType: Exact
              backend:
                service:
                  name: service1
                  port: 
                    number: 80
      ---
      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        annotations:
          mse.ingress.kubernetes.io/service-subset: cluster-demo-2
          mse.ingress.kubernetes.io/subset-labels: |
            topology.istio.io/cluster ${cluster2-id}
          nginx.ingress.kubernetes.io/canary: "true"
          nginx.ingress.kubernetes.io/canary-weight: "10"
        name: web-demo-weight-canary
        namespace: web-demo
      spec:
        ingressClassName: mse
        rules:
        - host: example.com
          http:
            paths:
            - path: /svc1-w
              pathType: Exact
              backend:
                service:
                  name: service1
                  port: 
                    number: 80
  2. Run the following command to deploy the Ingress in the ACK One Fleet instance:

    kubectl apply -f ingress-weight.yaml -nargocd
  3. Run the following command to query the public IP address of the multi-cluster gateway:

    kubectl get ingress web-demo -nargocd -ojsonpath="{.status.loadBalancer}"
  4. Run the following command to query the traffic routing information.

    Replace XX.XX.XX.XX with the public IP address of the multi-cluster gateway that you obtained in the preceding step.

    for i in {1..50}; do curl -H "host: example.com" XX.XX.XX.XX/svc1-w; sleep 1;  done

    Expected output:

    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster2 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster1 !
    Config file is 
    This is env cluster2 !
    Config file is 
    This is env cluster1 !
    Config file is 
    ...

    The output indicates that 90% traffic is distributed to Cluster 1 and 10% traffic is distributed to Cluster 2.