All Products
Search

This Product

    Container Service for Kubernetes:Use multi-cluster gateways to manage north-south traffic

    Document Center

    Container Service for Kubernetes:Use multi-cluster gateways to manage north-south traffic

    Last Updated:Jul 26, 2024

    Multi-cluster gateways provided by Distributed Cloud Container Platform for Kubernetes (ACK One) can manage north-south traffic in multi-cluster deployments based on MSE Ingresses. This allows you to use features such as active zone-redundancy, traffic load balancing, and header-based traffic routing. This topic describes how to use multi-cluster gateways to manage north-south traffic.

    Background information

    Benefits of multi-cluster gateways

    Ingresses are commonly used in the container service sector to manage north-south traffic of Services deployed in Kubernetes clusters. Ingresses are scoped to clusters. You cannot use Ingresses to manage the traffic of multi-cluster applications. ACK One allows you to use MSE Ingresses as global Ingresses to centrally manage the north-south traffic of multi-cluster applications in a region. These Ingresses provide powerful traffic management capabilities to help you implement active zone-redundancy, traffic load balancing, and header-based traffic routing with low costs. In addition, multi-cluster gateways provide easy-to-use Ingress APIs to narrow the technical gap for beginners.

    Important

    Fees are charged when you use multi-cluster gateways. For more information about the billing of multi-cluster gateways, see Billing overview.

    MSE references

    • An MseIngressConfig is a CustomResourceDefinition (CRD) that is provided by Microservices Engine (MSE) Ingress Controller. MseIngressConfigs are used to manage the lifecycles of MSE cloud-native gateways and configure Ingress listening options and global settings. For more information about how to use MseIngressConfigs, see Configure an MseIngressConfig.

    • MSE Ingresses support core and commonly used annotations of Nginx-Ingress. MSE Ingresses also provide other annotations to cover the disadvantages of Nginx-Ingress. For more information, see Annotations supported by MSE Ingress gateways.

    Prerequisites

    Step 1: Create a multi-cluster gateway on an ACK One Fleet instance

    Create a multi-cluster gateway on the ACK One Fleet instance and attach associated clusters to the multi-cluster gateway. By default, the multi-cluster gateway is deployed across zones to ensure high availability.

    Use the console

    1. Log on to the ACK One console. In the left-side navigation pane, choose Fleet > Multi-cluster Gateways.

    2. In the upper-right corner of the Multi-cluster Gateway page, click Create Gateway.

    3. In the panel that appears, modify the YAML file that is used to create the multi-cluster gateway based on your business requirements and click Create.

    Use the CLI

    1. Obtain and record the vSwitch ID of the ACK One Fleet instance.

      1. Run the following command to query the vSwitch ID: 

      aliyun adcp DescribeHubClusterDetails --ClusterId <your_fleet_clusterid>

      1. Record the vSwitch ID in the VSwitches field of the output.

    2. Create a file named mseingressconfig.yaml and add the following content to the file.

      Replace ${vsw-id1} with the vSwitch ID that you recorded. You can add an annotation to the gateway configuration file to specify the associated clusters that you want to add to the gateway. 

      apiVersion: mse.alibabacloud.com/v1alpha1
kind: MseIngressConfig
metadata:
  name: ackone-gateway
  # Connect associated clusters to the MSE gateway. 
  #annotations:
  #  mse.alibabacloud.com/remote-clusters: ${cluster1},${cluster2}
spec:
  common:
    instance:
      replicas: 3
      spec: 2c4g
    network:
      # You can configure both an Internet-facing Server load Balancer (SLB) instance and an internal-facing SLB instance. If no SLB instance is specified, an Internet-facing SLB instance is used by default. 
      #publicSLBSpec: slb.s2.small
      #privateSLBSpec: slb.s2.small
      vSwitches:
      - ${vsw-id1}
  ingress:
    local:
      ingressClass: mse
  name: mse-ingress

    3. Run the following command to create a gateway named mse-ingress on the ACK One Fleet instance: 

      kubectl apply -f mseingressconfig.yaml

    4. Run the following command to check whether the gateway is created: 

      kubectl get mseingressconfig ackone-gateway

      Expected output:

      NAME             STATUS      AGE
ackone-gateway   Listening   3m15s

      The output indicates that the gateway is in the Listening state. This means that the cloud-native gateway is created and running. The gateway listens on Ingresses whose IngressClasses are mse.

      The status of a gateway created from an MseIngressConfig changes in the following order: Pending, Running, and Listening. State description:

      • Pending: The cloud-native gateway is being created. This process may take about 3 minutes.

      • Running: The cloud-native gateway is created and running.

      • Listening: The cloud-native gateway is running and listens on Ingresses.

      • Failed: The cloud-native gateway is invalid. You can check the message in the Status field to troubleshoot the issue.

    Step 2: Attach associated clusters to the multi-cluster gateway

    Use the console

    1. Log on to the ACK One console. In the left-side navigation pane, choose Fleet > Multi-cluster Gateways.

    2. In the upper part of the Multi-cluster Gateway page, select the multi-cluster gateway that you want to manage from the Select a gateway drop-down list and click Modify in the upper-right corner.

    3. In the ModifyGateway panel, modify the MseIngresConfig.yaml file. Modify the cluster IDs in the annotations parameter and click Update.

      Sample code:

      annotations:
  mse.alibabacloud.com/remote-clusters: ${cluster1-id},${cluster2-id}

      • ${cluster1-id} and ${cluster2-id} are the IDs of the associated clusters. Separate multiple cluster IDs with commas (,). You can modify the cluster IDs to add or remove associated clusters.

      • If you do not add associated clusters to a multi-cluster gateway when you create the multi-cluster gateway, the MseIngresConfig.yaml file does not have the annotations parameter. To add specified associated clusters to the multi-cluster gateway, you must add the preceding code to the metadata object in the MseIngresConfig.yaml file and then modify the cluster IDs.

    Use the CLI

    1. You can modify the corresponding annotation in the mseingressconfig of the ACK One Fleet instance to add or remove associated clusters. Replace ${cluster1-id} and ${cluster2-id} with the IDs of the associated clusters to be attached. Separate cluster IDs with commas (,). 

      annotations:
  mse.alibabacloud.com/remote-clusters: ${cluster1-id},${cluster2-id}

    2. Run the following command to check whether the associated clusters are attached to the multi-cluster gateway: 

      kubectl get mseingressconfig ackone-gateway -ojsonpath="{.status.remoteClusters}"

      Expected output:

      [{"clusterId":"c7fb82****"},{"clusterId":"cd3007****"}]

      The output indicates the IDs of the associated clusters and no Failed information is returned. This means that the associated clusters are attached to the multi-cluster gateway.

    Step 3: Use GitOps to deploy a sample application

    1. Use GitOps to deploy a sample application to the associated clusters. For more information, see Work with GitOps.

      1. Create a GitOps application for each of the associated clusters. In this example, the web-demo-cluster1 and web-demo-cluster2 applications are created.

      2. Source:

        • Set Repository URL to https://github.com/AliyunContainerService/gitops-demo.git.

        • Set Revision to HEAD.

        • Set Path to manifests/helm/web-demo.

      3. Specify the associated clusters as DESTINATION and set namespace to web-demo.

      4. The names of the environment variables in Helm Values Files are envName and the values are cluster1 and cluster2.

      The following code block shows the YAML content of the Deployment and Service.

      View the YAML content of the Deployment and Service

      apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: web-demo
    app.kubernetes.io/instance: web-demo-cluster1
  name: web-demo
  namespace: web-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-demo
  template:
    metadata:
      labels:
        app: web-demo
    spec:
      containers:
      - name: web-demo
        image: acr-multiple-clusters-registry.cn-hangzhou.cr.aliyuncs.com/ack-multiple-clusters/web-demo:0.4.0
        env:
        - name: ENV_NAME
          value: cluster1
---
apiVersion: v1       
kind: Service
metadata:
  name: service1
  namespace: web-demo
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: web-demo
  sessionAffinity: None
  type: ClusterIP

    Step 4: Create an MSE Ingress to manage traffic in the associated clusters

    You can set the IngressClass of an Ingress to MSE Ingress to create an MSE Ingress and then use traffic management capabilities with different annotations. MSE Ingresses support annotations for NGINX Ingresses. MSE Ingresses also provide additional annotations to allow you to use traffic governance capabilities that are not supported by NGINX Ingresses. For more information about the annotations supported by MSE Ingresses, see Annotations supported by MSE Ingress gateways. The following examples describe the use scenarios of multi-cluster traffic management.

    Important

    The Ingress objects and Service objects in the Deployment of the sample application must belong to the same namespace.

    Example 1: Use load balancing to distribute traffic to all backend pods by default

    Create an Ingress object on the ACK One Fleet instance to distribute traffic to backend pods whose names are the same as the names of the associated clusters. Set the traffic ratio to the ratio of pods in Cluster 1 to pods in Cluster 2. For example, if the ratio of pods in Cluster 1 to pods in Cluster 2 is 9:1, set the traffic ratio to 9:1. In this example, the ratio of pods in Cluster 1 to pods in Cluster 2 is 1:1. The following figure shows the topology.image.png

    1. Create a file named ingress-demo.yaml and copy the following content to the file.

      In the YAML file of the following Ingress object, use /svc1 below the domain name example.com to expose the backend Service service1. 

      apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-demo
  namespace: web-demo
spec:
  ingressClassName: mse
  rules:
  - host: example.com
    http:
      paths:
      - path: /svc1
        pathType: Exact
        backend:
          service:
            name: service1
            port: 
              number: 80

    2. Run the following command to deploy the Ingress in the ACK One Fleet instance: 

      kubectl apply -f ingress-demo.yaml

    3. Run the following command to query the public IP address of the multi-cluster gateway: 

      kubectl get ingress web-demo -nargocd -ojsonpath="{.status.loadBalancer}"

    4. Run the following command to query the traffic routing information.

      Replace XX.XX.XX.XX with the public IP address of the multi-cluster gateway that you obtained in the preceding step. 

      for i in {1..50}; do curl -H "host: example.com" XX.XX.XX.XX/svc1; sleep 1;  done

      Expected output:

      This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster2 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is

      The output indicates that traffic is distributed to both clusters.

    Example 2: Distribute traffic to only the specified cluster

    Create an Ingress object on the ACK One Fleet instance to distribute traffic only to the backend pod of Cluster 1. The following figure shows the topology.image.png

    1. Create a file named ingress-demo-cluster-one.yaml and add the following content to the file.

      Add the mse.ingress.kubernetes.io/service-subset and mse.ingress.kubernetes.io/subset-labels annotations to the YAML file of the Ingress object to use /service1 below the domain name example.com to expose the backend Service service1. For more information about the annotations supported by MSE Ingresses, see Annotations supported by MSE Ingress gateways.

      • mse.ingress.kubernetes.io/service-subset: The name of the subset of the Service. We recommend that you use a name related to the cluster.

      • mse.ingress.kubernetes.io/subset-labels: The ID of the associated cluster. 

        apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/service-subset: cluster-demo-1
    mse.ingress.kubernetes.io/subset-labels: |
      topology.istio.io/cluster ${cluster1-id}
  name: web-demo-cluster-one
  namespace: web-demo
spec:
  ingressClassName: mse
  rules:
  - host: example.com
    http:
      paths:
      - path: /service1
        pathType: Exact
        backend:
          service:
            name: service1
            port: 
              number: 80

    2. Run the following command to deploy the Ingress in the ACK One Fleet instance: 

      kubectl apply -f ingress-demo-cluster-one.yaml

    3. Run the following command to query the public IP address of the multi-cluster gateway: 

      kubectl get ingress web-demo -nargocd -ojsonpath="{.status.loadBalancer}"

    4. Run the following command to query the traffic routing information.

      Replace XX.XX.XX.XX with the public IP address of the multi-cluster gateway that you obtained in the preceding step. 

      for i in {1..50}; do curl -H "host: example.com" XX.XX.XX.XX/service1; sleep 1;  done

      Expected output:

      This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
...

      The output indicates that traffic is distributed to Cluster 1.

    Example 3: Distribute traffic that matches the header to the specified cluster

    To distribute traffic that matches a header to the backend pod of the specified cluster, you need to create an Ingress object named Example 1 or Example 2 on the ACK One Fleet instance. Then, create the following Ingress object. The Ingress object cannot be used separately. The following figure shows the topology.

    Important

    When you configure header-based traffic scheduling, you need to create an Ingress that is configured with the canary annotation and a header match policy, and create another Ingress without the canary annotation. Both Ingresses are configured with the same host and path. This way, the Ingress without the canary annotation can route traffic to the Service in another cluster. This is because Ingresses that use header-based traffic scheduling cannot be used separately. They must be used together with Ingresses that do not use header-based traffic scheduling so that unmatched traffic can be routed to another cluster.

    image.png

    1. Create a file named ingress-demo-header.yaml and add the following content to the file.

      In the YAML files of the following Ingress objects, use /service1 below the domain name example.com to expose the backend Service service1. 

      apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/service-subset: cluster-demo-2
    mse.ingress.kubernetes.io/subset-labels: |
      topology.istio.io/cluster c15d48ca9d1fd43f9bbb89c56a474843c
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "stage"
    nginx.ingress.kubernetes.io/canary-by-header-value: "gray"
  name: web-demo-cluster-second
  namespace: web-demo
spec:
  ingressClassName: mse
  rules:
  - host: example.com
    http:
      paths:
      - path: /service1
        pathType: Exact
        backend:
          service:
            name: service1
            port: 
              number: 80

    2. Run the following command to deploy the Ingress in the ACK One Fleet instance: 

      kubectl apply -f ingress-demo-header.yaml

    3. Run the following command to query the public IP address of the multi-cluster gateway: 

      kubectl get ingress web-demo -nargocd -ojsonpath="{.status.loadBalancer}"

    4. Run the following command to query the traffic routing information.

      Replace XX.XX.XX.XX with the public IP address of the multi-cluster gateway that you obtained in the preceding step. 

      for i in {1..50}; do curl -H "host: example.com" -H "stage: gray" xx.xx.xx.xx/service1; sleep 1;  done

      Expected output:

      This is env cluster2 !
Config file is 
This is env cluster2 !
Config file is 
This is env cluster2 !
Config file is 
This is env cluster2 !
Config file is 
This is env cluster2 !
Config file is 
...

      The output indicates that traffic with the stage: gray header is distributed to Cluster 2.

    Example 4: Use cross-cluster disaster recovery for multi-cluster applications

    Multi-cluster gateways provide the cross-cluster disaster recovery feature for multi-cluster applications. You can directly use this feature without configuration. For example, the preceding multi-cluster gateway manages the traffic of two associated clusters. If the Service in one of the clusters is down or deleted, traffic is automatically failed over to the other cluster. In Example 1, Example 2, and Example 3, when the Service in one of the clusters is down, traffic is failed over to the other cluster.

    In the following section, Example 3 is used to demonstrate how disaster recovery is implemented. Traffic that carries the stage: gray header is routed to Cluster 2. When the number of pods created by the Deployment in Cluster 2 is scaled to 0, traffic is failed over to Cluster 1. The following figure shows the topology.image.png

    1. Run the following command to query the public IP address of the multi-cluster gateway: 

      kubectl get ingress web-demo -nargocd -ojsonpath="{.status.loadBalancer}"

    2. Run the following command to query the traffic routing information.

      Replace XX.XX.XX.XX with the public IP address of the multi-cluster gateway that you obtained in the preceding step. 

      for i in {1..50}; do curl -H "host: example.com" -H "stage: gray" XX.XX.XX.XX/service1; sleep 1;  done

      Expected output:

      This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
...

      The output indicates that traffic is automatically failed over to Cluster 1.

    Example 5: Distribute traffic based on weights

    In Example 1, you can modify the pod ratio to adjust the proportion of traffic distributed to each cluster. This example demonstrates how to use annotations to distribute traffic to different clusters based on weights. You can use this method to perform canary releases. Create the following Ingress objects on the ACK One Fleet instance. The following figure shows the topology.

    Important

    When you configure weight-based traffic scheduling, you need to create an Ingress that is configured with the canary annotation and a header match policy, and create another Ingress without the canary annotation. Both Ingresses are configured with the same host and path. This way, the Ingress without the canary annotation can route traffic to the Service in another cluster. This is because Ingresses that use weight-based traffic scheduling cannot be used separately. They must be used together with Ingresses that do not use weight-based traffic scheduling so that traffic can be routed to another cluster.

    image.png

    1. Create a file named ingress-weight.yaml and add the following content to the file.

      In the YAML files of the following Ingress objects, replace ${cluster1-id} with the ID of the associated cluster. Add annotations to use /svc1-w below the domain name example.com to expose the backend Service service1.

      • mse.ingress.kubernetes.io/service-subset: The name of the subset of the Service. We recommend that you use a name related to the cluster.

      • mse.ingress.kubernetes.io/subset-labels: Specify the ID of the associated cluster.

      • nginx.ingress.kubernetes.io/canary: Set the value to "true" to enable canary releases.

      • nginx.ingress.kubernetes.io/canary-weight: Specify the percentage of traffic distributed to the cluster in a range of 0 to 100. 

        apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/service-subset: cluster-demo-1
    mse.ingress.kubernetes.io/subset-labels: |
      topology.istio.io/cluster ${cluster1-id}
  name: web-demo-weight
  namespace: web-demo
spec:
  ingressClassName: mse
  rules:
  - host: example.com
    http:
      paths:
      - path: /svc1-w
        pathType: Exact
        backend:
          service:
            name: service1
            port: 
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/service-subset: cluster-demo-2
    mse.ingress.kubernetes.io/subset-labels: |
      topology.istio.io/cluster ${cluster2-id}
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
  name: web-demo-weight-canary
  namespace: web-demo
spec:
  ingressClassName: mse
  rules:
  - host: example.com
    http:
      paths:
      - path: /svc1-w
        pathType: Exact
        backend:
          service:
            name: service1
            port: 
              number: 80

    2. Run the following command to deploy the Ingress in the ACK One Fleet instance: 

      kubectl apply -f ingress-weight.yaml -nargocd

    3. Run the following command to query the public IP address of the multi-cluster gateway: 

      kubectl get ingress web-demo -nargocd -ojsonpath="{.status.loadBalancer}"

    4. Run the following command to query the traffic routing information.

      Replace XX.XX.XX.XX with the public IP address of the multi-cluster gateway that you obtained in the preceding step. 

      for i in {1..50}; do curl -H "host: example.com" XX.XX.XX.XX/svc1-w; sleep 1;  done

      Expected output:

      This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster2 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster1 !
Config file is 
This is env cluster2 !
Config file is 
This is env cluster1 !
Config file is 
...

      The output indicates that 90% traffic is distributed to Cluster 1 and 10% traffic is distributed to Cluster 2.