All Products
Search
Document Center

Alibaba Cloud Service Mesh:Implement cross-region disaster recovery and load balancing by using multiple clusters

Last Updated:Feb 28, 2024

Service Mesh (ASM) provides cross-region traffic distribution and failover capabilities for applications. The cross-region traffic distribution feature implements cross-region load balancing by routing traffic to multiple clusters based on their weights. The cross-region failover feature implements cross-region disaster recovery by transferring traffic from a faulty region to another region. This topic describes how to use the cross-region failover and traffic distribution features to implement cross-region disaster recovery and load balancing. In the example of this topic, the Bookinfo application is used.

Plan a network

Before you use ASM, you must complete network configurations for ASM. This involves the CIDR blocks and names of vSwitches, virtual private clouds (VPCs), and clusters. In this example, a network is created based on the following plan:

Note

For more information about how to plan CIDR blocks for multiple clusters on the data plane, see Plan CIDR blocks for multiple clusters on the data plane.

  • Network plan for vSwitches and VPCs

    • vSwitches

      Important

      To prevent route conflicts when you use Cloud Enterprise Network (CEN) to connect VPCs, specify a unique CIDR block for each vSwitch.

      Category

      vSwitch

      VPC

      IPv4 CIDR block

      Cluster

      vpc-hangzhou-switch-1

      vpc-hangzhou

      20.0.0.0/16

      vpc-shanghai-switch-1

      vpc-shanghai

      21.0.0.0/16

      ASM instance

      vpc-hangzhou-switch-2

      vpc-hangzhou2

      192.168.0.0/24

    • VPCs

      Category

      VPC

      Region

      IPv4 CIDR block

      Cluster

      vpc-hangzhou

      cn-hangzhou

      20.0.0.0/8

      vpc-shanghai

      cn-shanghai

      21.0.0.0/8

      ASM instance

      vpc-hangzhou2

      cn-hangzhou

      192.168.0.0/16

  • Network plan for pods and services in clusters

    Cluster

    Region

    VPC

    Pod CIDR

    Service CIDR

    ack-hangzhou

    cn-hangzhou

    vpc-hangzhou

    10.0.0.0/16

    172.16.0.0/16

    ack-shanghai

    cn-shanghai

    vpc-shanghai

    10.1.0.0/16

    172.17.0.0/16

Step 1: Create clusters in different regions

  1. Create two vSwitches in the China (Hangzhou) and China (Shanghai) regions based on the preceding plan, and then create VPCs that are associated with the vSwitches. For more information, see Create a vSwitch and Create a VPC and a vSwitch.

  2. Use the VPCs that you created and the preceding network plan to create clusters in the China (Hangzhou) and China (Shanghai) regions. For more information, see Create an ACK managed cluster.

  3. Create an ASM instance in the China (Hangzhou) region based on the preceding network plan. For more information, see Create an ASM instance.

Step 2: Use CEN to implement cross-region VPC communication

You can connect the VPCs among clusters and those between the clusters and the ASM instance by using CEN.Service Mesh

  1. Log on to the CEN console and create a CEN instance. For more information, see Create a CEN instance.

  2. Create transit routers.

    1. On the Instances page, click the name of the CEN instance that you created in Substep 1. On the Basic Settings tab, click Create Transit Router.

    2. In the Create Transit Router dialog box, select a region from the Region drop-down list, specify Name, and then click OK.

    In this example, two transit routers are configured:

    • For one transit router, set Region to China (Shanghai) and Name to shanghai-router.

    • For the other transit router, set Region to China (Hangzhou) and Name to hangzhou-router.

  3. Repeat the following steps for the two transit routers in the China (Shanghai) and China (Hangzhou) regions to connect the VPCs to the transit routers.

    1. On the Instances page, click the name of the CEN instance that you created in Substep 1. On the Basic Settings tab, click the ID of a created transit router to go to the transit router details page.

    2. On the Intra-region Connections tab, click Create Connection. Select VPC for Network Type and select the VPC that corresponds to the region for Network Instance.

    3. Use the default values for other parameters and click OK.

  4. Set the cross-region network bandwidth.

    1. Click the name of a transit router that you created in Substep 2. In the upper-right corner of the page, click Create Connection.

    2. On the Connection with Peer Network Instance page, configure the related parameters and click OK.

      The following figure shows an example connection from Hangzhou to Shanghai. Region indicates the current region and Peer Region indicates the destination region. For more information about the configuration items, see Manage inter-region connections. 2023-09-27_15-11-25.png

      After the instance is created, you can view the instance on the Cross-region Connections tab.

  5. Add security group rules.

    In this example, the Flannel network plug-in is used. Add the pod CIDR block of the ack-shanghai cluster to the security group of the ack-hangzhou cluster and vice versa. This allows inbound communication from IP addresses within the pod CIDR block of the peer cluster.

    Note

    If the Terway network plug-in is used, add the vSwitch CIDR block of one cluster to the security group of the other cluster and vice versa. You can log on to the VPC console and obtain the CIDR block of a vSwitch from the IPv4 CIDR Block column on the vSwitch page.

    1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

    2. On the Clusters page, click the name of the ack-shanghai cluster. On the Basic Information tab of the Cluster Information page, view the pod CIDR block of the ack-shanghai cluster.

    3. On the Clusters page, find the ack-hangzhou cluster and click Details in the Actions column.

    4. On the Cluster Information page, click the Cluster Resources tab. Then, click the security group ID next to Security Group.

    5. On the Security Group Details tab, click Add Rule on the Inbound tab.

    6. Set Protocol Type to All and Source to the pod CIDR block of the ack-shanghai cluster. Retain other default values. Then, click Save in the Actions column.

    7. Repeat the preceding substeps to view the pod CIDR block of the ack-hangzhou cluster. Then, add the pod CIDR block of the ack-hangzhou cluster to the security group of the ack-shanghai cluster.

    8. Log on to an ECS instance in the ack-hangzhou cluster and run the ping command to check whether the ECS instance in the ack-hangzhou cluster can communicate with any ECS instance in the ack-shanghai cluster.

Step 3: Add the clusters to an ASM instance and create a managed ingress gateway

  1. Add the ack-hangzhou and ack-shanghai clusters that you created to an ASM instance. For more information, see Add a cluster to an ASM instance.

  2. Use the following YAML file to create a managed ingress gateway. For more information, see Create an ingress gateway.

    Expand to view the YAML file of the managed ingress gateway

    apiVersion: istio.alibabacloud.com/v1beta1
    kind: IstioGateway
    metadata:
      annotations:
        asm.alibabacloud.com/managed-by-asm: 'true'
      name: ingressgateway
      namespace: istio-system
    spec:
      gatewayType: ingress
      dnsPolicy: ClusterFirst
      externalTrafficPolicy: Local
      hostNetwork: false
      ports:
      - name: http
        port: 80
        protocol: TCP
        targetPort: 80
      - name: https
        port: 443
        protocol: TCP
        targetPort: 443
      replicaCount: 1
      resources:
        limits:
          cpu: '2'
          memory: 2G
        requests:
          cpu: 200m
          memory: 256Mi
      rollingMaxSurge: 100%
      rollingMaxUnavailable: 25%
      runAsRoot: true
      serviceType: LoadBalancer

Step 4: Deploy the Bookinfo application

  1. Deploy the Bookinfo application in the ack-hangzhou cluster.

    1. Use kubectl to connect to the ack-hangzhou cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

    2. Create an ack-hangzhou-k8s.yaml file that contains the following content:

      Expand to view the ack-hangzhou-k8s.yaml file

      # Details service
      apiVersion: v1
      kind: Service
      metadata:
        name: details
        labels:
          app: details
          service: details
      spec:
        ports:
        - port: 9080
          name: http
        selector:
          app: details
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: bookinfo-details
        labels:
          account: details
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: details-v1
        labels:
          app: details
          version: v1
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: details
            version: v1
        template:
          metadata:
            labels:
              app: details
              version: v1
          spec:
            serviceAccountName: bookinfo-details
            containers:
            - name: details
              image: docker.io/istio/examples-bookinfo-details-v1:1.16.2
              imagePullPolicy: IfNotPresent
              ports:
              - containerPort: 9080
              securityContext:
                runAsUser: 1000
      ---
      # Ratings service
      apiVersion: v1
      kind: Service
      metadata:
        name: ratings
        labels:
          app: ratings
          service: ratings
      spec:
        ports:
        - port: 9080
          name: http
        selector:
          app: ratings
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: bookinfo-ratings
        labels:
          account: ratings
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: ratings-v1
        labels:
          app: ratings
          version: v1
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: ratings
            version: v1
        template:
          metadata:
            labels:
              app: ratings
              version: v1
          spec:
            serviceAccountName: bookinfo-ratings
            containers:
            - name: ratings
              image: docker.io/istio/examples-bookinfo-ratings-v1:1.16.2
              imagePullPolicy: IfNotPresent
              ports:
              - containerPort: 9080
              securityContext:
                runAsUser: 1000
      ---
      # Reviews service
      apiVersion: v1
      kind: Service
      metadata:
        name: reviews
        labels:
          app: reviews
          service: reviews
      spec:
        ports:
        - port: 9080
          name: http
        selector:
          app: reviews
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: bookinfo-reviews
        labels:
          account: reviews
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: reviews-v1
        labels:
          app: reviews
          version: v1
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: reviews
            version: v1
        template:
          metadata:
            labels:
              app: reviews
              version: v1
          spec:
            serviceAccountName: bookinfo-reviews
            containers:
            - name: reviews
              image: docker.io/istio/examples-bookinfo-reviews-v1:1.16.2
              imagePullPolicy: IfNotPresent
              env:
              - name: LOG_DIR
                value: "/tmp/logs"
              ports:
              - containerPort: 9080
              volumeMounts:
              - name: tmp
                mountPath: /tmp
              - name: wlp-output
                mountPath: /opt/ibm/wlp/output
              securityContext:
                runAsUser: 1000
            volumes:
            - name: wlp-output
              emptyDir: {}
            - name: tmp
              emptyDir: {}
      ---
      # Productpage services
      apiVersion: v1
      kind: Service
      metadata:
        name: productpage
        labels:
          app: productpage
          service: productpage
      spec:
        ports:
        - port: 9080
          name: http
        selector:
          app: productpage
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: bookinfo-productpage
        labels:
          account: productpage
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: productpage-v1
        labels:
          app: productpage
          version: v1
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: productpage
            version: v1
        template:
          metadata:
            labels:
              app: productpage
              version: v1
          spec:
            serviceAccountName: bookinfo-productpage
            containers:
            - name: productpage
              image: docker.io/istio/examples-bookinfo-productpage-v1:1.16.2
              imagePullPolicy: IfNotPresent
              ports:
              - containerPort: 9080
              volumeMounts:
              - name: tmp
                mountPath: /tmp
              securityContext:
                runAsUser: 1000
            volumes:
            - name: tmp
              emptyDir: {}
      ---
      
    3. Run the following command to deploy the Bookinfo application in the ack-hangzhou cluster:

      kubectl apply -f ack-hangzhou-k8s.yaml
  2. Deploy the Bookinfo application in the ack-shanghai cluster.

    1. Use kubectl to connect to the ack-shanghai cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

      Note

      When you use kubectl to connect to the ack-shanghai cluster, you must switch the kubeconfig file of the ack-hangzhou cluster to that of the ack-shanghai cluster.

    2. Create an ack-shanghai.yaml file that contains the following content:

      Expand to view the ack-shanghai.yaml file

      # Details service
      apiVersion: v1
      kind: Service
      metadata:
        name: details
        labels:
          app: details
          service: details
      spec:
        ports:
        - port: 9080
          name: http
        selector:
          app: details
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: bookinfo-details
        labels:
          account: details
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: details-v1
        labels:
          app: details
          version: v1
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: details
            version: v1
        template:
          metadata:
            labels:
              app: details
              version: v1
          spec:
            serviceAccountName: bookinfo-details
            containers:
            - name: details
              image: docker.io/istio/examples-bookinfo-details-v1:1.16.2
              imagePullPolicy: IfNotPresent
              ports:
              - containerPort: 9080
              securityContext:
                runAsUser: 1000
      ---
      # Ratings service
      apiVersion: v1
      kind: Service
      metadata:
        name: ratings
        labels:
          app: ratings
          service: ratings
      spec:
        ports:
        - port: 9080
          name: http
        selector:
          app: ratings
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: bookinfo-ratings
        labels:
          account: ratings
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: ratings-v1
        labels:
          app: ratings
          version: v1
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: ratings
            version: v1
        template:
          metadata:
            labels:
              app: ratings
              version: v1
          spec:
            serviceAccountName: bookinfo-ratings
            containers:
            - name: ratings
              image: docker.io/istio/examples-bookinfo-ratings-v1:1.16.2
              imagePullPolicy: IfNotPresent
              ports:
              - containerPort: 9080
              securityContext:
                runAsUser: 1000
      ---
      # Reviews service
      apiVersion: v1
      kind: Service
      metadata:
        name: reviews
        labels:
          app: reviews
          service: reviews
      spec:
        ports:
        - port: 9080
          name: http
        selector:
          app: reviews
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: bookinfo-reviews
        labels:
          account: reviews
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: reviews-v2
        labels:
          app: reviews
          version: v2
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: reviews
            version: v2
        template:
          metadata:
            labels:
              app: reviews
              version: v2
          spec:
            serviceAccountName: bookinfo-reviews
            containers:
            - name: reviews
              image: docker.io/istio/examples-bookinfo-reviews-v2:1.16.2
              imagePullPolicy: IfNotPresent
              env:
              - name: LOG_DIR
                value: "/tmp/logs"
              ports:
              - containerPort: 9080
              volumeMounts:
              - name: tmp
                mountPath: /tmp
              - name: wlp-output
                mountPath: /opt/ibm/wlp/output
              securityContext:
                runAsUser: 1000
            volumes:
            - name: wlp-output
              emptyDir: {}
            - name: tmp
              emptyDir: {}
      ---
      # Productpage services
      apiVersion: v1
      kind: Service
      metadata:
        name: productpage
        labels:
          app: productpage
          service: productpage
      spec:
        ports:
        - port: 9080
          name: http
        selector:
          app: productpage
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: bookinfo-productpage
        labels:
          account: productpage
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: productpage-v1
        labels:
          app: productpage
          version: v1
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: productpage
            version: v1
        template:
          metadata:
            labels:
              app: productpage
              version: v1
          spec:
            serviceAccountName: bookinfo-productpage
            containers:
            - name: productpage
              image: docker.io/istio/examples-bookinfo-productpage-v1:1.16.2
              imagePullPolicy: IfNotPresent
              ports:
              - containerPort: 9080
              volumeMounts:
              - name: tmp
                mountPath: /tmp
              securityContext:
                runAsUser: 1000
            volumes:
            - name: tmp
              emptyDir: {}
      ---
      	  
    3. Run the following command to deploy the Bookinfo application in the ack-shanghai cluster:

      kubectl apply -f ack-shanghai.yaml
    4. Use kubectl to connect to the ASM instance. For more information, see Use kubectl on the control plane to access Istio resources.

      Note

      When you use kubectl to connect to the ASM instance, you must switch the kubeconfig file of the ack-shanghai cluster to that of the ASM instance.

  3. Create a routing rule in ASM.

    1. Create an asm.yaml file that contains the following content:

      Expand to view the asm.yaml file

      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: bookinfo-gateway
      spec:
        selector:
          istio: ingressgateway # use istio default controller
        servers:
        - port:
            number: 80
            name: http
            protocol: HTTP
          hosts:
          - "*"
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: bookinfo
      spec:
        hosts:
        - "*"
        gateways:
        - bookinfo-gateway
        http:
        - match:
          - uri:
              exact: /productpage
          - uri:
              prefix: /static
          - uri:
              exact: /login
          - uri:
              exact: /logout
          - uri:
              prefix: /api/v1/products
          route:
          - destination:
              host: productpage
              port:
                number: 9080
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: productpage
      spec:
        host: productpage
        subsets:
        - name: v1
          labels:
            version: v1
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: reviews
      spec:
        host: reviews
        subsets:
        - name: v1
          labels:
            version: v1
        - name: v2
          labels:
            version: v2
        - name: v3
          labels:
            version: v3
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: ratings
      spec:
        host: ratings
        subsets:
        - name: v1
          labels:
            version: v1
        - name: v2
          labels:
            version: v2
        - name: v2-mysql
          labels:
            version: v2-mysql
        - name: v2-mysql-vm
          labels:
            version: v2-mysql-vm
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: details
      spec:
        host: details
        subsets:
        - name: v1
          labels:
            version: v1
        - name: v2
          labels:
            version: v2
      ---
      	  
    2. Run the following command to create a routing rule:

      kubectl apply -f asm.yaml
  4. Verify whether the Bookinfo application is deployed.

    1. Log on to the ACK console.

    2. In the left-side navigation pane of the ACK console, click Clusters.

    3. On the Clusters page, find the ack-hangzhou cluster and click Details in the Actions column.

    4. In the left-side navigation pane of the details page, choose Network > Services

    5. At the top of the Services page, select istio-system from the Namespace drop-down list. Find the ingress gateway named istio-ingressgateway and view the IP address whose port is 80 in the External IP column.

    6. Enter IP address of the ingress gateway/productpage in the address bar of your browser.

      Refresh the page multiple times. The following images alternately appear on the screen.review1review1

Step 5: Use the cross-region traffic distribution and failover features

Configure cross-region traffic distribution

  1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

  2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.

  3. On the Base Information page, click Enable locality traffic distribution on the right of Locality-Failover.

    Note

    If you have enabled cross-region failover, you must disable cross-region failover before you can enable cross-region traffic distribution.

  4. In the Locality-Traffic-Distribution dialog box, set the Policy parameter to cn-hangzhou and click New Policy.

  5. Click the 展开 icon and then the 添加 icon. Set To to cn-hangzhou and Weight to 90%.

  6. Click the 添加 icon, set To to cn-shanghai and Weight to 10%, and then click OK.

  7. Run the following command to request the Bookinfo application 10 times to verify whether the cross-region traffic distribution is successful:

    for ((i=1;i<=10;i++));do curl http://<Port 80 of the IP address of the ingress gateway in the ack-hangzhou cluster>/productpage 2>&1|grep full.stars;done

    Expected output:

    <!-- full stars: -->
    <!-- full stars: -->

    The output indicates that 10 access requests are made and two rows of full stars output are returned. This indicates that 9 of the 10 requests are routed to the v1 version of the reviews service in the ack-hangzhou cluster and 1 request is routed to the v2 version of the reviews service in the ack-shanghai cluster. Traffic is routed to different clusters based on the weights of the clusters.

Configure cross-region failover

  1. Disable the reviews service in the ack-hangzhou cluster.

    1. Log on to the ACK console.

    2. In the left-side navigation pane of the ACK console, click Clusters.

    3. In the left-side navigation pane of the details page, choose Workloads > Deployments.

    4. On the Deployments page, select default from the Namespace drop-down list and click Scale in the Actions column of reviews-v1.

    5. In the Scale dialog box, set the Desired Number of Pods parameter to 0 and click OK.

  2. Configure a destination rule.

    Configure the following destination rule: If the reviews service cannot be accessed within 1 second, the reviews service will be ejected for 1 minute.

    1. Log on to the ASM console.

    2. In the left-side navigation pane, choose Service Mesh > Mesh Management.

    3. On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column.

    4. On the details page of the ASM instance, choose Traffic Management Center > DestinationRule in the left-side navigation pane.

    5. On the DestinationRule page, find the reviews service and click YAML in the Actions column.

    6. In the Edit dialog box, copy the following content to the code editor and click OK.

      spec:
        ......
        trafficPolicy:
          connectionPool:
            http:
              maxRequestsPerConnection: 1
          outlierDetection:
            baseEjectionTime: 1m
            consecutive5xxErrors: 1
            interval: 1s
      • maxRequestsPerConnection: specifies the maximum number of requests per connection.

      • baseEjectionTime: specifies the minimum ejection duration.

      • consecutive5xxErrors: specifies the number of consecutive errors.

      • interval: specifies the time interval for ejection analysis.

  3. Enable cross-region failover.

    1. On the details page of the ASM instance, choose ASM Instance > Base Information in the left-side navigation pane.

    2. On the Base Information page, click Enable Locality-Failover on the right of Locality-Failover.

      Note

      If you have enabled cross-region traffic distribution, you must disable cross-region traffic distribution before you can enable cross-region failover.

    3. In the Locality Load-Balance dialog box, set the Failover to parameter to cn-hangzhou if the From parameter is set to cn-shanghai. Set the Failover to parameter to cn-shanghai if the From parameter is set to cn-hangzhou. Then, click Submit.

  4. Run the following command to request the Bookinfo application 10 times and record the number of requests that are routed to the v2 version of the reviews service:

    for ((i=1;i<=10;i++));do curl http://<Port 80 of the IP address of the ingress gateway in the ack-hangzhou cluster>/productpage 2>&1|grep full.stars;done | wc -l

    Expected output:

    20

    The output indicates that 10 access requests are made and 20 rows of results are returned. This is because a two-row result that contains full stars is returned each time a request is routed to the v2 version of the reviews service. This indicates that all 10 requests are routed to the v2 version of the reviews service in the ack-shanghai cluster, and the cross-region failover is successful.

FAQ

I connect the VPCs of Kubernetes clusters by using CEN. However, an error message is displayed when I add the clusters to the ASM instance. What do I do?

If your clusters reside in different regions, you must purchase a cross-region data transfer plan and configure valid settings for cross-region data transfer when you connect the VPCs of the clusters by using CEN. Otherwise, the ASM instance fails to connect to the clusters on the data plane. In this case, you fail to add the clusters to the ASM instance.

To resolve this issue, you need to reconfigure valid settings for cross-region data transfer in the CEN console to connect the VPCs of the clusters by using CEN. For more information, see Step 2: Use CEN to implement cross-region VPC communication.