Implement cross-region disaster recovery and load balancing by using multiple clusters - Container Service for Kubernetes

Service Mesh (ASM) provides cross-region traffic distribution and failover capabilities for applications. The cross-region traffic distribution feature implements cross-region load balancing by routing traffic to multiple clusters based on their weights. The cross-region failover feature implements cross-region disaster recovery by transferring traffic from a faulty region to another region. This topic describes how to use the cross-region failover and traffic distribution features to implement cross-region disaster recovery and load balancing. In the example of this topic, the Bookinfo application is used.

Plan a network

Before you use ASM, you must complete network configurations for ASM. This involves the CIDR blocks and names of vSwitches, virtual private clouds (VPCs), and clusters. In this example, a network is created based on the following plan:

Note

For more information about how to plan CIDR blocks for multiple clusters on the data plane, see Plan CIDR blocks for multiple clusters on the data plane.

Network plan for vSwitches and VPCs
- vSwitches
  Important
  To prevent route conflicts when you use Cloud Enterprise Network (CEN) to connect VPCs, specify a unique CIDR block for each vSwitch.
  Category
  vSwitch
  VPC
  IPv4 CIDR block
  Cluster
  vpc-hangzhou-switch-1
  vpc-hangzhou
  20.0.0.0/16
  vpc-shanghai-switch-1
  vpc-shanghai
  21.0.0.0/16
  ASM instance
  vpc-hangzhou-switch-2
  vpc-hangzhou2
  192.168.0.0/24
- VPCs
  Category
  VPC
  Region
  IPv4 CIDR block
  Cluster
  vpc-hangzhou
  cn-hangzhou
  20.0.0.0/8
  vpc-shanghai
  cn-shanghai
  21.0.0.0/8
  ASM instance
  vpc-hangzhou2
  cn-hangzhou
  192.168.0.0/16
Network plan for pods and services in clusters
Cluster
Region
VPC
Pod CIDR
Service CIDR
ack-hangzhou
cn-hangzhou
vpc-hangzhou
10.0.0.0/16
172.16.0.0/16
ack-shanghai
cn-shanghai
vpc-shanghai
10.1.0.0/16
172.17.0.0/16

Step 1: Create clusters in different regions

Create two vSwitches in the China (Hangzhou) and China (Shanghai) regions based on the preceding plan, and then create VPCs that are associated with the vSwitches. For more information, see Create a vSwitch and Create a VPC and a vSwitch.
Use the VPCs that you created and the preceding network plan to create clusters in the China (Hangzhou) and China (Shanghai) regions. For more information, see Create an ACK managed cluster.
Create an ASM instance in the China (Hangzhou) region based on the preceding network plan. For more information, see Create an ASM instance.

Step 2: Use CEN to implement cross-region VPC communication

You can connect the VPCs among clusters and those between the clusters and the ASM instance by using CEN.

Log on to the CEN console and create a CEN instance. For more information, see Create a CEN instance.
Create transit routers.
1. On the Instances page, click the name of the CEN instance that you created in Substep 1. On the Basic Settings tab, click Create Transit Router.
2. In the Create Transit Router dialog box, select a region from the Region drop-down list, specify Name, and then click OK.
In this example, two transit routers are configured:
- For one transit router, set Region to China (Shanghai) and Name to shanghai-router.
- For the other transit router, set Region to China (Hangzhou) and Name to hangzhou-router.
Repeat the following steps for the two transit routers in the China (Shanghai) and China (Hangzhou) regions to connect the VPCs to the transit routers.
1. On the Instances page, click the name of the CEN instance that you created in Substep 1. On the Basic Settings tab, click the ID of a created transit router to go to the transit router details page.
2. On the Intra-region Connections tab, click Create Connection. Select VPC for Network Type and select the VPC that corresponds to the region for Network Instance.
3. Use the default values for other parameters and click OK.
Set the cross-region network bandwidth.
1. Click the name of a transit router that you created in Substep 2. In the upper-right corner of the page, click Create Connection.
2. On the Connection with Peer Network Instance page, configure the related parameters and click OK.
  The following figure shows an example connection from Hangzhou to Shanghai. Region indicates the current region and Peer Region indicates the destination region. For more information about the configuration items, see Manage inter-region connections.
  After the instance is created, you can view the instance on the Cross-region Connections tab.
Add security group rules.
In this example, the Flannel network plug-in is used. Add the pod CIDR block of the ack-shanghai cluster to the security group of the ack-hangzhou cluster and vice versa. This allows inbound communication from IP addresses within the pod CIDR block of the peer cluster.
Note
If the Terway network plug-in is used, add the vSwitch CIDR block of one cluster to the security group of the other cluster and vice versa. You can log on to the VPC console and obtain the CIDR block of a vSwitch from the IPv4 CIDR Block column on the vSwitch page.
1. Log on to the ACK console. In the left-side navigation pane, click Clusters.
2. On the Clusters page, click the name of the ack-shanghai cluster. On the Basic Information tab of the Cluster Information page, view the pod CIDR block of the ack-shanghai cluster.
3. On the Clusters page, find the ack-hangzhou cluster and click Details in the Actions column.
4. On the Cluster Information page, click the Basic Information tab. Then, click the security group ID next to Security Group.
5. On the Security Group Details tab, click Add Rule on the Inbound tab.
6. Set Protocol Type to All and Source to the pod CIDR block of the ack-shanghai cluster. Retain other default values. Then, click Save in the Actions column.
7. Repeat the preceding substeps to view the pod CIDR block of the ack-hangzhou cluster. Then, add the pod CIDR block of the ack-hangzhou cluster to the security group of the ack-shanghai cluster.
8. Log on to an ECS instance in the ack-hangzhou cluster and run the ping command to check whether the ECS instance in the ack-hangzhou cluster can communicate with any ECS instance in the ack-shanghai cluster.

Step 3: Add the clusters to an ASM instance and create a managed ingress gateway

Add the ack-hangzhou and ack-shanghai clusters that you created to an ASM instance. For more information, see Add a cluster to an ASM instance.

Use the following YAML file to create a managed ingress gateway. For more information, see Create an ingress gateway.

Expand to view the YAML file of the managed ingress gateway

apiVersion: istio.alibabacloud.com/v1beta1
kind: IstioGateway
metadata:
  annotations:
    asm.alibabacloud.com/managed-by-asm: 'true'
  name: ingressgateway
  namespace: istio-system
spec:
  gatewayType: ingress
  dnsPolicy: ClusterFirst
  externalTrafficPolicy: Local
  hostNetwork: false
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  replicaCount: 1
  resources:
    limits:
      cpu: '2'
      memory: 2G
    requests:
      cpu: 200m
      memory: 256Mi
  rollingMaxSurge: 100%
  rollingMaxUnavailable: 25%
  runAsRoot: true
  serviceType: LoadBalancer

Step 4: Deploy the Bookinfo application

Deploy the Bookinfo application in the ack-hangzhou cluster.

Use kubectl to connect to the ack-hangzhou cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Create an ack-hangzhou-k8s.yaml file that contains the following content:

Expand to view the ack-hangzhou-k8s.yaml file

# Details service
apiVersion: v1
kind: Service
metadata:
  name: details
  labels:
    app: details
    service: details
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: details
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-details
  labels:
    account: details
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: details-v1
  labels:
    app: details
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: details
      version: v1
  template:
    metadata:
      labels:
        app: details
        version: v1
    spec:
      serviceAccountName: bookinfo-details
      containers:
      - name: details
        image: docker.io/istio/examples-bookinfo-details-v1:1.16.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        securityContext:
          runAsUser: 1000
---
# Ratings service
apiVersion: v1
kind: Service
metadata:
  name: ratings
  labels:
    app: ratings
    service: ratings
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: ratings
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-ratings
  labels:
    account: ratings
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ratings-v1
  labels:
    app: ratings
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ratings
      version: v1
  template:
    metadata:
      labels:
        app: ratings
        version: v1
    spec:
      serviceAccountName: bookinfo-ratings
      containers:
      - name: ratings
        image: docker.io/istio/examples-bookinfo-ratings-v1:1.16.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        securityContext:
          runAsUser: 1000
---
# Reviews service
apiVersion: v1
kind: Service
metadata:
  name: reviews
  labels:
    app: reviews
    service: reviews
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: reviews
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-reviews
  labels:
    account: reviews
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v1
  labels:
    app: reviews
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v1
  template:
    metadata:
      labels:
        app: reviews
        version: v1
    spec:
      serviceAccountName: bookinfo-reviews
      containers:
      - name: reviews
        image: docker.io/istio/examples-bookinfo-reviews-v1:1.16.2
        imagePullPolicy: IfNotPresent
        env:
        - name: LOG_DIR
          value: "/tmp/logs"
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: wlp-output
          mountPath: /opt/ibm/wlp/output
        securityContext:
          runAsUser: 1000
      volumes:
      - name: wlp-output
        emptyDir: {}
      - name: tmp
        emptyDir: {}
---
# Productpage services
apiVersion: v1
kind: Service
metadata:
  name: productpage
  labels:
    app: productpage
    service: productpage
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: productpage
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-productpage
  labels:
    account: productpage
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage-v1
  labels:
    app: productpage
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productpage
      version: v1
  template:
    metadata:
      labels:
        app: productpage
        version: v1
    spec:
      serviceAccountName: bookinfo-productpage
      containers:
      - name: productpage
        image: docker.io/istio/examples-bookinfo-productpage-v1:1.16.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        securityContext:
          runAsUser: 1000
      volumes:
      - name: tmp
        emptyDir: {}
---

Run the following command to deploy the Bookinfo application in the ack-hangzhou cluster:
```
kubectl apply -f ack-hangzhou-k8s.yaml
```

Deploy the Bookinfo application in the ack-shanghai cluster.

Use kubectl to connect to the ack-shanghai cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.
Note
When you use kubectl to connect to the ack-shanghai cluster, you must switch the kubeconfig file of the ack-hangzhou cluster to that of the ack-shanghai cluster.

Create an ack-shanghai.yaml file that contains the following content:

Expand to view the ack-shanghai.yaml file

# Details service
apiVersion: v1
kind: Service
metadata:
  name: details
  labels:
    app: details
    service: details
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: details
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-details
  labels:
    account: details
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: details-v1
  labels:
    app: details
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: details
      version: v1
  template:
    metadata:
      labels:
        app: details
        version: v1
    spec:
      serviceAccountName: bookinfo-details
      containers:
      - name: details
        image: docker.io/istio/examples-bookinfo-details-v1:1.16.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        securityContext:
          runAsUser: 1000
---
# Ratings service
apiVersion: v1
kind: Service
metadata:
  name: ratings
  labels:
    app: ratings
    service: ratings
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: ratings
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-ratings
  labels:
    account: ratings
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ratings-v1
  labels:
    app: ratings
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ratings
      version: v1
  template:
    metadata:
      labels:
        app: ratings
        version: v1
    spec:
      serviceAccountName: bookinfo-ratings
      containers:
      - name: ratings
        image: docker.io/istio/examples-bookinfo-ratings-v1:1.16.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        securityContext:
          runAsUser: 1000
---
# Reviews service
apiVersion: v1
kind: Service
metadata:
  name: reviews
  labels:
    app: reviews
    service: reviews
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: reviews
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-reviews
  labels:
    account: reviews
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v2
  labels:
    app: reviews
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v2
  template:
    metadata:
      labels:
        app: reviews
        version: v2
    spec:
      serviceAccountName: bookinfo-reviews
      containers:
      - name: reviews
        image: docker.io/istio/examples-bookinfo-reviews-v2:1.16.2
        imagePullPolicy: IfNotPresent
        env:
        - name: LOG_DIR
          value: "/tmp/logs"
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: wlp-output
          mountPath: /opt/ibm/wlp/output
        securityContext:
          runAsUser: 1000
      volumes:
      - name: wlp-output
        emptyDir: {}
      - name: tmp
        emptyDir: {}
---
# Productpage services
apiVersion: v1
kind: Service
metadata:
  name: productpage
  labels:
    app: productpage
    service: productpage
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: productpage
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-productpage
  labels:
    account: productpage
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage-v1
  labels:
    app: productpage
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productpage
      version: v1
  template:
    metadata:
      labels:
        app: productpage
        version: v1
    spec:
      serviceAccountName: bookinfo-productpage
      containers:
      - name: productpage
        image: docker.io/istio/examples-bookinfo-productpage-v1:1.16.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        securityContext:
          runAsUser: 1000
      volumes:
      - name: tmp
        emptyDir: {}
---

Run the following command to deploy the Bookinfo application in the ack-shanghai cluster:
```
kubectl apply -f ack-shanghai.yaml
```
Use kubectl to connect to the ASM instance. For more information, see Use kubectl on the control plane to access Istio resources.
Note
When you use kubectl to connect to the ASM instance, you must switch the kubeconfig file of the ack-shanghai cluster to that of the ASM instance.

Create a routing rule in ASM.

Create an asm.yaml file that contains the following content:

Expand to view the asm.yaml file

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
spec:
  hosts:
  - "*"
  gateways:
  - bookinfo-gateway
  http:
  - match:
    - uri:
        exact: /productpage
    - uri:
        prefix: /static
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: productpage
        port:
          number: 9080
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: productpage
spec:
  host: productpage
  subsets:
  - name: v1
    labels:
      version: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: ratings
spec:
  host: ratings
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v2-mysql
    labels:
      version: v2-mysql
  - name: v2-mysql-vm
    labels:
      version: v2-mysql-vm
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: details
spec:
  host: details
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
---

Run the following command to create a routing rule:
```
kubectl apply -f asm.yaml
```

Verify whether the Bookinfo application is deployed.
1. Log on to the ACK console.
2. In the left-side navigation pane of the ACK console, click Clusters.
3. On the Clusters page, find the ack-hangzhou cluster and click Details in the Actions column.
4. In the left-side navigation pane of the details page, choose Network > Services
5. At the top of the Services page, select istio-system from the Namespace drop-down list. Find the ingress gateway named istio-ingressgateway and view the IP address whose port is 80 in the External IP column.
6. Enter IP address of the ingress gateway/productpage in the address bar of your browser.
  Refresh the page multiple times. The following images alternately appear on the screen.

Step 5: Use the cross-region traffic distribution and failover features

Configure cross-region traffic distribution

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.
On the Base Information page, click Enable locality traffic distribution on the right of Locality-Failover.
Note
If you have enabled cross-region failover, you must disable cross-region failover before you can enable cross-region traffic distribution.
In the Locality-Traffic-Distribution dialog box, set the Policy parameter to cn-hangzhou and click New Policy.
Click the icon and then the icon. Set To to cn-hangzhou and Weight to 90%.
Click the icon, set To to cn-shanghai and Weight to 10%, and then click OK.
Run the following command to request the Bookinfo application 10 times to verify whether the cross-region traffic distribution is successful:
```
for ((i=1;i<=10;i++));do curl http://<Port 80 of the IP address of the ingress gateway in the ack-hangzhou cluster>/productpage 2>&1|grep full.stars;done
```
Expected output:
```


```
The output indicates that 10 access requests are made and two rows of full stars output are returned. This indicates that 9 of the 10 requests are routed to the v1 version of the reviews service in the ack-hangzhou cluster and 1 request is routed to the v2 version of the reviews service in the ack-shanghai cluster. Traffic is routed to different clusters based on the weights of the clusters.

Configure cross-region failover

Disable the reviews service in the ack-hangzhou cluster.
1. Log on to the ACK console.
2. In the left-side navigation pane of the ACK console, click Clusters.
3. In the left-side navigation pane of the details page, choose Workloads > Deployments.
4. On the Deployments page, select default from the Namespace drop-down list and click Scale in the Actions column of reviews-v1.
5. In the Scale dialog box, set the Desired Number of Pods parameter to 0 and click OK.
Configure a destination rule.
Configure the following destination rule: If the reviews service cannot be accessed within 1 second, the reviews service will be ejected for 1 minute.
1. Log on to the ASM console.
2. In the left-side navigation pane, choose Service Mesh > Mesh Management.
3. On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column.
4. On the details page of the ASM instance, choose Traffic Management Center > DestinationRule in the left-side navigation pane.
5. On the DestinationRule page, find the reviews service and click YAML in the Actions column.
6. In the Edit dialog box, copy the following content to the code editor and click OK.
```
spec:
  ......
  trafficPolicy:
    connectionPool:
      http:
        maxRequestsPerConnection: 1
    outlierDetection:
      baseEjectionTime: 1m
      consecutive5xxErrors: 1
      interval: 1s
```
  - maxRequestsPerConnection: specifies the maximum number of requests per connection.
  - baseEjectionTime: specifies the minimum ejection duration.
  - consecutive5xxErrors: specifies the number of consecutive errors.
  - interval: specifies the time interval for ejection analysis.
Enable cross-region failover.
1. On the details page of the ASM instance, choose ASM Instance > Base Information in the left-side navigation pane.
2. On the Base Information page, click Enable Locality-Failover on the right of Locality-Failover.
  Note
  If you have enabled cross-region traffic distribution, you must disable cross-region traffic distribution before you can enable cross-region failover.
3. In the Locality Load-Balance dialog box, set the Failover to parameter to cn-hangzhou if the From parameter is set to cn-shanghai. Set the Failover to parameter to cn-shanghai if the From parameter is set to cn-hangzhou. Then, click Submit.
Run the following command to request the Bookinfo application 10 times and record the number of requests that are routed to the v2 version of the reviews service:
```
for ((i=1;i<=10;i++));do curl http://<Port 80 of the IP address of the ingress gateway in the ack-hangzhou cluster>/productpage 2>&1|grep full.stars;done | wc -l
```
Expected output:
```
20
```
The output indicates that 10 access requests are made and 20 rows of results are returned. This is because a two-row result that contains full stars is returned each time a request is routed to the v2 version of the reviews service. This indicates that all 10 requests are routed to the v2 version of the reviews service in the ack-shanghai cluster, and the cross-region failover is successful.

FAQ

I connect the VPCs of Kubernetes clusters by using CEN. However, an error message is displayed when I add the clusters to the ASM instance. What do I do?

If your clusters reside in different regions, you must purchase a cross-region data transfer plan and configure valid settings for cross-region data transfer when you connect the VPCs of the clusters by using CEN. Otherwise, the ASM instance fails to connect to the clusters on the data plane. In this case, you fail to add the clusters to the ASM instance.

To resolve this issue, you need to reconfigure valid settings for cross-region data transfer in the CEN console to connect the VPCs of the clusters by using CEN. For more information, see Step 2: Use CEN to implement cross-region VPC communication.

Category	vSwitch	VPC	IPv4 CIDR block
Cluster	vpc-hangzhou-switch-1	vpc-hangzhou	20.0.0.0/16
Cluster	vpc-shanghai-switch-1	vpc-shanghai	21.0.0.0/16
ASM instance	vpc-hangzhou-switch-2	vpc-hangzhou2	192.168.0.0/24

Category	VPC	Region	IPv4 CIDR block
Cluster	vpc-hangzhou	cn-hangzhou	20.0.0.0/8
Cluster	vpc-shanghai	cn-shanghai	21.0.0.0/8
ASM instance	vpc-hangzhou2	cn-hangzhou	192.168.0.0/16

Cluster	Region	VPC	Pod CIDR	Service CIDR
ack-hangzhou	cn-hangzhou	vpc-hangzhou	10.0.0.0/16	172.16.0.0/16
ack-shanghai	cn-shanghai	vpc-shanghai	10.1.0.0/16	172.17.0.0/16