Service Mesh (ASM) provides cross-region traffic distribution and failover capabilities for applications. The cross-region traffic distribution feature implements cross-region load balancing by routing traffic to multiple clusters based on their weights. The cross-region failover feature implements cross-region disaster recovery by transferring traffic from a faulty region to another region. This topic describes how to use the cross-region failover and traffic distribution features to implement cross-region disaster recovery and load balancing. In the example of this topic, the Bookinfo application is used.
Plan a network
Before you use ASM, you must complete network configurations for ASM. This involves the CIDR blocks and names of vSwitches, virtual private clouds (VPCs), and clusters. In this example, a network is created based on the following plan:
For more information about how to plan CIDR blocks for multiple clusters on the data plane, see Plan CIDR blocks for multiple clusters on the data plane.
Network plan for vSwitches and VPCs
vSwitches
ImportantTo prevent route conflicts when you use Cloud Enterprise Network (CEN) to connect VPCs, specify a unique CIDR block for each vSwitch.
Category
vSwitch
VPC
IPv4 CIDR block
Cluster
vpc-hangzhou-switch-1
vpc-hangzhou
20.0.0.0/16
vpc-shanghai-switch-1
vpc-shanghai
21.0.0.0/16
ASM instance
vpc-hangzhou-switch-2
vpc-hangzhou2
192.168.0.0/24
VPCs
Category
VPC
Region
IPv4 CIDR block
Cluster
vpc-hangzhou
cn-hangzhou
20.0.0.0/8
vpc-shanghai
cn-shanghai
21.0.0.0/8
ASM instance
vpc-hangzhou2
cn-hangzhou
192.168.0.0/16
Network plan for pods and services in clusters
Cluster
Region
VPC
Pod CIDR
Service CIDR
ack-hangzhou
cn-hangzhou
vpc-hangzhou
10.0.0.0/16
172.16.0.0/16
ack-shanghai
cn-shanghai
vpc-shanghai
10.1.0.0/16
172.17.0.0/16
Step 1: Create clusters in different regions
Create two vSwitches in the China (Hangzhou) and China (Shanghai) regions based on the preceding plan, and then create VPCs that are associated with the vSwitches. For more information, see Create a vSwitch and Create a VPC and a vSwitch.
Use the VPCs that you created and the preceding network plan to create clusters in the China (Hangzhou) and China (Shanghai) regions. For more information, see Create an ACK managed cluster.
Create an ASM instance in the China (Hangzhou) region based on the preceding network plan. For more information, see Create an ASM instance.
Step 2: Use CEN to implement cross-region VPC communication
You can connect the VPCs among clusters and those between the clusters and the ASM instance by using CEN.
Log on to the CEN console and create a CEN instance. For more information, see Create a CEN instance.
Create transit routers.
On the Instances page, click the name of the CEN instance that you created in Substep 1. On the Basic Settings tab, click Create Transit Router.
In the Create Transit Router dialog box, select a region from the Region drop-down list, specify Name, and then click OK.
In this example, two transit routers are configured:
For one transit router, set Region to China (Shanghai) and Name to shanghai-router.
For the other transit router, set Region to China (Hangzhou) and Name to hangzhou-router.
Repeat the following steps for the two transit routers in the China (Shanghai) and China (Hangzhou) regions to connect the VPCs to the transit routers.
On the Instances page, click the name of the CEN instance that you created in Substep 1. On the Basic Settings tab, click the ID of a created transit router to go to the transit router details page.
On the Intra-region Connections tab, click Create Connection. Select VPC for Network Type and select the VPC that corresponds to the region for Network Instance.
Use the default values for other parameters and click OK.
Set the cross-region network bandwidth.
Click the name of a transit router that you created in Substep 2. In the upper-right corner of the page, click Create Connection.
On the Connection with Peer Network Instance page, configure the related parameters and click OK.
The following figure shows an example connection from Hangzhou to Shanghai. Region indicates the current region and Peer Region indicates the destination region. For more information about the configuration items, see Manage inter-region connections.
After the instance is created, you can view the instance on the Cross-region Connections tab.
Add security group rules.
In this example, the Flannel network plug-in is used. Add the pod CIDR block of the ack-shanghai cluster to the security group of the ack-hangzhou cluster and vice versa. This allows inbound communication from IP addresses within the pod CIDR block of the peer cluster.
NoteIf the Terway network plug-in is used, add the vSwitch CIDR block of one cluster to the security group of the other cluster and vice versa. You can log on to the VPC console and obtain the CIDR block of a vSwitch from the IPv4 CIDR Block column on the vSwitch page.
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of the ack-shanghai cluster. On the Basic Information tab of the Cluster Information page, view the pod CIDR block of the ack-shanghai cluster.
On the Clusters page, find the ack-hangzhou cluster and click Details in the Actions column.
On the Cluster Information page, click the Basic Information tab. Then, click the security group ID next to Security Group.
On the Security Group Details tab, click Add Rule on the Inbound tab.
Set Protocol Type to All and Source to the pod CIDR block of the ack-shanghai cluster. Retain other default values. Then, click Save in the Actions column.
Repeat the preceding substeps to view the pod CIDR block of the ack-hangzhou cluster. Then, add the pod CIDR block of the ack-hangzhou cluster to the security group of the ack-shanghai cluster.
Log on to an ECS instance in the ack-hangzhou cluster and run the ping command to check whether the ECS instance in the ack-hangzhou cluster can communicate with any ECS instance in the ack-shanghai cluster.
Step 3: Add the clusters to an ASM instance and create a managed ingress gateway
Add the ack-hangzhou and ack-shanghai clusters that you created to an ASM instance. For more information, see Add a cluster to an ASM instance.
Use the following YAML file to create a managed ingress gateway. For more information, see Create an ingress gateway.
Step 4: Deploy the Bookinfo application
Deploy the Bookinfo application in the ack-hangzhou cluster.
Use kubectl to connect to the ack-hangzhou cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.
Create an ack-hangzhou-k8s.yaml file that contains the following content:
Run the following command to deploy the Bookinfo application in the ack-hangzhou cluster:
kubectl apply -f ack-hangzhou-k8s.yaml
Deploy the Bookinfo application in the ack-shanghai cluster.
Use kubectl to connect to the ack-shanghai cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.
NoteWhen you use kubectl to connect to the ack-shanghai cluster, you must switch the kubeconfig file of the ack-hangzhou cluster to that of the ack-shanghai cluster.
Create an ack-shanghai.yaml file that contains the following content:
Run the following command to deploy the Bookinfo application in the ack-shanghai cluster:
kubectl apply -f ack-shanghai.yaml
Use kubectl to connect to the ASM instance. For more information, see Use kubectl on the control plane to access Istio resources.
NoteWhen you use kubectl to connect to the ASM instance, you must switch the kubeconfig file of the ack-shanghai cluster to that of the ASM instance.
Create a routing rule in ASM.
Create an asm.yaml file that contains the following content:
Run the following command to create a routing rule:
kubectl apply -f asm.yaml
Verify whether the Bookinfo application is deployed.
Log on to the ACK console.
In the left-side navigation pane of the ACK console, click Clusters.
On the Clusters page, find the ack-hangzhou cluster and click Details in the Actions column.
In the left-side navigation pane of the details page, choose
At the top of the Services page, select istio-system from the Namespace drop-down list. Find the ingress gateway named istio-ingressgateway and view the IP address whose port is 80 in the External IP column.
Enter IP address of the ingress gateway/productpage in the address bar of your browser.
Refresh the page multiple times. The following images alternately appear on the screen.
Step 5: Use the cross-region traffic distribution and failover features
Configure cross-region traffic distribution
Log on to the ASM console. In the left-side navigation pane, choose .
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose .
On the Base Information page, click Enable locality traffic distribution on the right of Locality-Failover.
NoteIf you have enabled cross-region failover, you must disable cross-region failover before you can enable cross-region traffic distribution.
In the Locality-Traffic-Distribution dialog box, set the Policy parameter to cn-hangzhou and click New Policy.
Click the icon and then the icon. Set To to cn-hangzhou and Weight to 90%.
Click the icon, set To to cn-shanghai and Weight to 10%, and then click OK.
Run the following command to request the Bookinfo application 10 times to verify whether the cross-region traffic distribution is successful:
for ((i=1;i<=10;i++));do curl http://<Port 80 of the IP address of the ingress gateway in the ack-hangzhou cluster>/productpage 2>&1|grep full.stars;done
Expected output:
<!-- full stars: --> <!-- full stars: -->
The output indicates that 10 access requests are made and two rows of
full stars
output are returned. This indicates that 9 of the 10 requests are routed to the v1 version of the reviews service in the ack-hangzhou cluster and 1 request is routed to the v2 version of the reviews service in the ack-shanghai cluster. Traffic is routed to different clusters based on the weights of the clusters.
Configure cross-region failover
Disable the reviews service in the ack-hangzhou cluster.
Log on to the ACK console.
In the left-side navigation pane of the ACK console, click Clusters.
In the left-side navigation pane of the details page, choose .
On the Deployments page, select default from the Namespace drop-down list and click Scale in the Actions column of reviews-v1.
In the Scale dialog box, set the Desired Number of Pods parameter to 0 and click OK.
Configure a destination rule.
Configure the following destination rule: If the reviews service cannot be accessed within 1 second, the reviews service will be ejected for 1 minute.
Log on to the ASM console.
In the left-side navigation pane, choose .
On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column.
On the details page of the ASM instance, choose in the left-side navigation pane.
On the DestinationRule page, find the reviews service and click YAML in the Actions column.
In the Edit dialog box, copy the following content to the code editor and click OK.
spec: ...... trafficPolicy: connectionPool: http: maxRequestsPerConnection: 1 outlierDetection: baseEjectionTime: 1m consecutive5xxErrors: 1 interval: 1s
maxRequestsPerConnection: specifies the maximum number of requests per connection.
baseEjectionTime: specifies the minimum ejection duration.
consecutive5xxErrors: specifies the number of consecutive errors.
interval: specifies the time interval for ejection analysis.
Enable cross-region failover.
On the details page of the ASM instance, choose in the left-side navigation pane.
On the Base Information page, click Enable Locality-Failover on the right of Locality-Failover.
NoteIf you have enabled cross-region traffic distribution, you must disable cross-region traffic distribution before you can enable cross-region failover.
In the Locality Load-Balance dialog box, set the Failover to parameter to cn-hangzhou if the From parameter is set to cn-shanghai. Set the Failover to parameter to cn-shanghai if the From parameter is set to cn-hangzhou. Then, click Submit.
Run the following command to request the Bookinfo application 10 times and record the number of requests that are routed to the v2 version of the reviews service:
for ((i=1;i<=10;i++));do curl http://<Port 80 of the IP address of the ingress gateway in the ack-hangzhou cluster>/productpage 2>&1|grep full.stars;done | wc -l
Expected output:
20
The output indicates that 10 access requests are made and 20 rows of results are returned. This is because a two-row result that contains
full stars
is returned each time a request is routed to the v2 version of the reviews service. This indicates that all 10 requests are routed to the v2 version of the reviews service in the ack-shanghai cluster, and the cross-region failover is successful.
FAQ
I connect the VPCs of Kubernetes clusters by using CEN. However, an error message is displayed when I add the clusters to the ASM instance. What do I do?
If your clusters reside in different regions, you must purchase a cross-region data transfer plan and configure valid settings for cross-region data transfer when you connect the VPCs of the clusters by using CEN. Otherwise, the ASM instance fails to connect to the clusters on the data plane. In this case, you fail to add the clusters to the ASM instance.
To resolve this issue, you need to reconfigure valid settings for cross-region data transfer in the CEN console to connect the VPCs of the clusters by using CEN. For more information, see Step 2: Use CEN to implement cross-region VPC communication.