Use canary release to enhance update stability - Alibaba Cloud Service Mesh

A canary release is a deployment method that is used to test the performance of a new version while the original deployment version remains available. A canary release can help you ensure system stability. During a canary release, you can identify and fix errors at the earliest opportunity. This topic describes how to use canary release to enhance the stability of a Service Mesh (ASM) instance during an update.

Prerequisites

An ASM instance of Enterprise Edition or Ultimate Edition is created and the instance is of version 1.16.4.93 or later. For more information, see Create an ASM instance.
A Container Service for Kubernetes (ACK) cluster is created and added to the ASM instance. For more information, see Add a cluster to an ASM instance.
The Bookinfo application is deployed. For more information, see Deploy an application in an ACK cluster that is added to an ASM instance.

Feature description

Service Mesh of Alibaba Cloud supports revision- and label-based canary updates of a control plane of a new version in a more stable and secure manner. In this new update mode, sidecar proxies on the data plane are associated with the specific control plane versions. This way, the new version can be deployed in clusters with low risk. Before you confirm your choice, no proxies are connected to the new version. You can also gradually migrate workloads to the new independent control plane, which is called a revision and carries the istio.io/rev label.

To support revision-based updates, Istio introduces the istio.io/rev label for namespaces. The label indicates the control plane version or the destination version of sidecar proxies injected into workloads in the corresponding namespace. For example, the istio.io/rev=1-17-2 label indicates that sidecar proxies of version 1.17.2 should be injected into workloads in the namespace.

During a canary release, you can update some services to check whether the destination version meets expectations. If the destination version does not meet expectations, you can roll back the services to ensure service stability. If the canary version meets expectations, you can use the canary version as the current version and perform a rolling update to migrate all workloads to this version. Then, the update of the data plane is complete, and you can remove the old version.

Preparations

During a canary release, you need to explicitly specify the destination version of injected sidecar proxies for verification by using a namespace label. Therefore, the injection policy configuration must meet the requirements of the canary release. Perform the following steps to check whether the injection policy configuration meets the canary release requirements:

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Data Plane Component Management > Sidecar Proxy injection.
In the Injection strategy configuration management section of the Sidecar Proxy injection page, make sure that the value of Pod namespace label should meet condition: is Include istio-injection: enabled.
Note
The istio-injection: enabled label has the same semantics as the istio.io/rev: stable label. During a canary release, the istio.io/rev: stable label is used to inject sidecar proxies of the stable version to the pods in the corresponding namespaces, and the istio.io/rev: canary label is used to inject sidecar proxies of the canary version to the pods in the corresponding namespaces.
After the update, sidecar proxies can be injected even if you do not replace istio.io/rev:stable with istio-injection: enabled. This is because the semantics of the two labels are the same.
The following figure shows the ASM instance before the update.

Step 1: Update the control plane of the ASM instance

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
On the Upgrade Management page, click the Canary Upgrade tab. On the Control Plane tab, set Canary version and Create a CLB instance, and then click Submit. In the dialog box for confirming the update, confirm the operation.
During a canary release, you can skip only one minor version at most. In this example, the ASM instance is of version 1.16. You can update the ASM instance to version 1.17 or 1.18. This topic describes how to update the ASM instance to v1.17.2.37. During the deployment of the destination version of a canary release, an associated Classic Load Balancer (CLB) instance is created. We recommend that you use the default specifications of the CLB instance unless otherwise you have special requirements. For more information about the billing of CLB instances, see CLB billing overview.
The deployment of a canary version is asynchronous and takes several minutes. Wait until the related components are deployed. The current version and the canary version are shown on the following page after the new version is deployed.
The following figure outlines the ASM instance architecture at this point.

Step 2: Update the version of sidecar proxies injected into reviews-v2

An Istio control plane of v1.17.2 is deployed in the canary release manner in Step 1. This section describes how to update the version of sidecar proxies injected into reviews-v2 of the Bookinfo application to v1.17 and check whether the version meets expectations.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Global Namespace.
On the Global Namespace page, check whether the label of the default namespace is istio-injection: enabled in the Automatic Sidecar Injection column.
If the label is istio-injection: enabled, sidecar proxies of version 1.16 are injected.
On the Global Namespace page, find the default namespace and click Switch to injection of V1-17-2 sidecar proxy in the Automatic Sidecar Injection column. In the Confirm message, click OK.
Then, the label of the default namespace on the Global Namespace page is changed to istio.io/rev: canary, and istio.io/rev: canary is synchronized to the label of the default namespace on the data plane. The Global Namespace page shows that sidecar proxies of version 1.17 are injected in the default namespace. Sidecar proxies of version 1.17 will be injected into new workloads (pods) in the default namespace. The labels of other namespaces are not changed, and sidecar proxies of the current version 1.16 are injected in these namespaces.
Note
The preceding steps are used to switch the version of sidecar proxies injected in a namespace for which automatic sidecar proxy injection is enabled before the update. You can enable automatic sidecar proxy injection for namespaces for which automatic sidecar proxy injection has not been enabled. You can select the version of sidecar proxies injected based on your business requirements. Then, ASM adds the istio.io/rev:stable label or the istio.io/rev:canary label to the namespaces based on the version you select.
Perform a rolling update for the reviews-v2 workload.
1. Log on to the ACK console. In the left-side navigation pane, click Clusters.
2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, choose Workloads > Deployments.
3. On the Deployments page, find the reviews-v2 workload and choose More > Redeploy in the Actions column. In the Redeploy message, click Confirm.
On the Deployments page, click reviews-v2. Then, click the Pods tab, and check whether the pods corresponding to reviews-v2 after the rolling update are started and whether sidecar proxies of version 1.17 are injected into the new pods.
As shown in the figure, the pods corresponding to reviews-v2 after the rolling update are started, and sidecar proxies of v1.17 are injected into the new pods.
Use a browser to access the Bookinfo application and check whether the traffic meets expectations.
As shown in the following figure, reviews-v2 can be accessed, which meets expectations.
The following figure outlines the ASM instance architecture at this point.

Step 3: Roll back reviews-v2 to version 1.16

If the verification fails in Step 2 or a rollback is required for other reasons, perform the following steps:

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Global Namespace.
On the Global Namespace page, find the default namespace and click Switch to injection of V1-16-4 sidecar proxy in the Automatic Sidecar Injection column. In the Confirm message, click OK.
Before the rollback, the label of the namespace is istio.io/rev: canary, and sidecar proxies of version 1.17 are injected in the default namespace.
After the rollback, the label of the namespace is changed to istio.io/rev: stable, and istio.io/rev: stable is synchronized to the label of the default namespace on the data plane. Sidecar proxies of version 1.16 are injected in the default namespace.
Redeploy the reviews-v2 workload in the ACK console.
1. Log on to the ACK console. In the left-side navigation pane, click Clusters.
2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, choose Workloads > Deployments.
3. On the Deployments page, select default from the Namespace drop-down list in the top navigation bar. Then, find reviews-v2 and choose More > Redeploy in the Actions column. In the Redeploy message, click Confirm.
4. On the Deployments page, click reviews-v2. Then, click the Pods tab, and check whether the pods corresponding to reviews-v2 after the rolling update are started and whether sidecar proxies of version 1.16 are injected into the new pods.
  As shown in the figure, the pods corresponding to reviews-v2 after the rolling update are started, and sidecar proxies of v1.16 are injected into the new pods.

Step 4: Undo the update

After the rollback, you can undo the canary release and switch the ASM instance back to version 1.16.

Important

Before you undo the update, you must make sure that sidecar proxies of the stable version are injected in all namespaces. Otherwise, the update cannot be undone.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
On the Canary Upgrade tab of the Upgrade Management page, click Undo upgrade. In the dialog box for confirming update undo, confirm the operation.
After you click Undo upgrade, the control plane component of the canary version is deleted. In this example, the control plane of version 1.17 is deleted. The control plane component of the stable version is retained. In this example, the control plane of version 1.16 is retained.
The following figure outlines the ASM instance architecture at this point.

Step 5: Perform an update and verification again

At this time, the ASM instance is in the status before you perform the update in Step 1. In this topic, the reviews-v1, reviews-v2, and reviews-v3 workloads are used for the update and verification. Follow the preceding steps to update a workload and verify the update until you confirm that the verification succeeds.

Perform Step 1 again to deploy a canary version on the control plane in the cluster of the ASM instance.
Perform Step 2 again to check whether the injected sidecar proxies of the new version meet expectations.
For example, you can redeploy the reviews-v1, reviews-v2, and reviews-v3 deployments in the ACK console. Then, sidecar proxies of version 1.17 are injected into the pods of reviews-v1, reviews-v2, and reviews-v3.

Step 6: Use the canary version as the official version if the verification result meets expectations

The preceding operations verify that sidecar proxies of the new version (1.17) can be used for review-v1 and review-v2, and that the features meet expectations. Therefore, you can use version 1.17 as the official version.

Important

Once you use the canary version as the official version, the old version goes offline. You can use only sidecar proxies of the new version (1.17) for all existing workloads and you cannot undo the update. Make sure that you complete the verification in the New version deployment step.
After you use the canary version as the official version, sidecar proxies of version 1.17 are injected into namespaces with the istio.io/rev: stable label or the istio-injection: enabled label. The istio.io/rev: canary label becomes invalid. Therefore, when you use version 1.17 as the official version, ASM changes all the istio.io/rev: canary label to istio.io/rev: stable. Be sure to confirm the operation before you click Version switching on the Canary Upgrade tab of the Upgrade Management page.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
On the Canary Upgrade tab of the Upgrade Management page, click Version switching. In the Confirm to perform version switching? dialog box, read the prompt and click Ok.
After you successfully use the new version, sidecar proxies of version 1.16 that are already injected are retained, and the corresponding workloads are not affected. However, sidecar proxies of version 1.17 will be injected into pods that are deployed after you use the canary version. The following page is displayed after you use the canary version.
The following figure outlines the ASM instance architecture at this point.

Step 7: Update the data plane

At this point, the version of the ASM instance is 1.17. Sidecar proxies of version 1.17 will be injected into namespaces with the istio.io/rev=stable, istio.io/rev=1-17-2, or istio-injection=enabled label. You can update workloads in a rolling manner to update injected sidecar proxies to the new version 1.17. This way, you can update the data plane.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
On the Canary Upgrade tab of the Upgrade Management page, click the Data plane tab and update ASM gateways or workloads based on your business requirements.
- To update an ASM gateway, find the desired ASM gateway in the ASM Gateways section and click Rolling Upgrade in the Actions column. In the Confirm to perform rolling upgrade? dialog box, Click Ok to update the ASM gateway to version 1.17.
- To update a workload, switch Namespace in the Workload to be upgraded section. Then, find the desired workload and click Rolling Upgrade in the Actions column. In the Confirm to perform rolling upgrade? dialog box, click Ok to update the workload to version 1.17.
  Note
  The list in the preceding page does not display ASM gateways or workloads that are already updated.
  You can click Instances Status in the left-side navigation pane to view workloads or gateways that are not updated in the instance.
  The following figure outlines the ASM instance architecture after the data plane is updated.

Step 8: Remove the old version

After all workloads on the data plane are updated, you can remove the old version 1.16.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
On the Canary Upgrade tab of the Upgrade Management page, click Offline old version. In the dialog box for confirming the removal of the control plane of the old version, confirm the operation.
The following figure outlines the ASM instance architecture at this point.