Use Kruise Rollouts to implement MSE-based end-to-end canary releases - Microservices Engine

As a Kubernetes application deployment tool, Kruise Rollouts provides a variety of canary release policies such as canary release and blue-green deployment. You can use Kruise Rollouts together with Microservices Engine (MSE) Microservices Governance to implement smooth canary releases for new versions of applications during the service call process. This ensures the stability of new versions.

Introduction to end-to-end canary releases

The traditional canary release mode cannot meet the complex delivery requirements in scenarios where the microservices architecture is used. The end-to-end canary release feature is introduced to route canary traffic to canary environments or groups for each microservice application. Developers expect that traffic flows from an upstream canary environment to a downstream canary environment. This way, a request remains within canary environments, and a traffic lane is generated. In a traffic lane, even if some microservice applications involved in the call process do not have their respective canary environments, requests for these microservice applications can still be routed to downstream canary environments. In this case, you can release changes to multiple services at the same time to ensure the system stability. The following figure shows the process of an end-to-end canary release.

Introduction to Kruise Rollouts

Kruise Rollouts is an open source progressive rollout framework provided by OpenKruise. For more information, see Kruise Rollouts. You can use Kruise Rollouts to perform canary releases, blue-green deployments, and A/B testing. You can also use Kruise Rollouts to control the canary traffic and pods. The release process can be automated in batches and paused based on the metrics of Managed Service for Prometheus. Kruise Rollouts also provides imperceptible connection of bypasses and is compatible with various workloads such as Deployments, CloneSets, and StatefulSets. For more information, see Kruise Rollouts.

Kruise Rollouts is a bypass component. You need to only create a Rollouts resource in your Container Service for Kubernetes (ACK) cluster to automate application releases and updates. Kruise Rollouts supports seamless integration with Helm and PaaS platforms at low costs. The following figure shows the architecture of a canary release by using Kruise Rollouts.

Prerequisites

An ACK cluster of V1.19 or later is created. For more information, see Create an ACK managed cluster.
kubectl-kruise is installed. For more information, see kubectl-kruise.
Microservices Governance is activated. For more information, see Activate Microservices Governance.

Step 1: Make preparations

Install the Kruise Rollouts component.
1. Log on to the ACK console. In the left-side navigation pane, click Clusters.
2. On the Clusters page, click the name of the cluster that you want to manage and choose Operations > Add-ons in the left-side navigation pane.
3. On the Manage Components page, click the Manage Applications tab.
4. Find the ack-kruise card and click Install.
5. In the message that appears, click OK.
Install the MSE Ingress Controller component. Create an MseIngressConfig and an IngressClass resource. For more information, see Use MSE Ingresses to access applications in ACK clusters.
Enable Microservices Governance for applications. For more information, see Enable Microservices Governance for microservice applications in an ACK cluster.

Step 2: Deploy demo applications

Create a deployment, a service, and an Ingress resource.

Create a file named mse-demo.yaml and copy the following content to the file.

Show code

# a service
apiVersion: v1
kind: Service
metadata:
  name: spring-cloud-a
spec:
  ports:
    - name: http
      port: 20001
      protocol: TCP
      targetPort: 20001
  selector:
    app: spring-cloud-a
# a application
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-a
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-a
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-a
      labels:
        app: spring-cloud-a
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-hangzhou.aliyuncs.com/mse-demo-hz/spring-cloud-a:mse-2.0.0
          imagePullPolicy: Always
          name: spring-cloud-a
          ports:
            - containerPort: 20001
# b application
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-b
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-b
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-b
      labels:
        app: spring-cloud-b
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-hangzhou.aliyuncs.com/mse-demo-hz/spring-cloud-b:mse-2.0.0
          imagePullPolicy: Always
          name: spring-cloud-b
# c application
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-c
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-c
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-c
      labels:
        app: spring-cloud-c
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-hangzhou.aliyuncs.com/mse-demo-hz/spring-cloud-c:mse-2.0.0
          imagePullPolicy: Always
          name: spring-cloud-c
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nacos-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nacos-server
  template:
    metadata:
      labels:
        app: nacos-server
    spec:
      containers:
      - env:
        - name: MODE
          value: standalone
        image: nacos/nacos-server:v2.2.0
        imagePullPolicy: Always
        name: nacos-server
      dnsPolicy: ClusterFirst
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: nacos-server
spec:
  type: ClusterIP
  ports:
    - name: nacos-server-8848-8848
      port: 8848
      protocol: TCP
      targetPort: 8848
    - name: nacos-server-9848-9848
      port: 9848
      protocol: TCP
      targetPort: 9848
  selector:
    app: nacos-server
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: demo-mysql
  name: demo-mysql
spec:
  ports:
    - port: 3306
      targetPort: 3306
  selector:
    app: demo-mysql
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-mysql
spec:
  selector:
    matchLabels:
      app: demo-mysql
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: demo-mysql
    spec:
      containers:
        - args:
            - --character-set-server=utf8mb4
            - --collation-server=utf8mb4_unicode_ci
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: root
          image: registry.cn-hangzhou.aliyuncs.com/mse-demo-hz/demo-mysql:3.0.1
          name: demo-mysql
          ports:
            - containerPort: 3306

Run the following command to deploy the business application:
```
kubectl apply -f mse-demo.yaml
```

Create a file named mse-Ingress.yaml and copy the following content to the file.

Show code

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/service-subset: ""
  name: spring-cloud-a
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: spring-cloud-a
                port:
                  number: 20001
            path: /
            pathType: Prefix

Run the following command to create an Ingress rule:
```
kubectl apply -f mse-ingress.yaml
```

Run the following command to query the public IP address:

kubectl get ingress

Expected output:

NAME         CLASS    HOSTS   ADDRESS        PORTS   AGE
spring-cloud-a   <none>   *       EXTERNAL_IP     80     12m

Run the following command to check the route status. Replace <EXTERNAL_IP> with the IP address that you obtained in the preceding step.
```
curl http://<EXTERNAL_IP>/A/a
```
Expected output:
```
A[192.168.42.115][config=base] -> B[192.168.42.118] -> C[192.168.42.101]% 
```

Step 3: Implement an automated end-to-end canary release by using Kruise Rollouts

Define the canary release rules of Kruise Rollouts.

Note

In this example, the canary release is performed in three batches:

A/B testing is performed. Requests that contain header[User-Agent]=xiaoming are forwarded to the canary version. Other requests are forwarded to the base version.
Half of the pods run the canary version, and half of requests are forwarded to the canary version.
All the pods run the canary version, and all the requests are forwarded to the canary version.

Create a file named rollout.yaml and copy the following content to the file:

Show code

# Specify the rollout configurations of applications for end-to-end canary releases.
# a rollout configuration
apiVersion: rollouts.kruise.io/v1alpha1
kind: Rollout
metadata:
  # Similar configurations of other microservice applications, such as rollout-b and rollout-c.
  name: rollout-a
  annotations:
    rollouts.kruise.io/trafficrouting: mse-traffic
spec:
  objectRef:
    workloadRef:
      apiVersion: apps/v1
      kind: Deployment
      name: spring-cloud-a
  strategy:
    canary:
      steps:
      - pause: {}
        replicas: 1
      patchPodTemplateMetadata:
        labels:
          alicloud.service.tag: gray
          opensergo.io/canary-gray: gray
---
# b rollout configuration
apiVersion: rollouts.kruise.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-b
  annotations:
    rollouts.kruise.io/trafficrouting: mse-traffic
spec:
  objectRef:
    workloadRef:
      apiVersion: apps/v1
      kind: Deployment
      name: spring-cloud-b
  strategy:
    canary:
      steps:
        - pause: {}
          replicas: 1
      patchPodTemplateMetadata:
        labels:
          alicloud.service.tag: gray
          opensergo.io/canary-gray: gray
---
# c rollout configuration
apiVersion: rollouts.kruise.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-c
  annotations:
    rollouts.kruise.io/trafficrouting: mse-traffic
spec:
  objectRef:
    workloadRef:
      apiVersion: apps/v1
      kind: Deployment
      name: spring-cloud-c
  strategy:
    canary:
      steps:
        - pause: {}
          replicas: 1
      patchPodTemplateMetadata:
        labels:
          alicloud.service.tag: gray
          opensergo.io/canary-gray: gray
---
# Configure end-to-end canary releases based on lanes.
apiVersion: rollouts.kruise.io/v1alpha1
kind: TrafficRouting
metadata:
  name: mse-traffic
spec:
  objectRef:
  - service: spring-cloud-a
    ingress:
      classType: mse
      name: spring-cloud-a
  strategy:
    matches:
    # Perform A/B testing. Traffic is routed based on request headers. 
    - headers:
      - type: Exact
        name: User-Agent
        value: xiaoming
        # Half of requests are forwarded to the canary version. 
    # The weight value is 30. 30% of requests are forwarded to the canary version.
    requestHeaderModifier:
      set:
      - name: x-mse-tag
        value: gray

Run the following command to deploy the Rollouts resource in your ACK cluster:
```
kubectl apply -f rollout.yaml
```
Run the following command to query the status of the Rollouts resource:
```
kubectl get rollout
```
If STATUS=Healthy is returned as expected, the Rollouts resource is working properly.

Update the application.

Kruise Rollouts is a common configuration and only needs to be distributed to your cluster. To release new application versions, you need to only update the deployment. You do not need to configure Kruise Rollouts again. For example, you can directly update the image versions of spring-cloud-a and spring-cloud-c to mse-2.0.1 and then run the kubectl apply -f mse-demo.yaml command to deploy the deployments to your cluster. In addition to kubectl, you can use Helm or Vela to deploy the deployments to your cluster.

Modify the mse-demo.yaml file to update the image versions of spring-cloud-a and spring-cloud-c to mse-2.0.1.

Show code

# a application
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-a
  ...
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-hangzhou.aliyuncs.com/mse-demo-hz/spring-cloud-a:mse-2.0.0
          imagePullPolicy: Always
          name: spring-cloud-a
          ports:
            - containerPort: 20001
...
# c application
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-c
spec:
  ...
          image: registry.cn-hangzhou.aliyuncs.com/mse-demo-hz/spring-cloud-c:mse-2.0.1

Run the following commands to query the status of the Rollouts resources:

kubectl get rollouts rollouts-a -n default
kubectl get rollouts rollouts-c -n default

Expected output:

NAME            STATUS        CANARY_STEP   CANARY_STATE   MESSAGE                                                                         AGE
rollouts-a   Progressing   1             StepPaused     Rollout is in step(1/1), and you need manually confirm to enter the next step   41m
rollouts-c   Progressing   1             StepPaused     Rollout is in step(1/1), and you need manually confirm to enter the next step   41m

You can check the STATUS, and CANARY_STATE columns to view the status of the Rollouts resources and the release stages.

If STATUS=Progressing is returned, the canary release stage is in progress.
If CANARY_STATE=StepPaused is returned, the current batch is released. Manual configuration is required for releasing the next batch.

You can release the remaining batches after you verify that the canary version runs as expected.
The preceding steps release only some applications of the canary version and forward a proportion of requests to the canary version. After you verify that the canary version runs as expected based on application logs and metric data, you can run the rollout.rollouts.kruise.io/<rollouts-demo> approved command to release the remaining applications of the canary version. <rollouts-demo> indicates the name of the Rollouts resource.
Optional. If the new application version does not run as expected, roll back the application version.
If the canary version does not run as expected during the release process, modify the deployment configurations and then run the kubectl apply -f mse-demo.yaml command to roll back the application version. You do not need to perform modification operations on the Rollouts resources.

The end-to-end canary release feature of Microservices Governance provides lanes, which greatly facilitate verification during testing and release. You can use Microservices Governance together with Kruise Rollouts to significantly improve the stability of online applications in a DevOps process.