Alibaba Cloud Service Mesh:Use traffic labels to implement an end-to-end canary release

Feature description

Canary releases can be implemented in various ways. For example, you can use ASM to deploy an application in blue-green release mode and phased release mode. The sample mode is applicable to the release of a single service. This mode uses VirtualService of open source Istio to specify the traffic weight of each version of the service. This mode cannot meet requirements in canary release scenarios of multiple services.

ASM provides the end-to-end canary release feature based on the traffic labeling and label-based routing features. This allows you to implement the canary release of multiple services at the same time. During canary testing of business, the system divides ingress traffic into regular traffic and canary traffic and identifies the characteristics of traffic. If the traffic of a request is identified as canary traffic, the request is routed to the canary release service. This way, the system no longer simply distributes traffic to backend services of different versions at specific traffic ratios. Instead, the canary traffic characteristics are passed among all the services that are used to process user requests. For more information, see Label traffic.

Configuration description

You can click here to download the sample configuration files. The following figure shows the call chains of the services in the example:

In this example, when Service A calls Service B, Service A contains the code implementation logic to pass the HTTP header my-trace-id. For more information, see the main.go configuration file in the src/mock-abc/go/main.go path.

apiVersion: v1
kind: Service
metadata:
  name: mocka
  labels:
    app: mocka
    service: mocka
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mocka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-base
  labels:
    app: mocka
    version: base
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: base
  template:
    metadata:
      labels:
        app: mocka
        version: base
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
        imagePullPolicy: Always
        env:
        - name: version
          value: base
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: mockb
  labels:
    app: mockb
    service: mockb
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mockb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-base
  labels:
    app: mockb
    version: base
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: base
  template:
    metadata:
      labels:
        app: mockb
        version: base
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
        imagePullPolicy: Always
        env:
        - name: version
          value: base
        - name: app
          value: mockb
        ports:
        - containerPort: 8000
                

apiVersion: v1
kind: Service
metadata:
  name: mocka
  labels:
    app: mocka
    service: mocka
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mocka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-canary
  labels:
    app: mocka
    version: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: canary
  template:
    metadata:
      labels:
        app: mocka
        version: canary
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
        imagePullPolicy: Always
        env:
        - name: version
          value: canary
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: mockb
  labels:
    app: mockb
    service: mockb
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mockb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-canary
  labels:
    app: mockb
    version: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: canary
  template:
    metadata:
      labels:
        app: mockb
        version: canary
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
        imagePullPolicy: Always
        env:
        - name: version
          value: canary
        - name: app
          value: mockb
        ports:
        - containerPort: 8000
                

Step 1: Deploy sample services in the ACK cluster

1. Enable automatic sidecar proxy injection for the desired namespace (the default namespace in this example). For more information, see Enable automatic sidecar proxy injection.

2. Use kubectl to connect to the ACK cluster based on the information in the kubeconfig file and run the following commands to deploy sample services.

   For more information about the YAML files, see Configuration description.

   kubectl apply -n default -f application-base.yaml
   kubectl apply -n default -f application-canary.yaml

Step 2: Configure initial routing rules

1. Create a file named routing.yaml and copy the following content to the fil:.

   Content of routing.yaml

   apiVersion: networking.istio.io/v1beta1
   kind: Gateway
   metadata:
     name: simple-gateway
   spec:
     selector:
       istio: ingressgateway
     servers:
       - hosts:
           - "*"
         port:
           name: http
           number: 80
           protocol: HTTP
   ---
   apiVersion: networking.istio.io/v1beta1
   kind: DestinationRule
   metadata:
     name: dr-mocka
   spec:
     host: mocka
     trafficPolicy:
       loadBalancer:
         simple: ROUND_ROBIN
     subsets:
       - labels:
           version: base
         name: version-base
       - labels:
           version: canary
         name: version-canary
   ---
   apiVersion: networking.istio.io/v1beta1
   kind: DestinationRule
   metadata:
     name: dr-mockb
   spec:
     host: mockb
     trafficPolicy:
       loadBalancer:
         simple: ROUND_ROBIN
     subsets:
       - labels:
           version: base
         name: version-base
       - labels:
           version: canary
         name: version-canary
   ---
   apiVersion: networking.istio.io/v1beta1
   kind: VirtualService
   metadata:
     name: vs-gateway-mocka
   spec:
     gateways:
     - simple-gateway
     hosts:
       - "*"
     http:
     - route:
         - destination:
             host: mocka
   ---
   apiVersion: networking.istio.io/v1beta1
   kind: VirtualService
   metadata:
     name: vs-mockb
   spec:
     hosts:
       - mockb
     http:
       - route:
           - destination:
               host: mockb

2. Use kubectl to connect to the ASM instance based on the information in the kubeconfig file and run the following command to configure routing rules:

   kubectl apply -n default -f routing.yaml

3. Check whether the services of different versions can be accessed.

   1. Obtain the public IP address of the ingress gateway in the ASM console. For more information, see Step 2: Obtain the IP address of the ASM ingress gateway.

   2. Run the following command to configure an environment variable.

      xxxx is the IP address obtained in Substep a.

      export ASM_GATEWAY_IP=xxxx

   3. Run the following command to check whether the services of different versions can be accessed:

      for i in {1..200};  do curl   -H'my-asm-prefer-tag: version-base'  -H'my-trace-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

      Expected output:

      -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
      -> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
      -> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)
      -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
      -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
      -> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
      -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)

      The output shows that the routing policy of random load balancing is adopted for traffic flows from the ingress gateway to Service A and from Service A to Service B. The my-asm-prefer-tag header or other request headers specified by using the curl command do not change the random routing policy.

Step 3: Configure traffic labels for the sample services

ASM allows you to configure traffic labels by using a TrafficLabel CustomResourceDefinition (CRD) and route traffic to different workloads based on the traffic labels. For more information, see Label traffic.

1. Create a file named trafficlabel.yaml and copy the following content to the file:

   Content of trafficlabel.yaml

   apiVersion: istio.alibabacloud.com/v1
   kind: TrafficLabel
   metadata:
     name: trafficlabel-ns
     namespace: default
   spec:
     rules:
     - labels:
       - name: asm-labels-sample
         valueFrom:
         - $getExternalInboundRequestHeader(my-asm-prefer-tag, my-trace-id)
   ---
   apiVersion: istio.alibabacloud.com/v1
   kind: TrafficLabel
   metadata:
     name: ingressgateway
     namespace: istio-system
   spec:
     rules:
     - labels:
       - name: asm-labels-sample
         valueFrom:
         - $getInboundRequestHeader(my-asm-prefer-tag)
     workloadSelector:
       labels:
         istio: ingressgateway

   Configuration description:

   • The trafficlabel-ns traffic label takes effect on all services in the default namespace, such as the mocka and mockb services deployed in this example. The ingressgateway traffic label takes effect only on the ingressgateway gateway.

   • In this example, the my-asm-prefer-tag request header is used to distinguish versions. Therefore, set the first parameter in getExternalInboundRequestHeader to my-asm-prefer-tag. Modify the configuration based on your business requirements.

   • In this example, my-trace-id is the HTTP header that needs to be passed. Therefore, set the second parameter in getExternalInboundRequestHeader to my-trace-id. Modify the configuration based on your business requirements.

2. Use kubectl to connect to the ASM instance based on the information in the kubeconfig file and run the following command to configure the traffic labels:

   kubectl apply -f trafficlabel.yaml

Step 4: Configure label-based routing rules

You can configure label-based routing rules to control traffic flows.

1. Check the canary release on the service provider side

Check whether the traffic routing from Service A to Service B meets the expectation. To be specific, check whether the canary traffic that flows into Service A is forwarded to the canary release version of Service B, and whether the base traffic that flows into Service A is forwarded to the base version of Service B.

1. Create a file named vs-tf-mockb.yaml and copy the following content to the file:

   apiVersion: networking.istio.io/v1beta1
   kind: VirtualService
   metadata:
     name: vs-mockb
   spec:
     hosts:
       - mockb
     http:
       - route:
           - destination:
               host: mockb
               subset: $asm-labels-sample

2. Run the following command to make the routing rule take effect for Service A:

   kubectl apply -n default -f vs-tf-mockb.yaml

3. Run the following command to check whether the canary traffic that flows into Service A is forwarded to the canary release version of Service B:

   for i in {1..200};  do curl -H'my-asm-prefer-tag: version-canary'  -H'my-trace-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

   Expected output:

   -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
   -> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)
   -> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)
   -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
   -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
   -> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)

4. Run the following command to check whether the base traffic that flows into Service A is forwarded to the base version of Service B:

   for i in {1..200};  do curl -H'my-asm-prefer-tag: version-base'  -H'my-trace-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

   Expected output:

   -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
   -> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
   -> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
   -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
   -> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
   -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
   -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
   -> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)

   The results show that the ingress canary traffic and ingress base traffic that flow into Service A are not routed to the expected versions of Service B. You need to configure a traffic label-based routing rule for Service A. For more information, proceed to the next step.

2. Check the end-to-end canary release feature

Configure a traffic label-based routing rule file named a-vs-tf.yaml for Service A and make the file take effect for the ingress gateway. The expected result is that the canary traffic of ingress requests is first forwarded to the canary release version of Service A and then to that of Service B, and the base traffic is first forwarded to the base version of Service A and then to that of Service B.

1. Create a file named vs-tf-mocka.yaml and copy the following content to the file:

   apiVersion: networking.istio.io/v1beta1
   kind: VirtualService
   metadata:
     name: vs-gateway-mocka
   spec:
     gateways:
     - simple-gateway
     hosts:
       - "*"
     http:
     - route:
         - destination:
             host: mocka
             subset: $asm-labels-sample

2. Run the following command to make the routing rule take effect for the ingress gateway:

   kubectl apply -n default -f vs-tf-mocka.yaml

3. Run the following command to check whether the canary traffic of ingress requests is first forwarded to the canary release version of Service A and then to that of Service B:

   for i in {1..200};  do curl -H'my-asm-prefer-tag: version-canary'  -H'my-trace-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

   Expected output:

   -> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)

4. Run the following command to check whether the base traffic of ingress requests is first forwarded to the base version of Service A and then to that of Service B:

   for i in {1..200};  do curl -H'my-asm-prefer-tag: version-base'  -H'my-trace-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

   Expected output:

   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)

3. Check whether the weight-based traffic distribution that corresponds to the traffic label-based routing meets the requirements

1. Create a file named vs-tf-mocka-90-10.yaml and copy the following content to the file:

   apiVersion: networking.istio.io/v1beta1
   kind: VirtualService
   metadata:
     name: vs-gateway-mocka
   spec:
     gateways:
     - simple-gateway
     hosts:
       - "*"
     http:
     - route:
         - destination:
             host: mocka
             subset: version-base
           weight: 90
         - destination:
             host: mocka
             subset: $asm-labels-sample
           weight: 10

2. Run the following command to make the routing rules take effect for the ingress gateway:

   kubectl apply -n default -f vs-tf-mocka-90-10.yaml

3. Run the following command to check whether the canary traffic of ingress requests is forwarded to the canary release and base versions of Service A:

   for i in {1..200};  do curl -H'my-asm-prefer-tag: version-canary'  -H'my-trace-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

   Expected output:

   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
   -> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)

   The results show that about 90% of the canary traffic of ingress requests is forwarded to the base version of Service A and about 10% is forwarded to the canary release version of Service A. According to the above routing rules, about 90% of the request traffic is forwarded to the base version of Service A, and the remaining 10% is forwarded based on the value of the my-asm-prefer-tag header. The request command sets the value of my-asm-prefer-tag to version-canary. Therefore, 10% of the request traffic is forwarded to the canary release version of Service A.

4. Run the following command to check whether all base traffic of ingress requests is forwarded to the base version of Service A:

   for i in {1..200};  do curl -H'my-asm-prefer-tag: version-base'  -H'my-trace-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

   Expected output:

   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
   -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)

   According to the above routing rules, about 90% of the request traffic is forwarded to the base version of Service A, and the remaining 10% is forwarded based on the value of the my-asm-prefer-tag header. The request command sets the value of my-asm-prefer-tag to version-base. Therefore, 10% of the request traffic is forwarded to the base version of Service A.