Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Practice of End-to-end Canary Release by Using Kruise Rollout

Practice of End-to-end Canary Release by Using Kruise Rollout

This article discusses the concept and practice of end-to-end canary releases, particularly in the context of microservices.

By Danjiang and Shimian

What is End-to-end Canary Release?

During application deployment, it is common practice to verify the normal functioning of a new version by using a small amount of specific traffic, ensuring overall stability. This process is known as canary release. Canary release gradually increases the release scope, thus ensuring the stability of the new version. If any issues arise, they can be detected promptly, and the scope of impact can be controlled to maintain overall stability.

1

A canary release generally has the following features:

• Gradual increase in release scope, avoiding simultaneous full release.

• Phased release process that allows careful verification of the new version's stability using canary release.

• Support for suspension, rollback, resumption, and automated status flow to provide flexible control over the release process and ensure stability.

According to relevant data, 70% of online issues are caused by changes. Canary release, observability, and rollback are often referred to as the three key elements of safe production because they help control risks and mitigate the impact of changes. By implementing canary release, organizations can release new versions more robustly and prevent losses caused by issues during the release process.

The end-to-end canary release is a best practice for canyon release solutions in microservice scenarios. Usually, each microservice will have a canary release environment or grouping to receive canary traffic. We hope that the traffic entering the upstream canary release environment can also enter the downstream canary release environment, ensuring that one request is always transmitted in the canary release environment, thus forming a traffic lane. Even if some microservices in the traffic lane do not have a canary release environment, these applications can still return to the canary release environment when requesting downstream.

2
End-to-end canary release escorts microservice release

The end-to-end canary release enables independent release and traffic control for individual services based on their specific needs. It also allows simultaneous release and modification of multiple services to ensure the stability of the entire system. Additionally, the use of automated deployment methods facilitates faster and more reliable releases, ultimately improving release efficiency and stability.

Technological Analysis of the End-to-end Canary Release by Using Istio

The implementation of the end-to-end canary release by using Istio is a common concern. This article discusses in detail several key technologies.

End-to-end Pass-through Using Traffic Labels

During the implementation of end-to-end canary release for microservices, an important consideration is the pass-through of headers in traffic. The challenge arises when some microservices retain specific headers for pass-through and discard others. Although Kruise Rollout effectively reduces the complexity of gateway resource configuration during deployment, it does not address the header pass-through issue. One crucial aspect is ensuring that the canary identifier can be continuously transmitted within the trace. Distributed tracing technology records detailed request call traces in large-scale distributed systems. The core concept involves recording the nodes and request duration through a globally unique trace ID and the span ID of each request trace. Leveraging the distributed tracing concept allows for the transmission of custom information, such as canary identifiers. Commonly used distributed tracing analysis products in the industry support the transmission of custom data through the chain. The figure below illustrates the data processing process.

3

We typically utilize the Tracing Baggage mechanism to convey the corresponding identifiers throughout the full chain. Most Tracing frameworks, including OpenTelemetry, Skywalking, and Jaeger, support the Baggage concept and capability. We simply extract the value of the specified key, such as 'x-mse-tag', from the Baggage at the designated location according to the Tracing protocol in the Envoy outbound Filter and insert it into the HTTP header for Envoy to route.

Traffic Routing

By labeling all nodes under a service with various label names and values, service consumers subscribing to the service node information can access specific groups of the service as needed, essentially a subset of all nodes. Service consumers can employ any label information present on the service provider's node. Depending on the label's meaning, consumers can apply tag routing to broader business scenarios.

4

In Istio, you can use Istio Gateway, DestinationRule, and VirtualService to configure routing and external access.

Add Labels to Nodes

How do we attach different labels to service nodes? This is achieved simply by labeling the node in the pod template within the Deployment of the business application description resource. In business systems that use Kubernetes as service discovery, a service provider exposes the service by submitting a service resource to the ApiServer. The service consumer monitors the endpoint resource associated with the service resource, retrieves the related service pod resource from the endpoint resource, reads the Labels data, and uses it as metadata information for the node.

5

Why Choose Kruise Rollout?

The technical details discussed above reveal that implementing end-to-end canary operations based on Istio is extremely complex and costly. Initially, creating a canary deployment and adding canary identifiers to the nodes is necessary. Additionally, configuring the traffic routing CRD of Istio, including the VirtualService and DestinationRule rules for each request to match the request identifiers, is crucial. While these technologies may sound acceptable in theory, their practical application is associated with high costs. Furthermore, errors during the configuration process could lead to issues with production traffic, significantly impacting business operations. To mitigate the cost of end-to-end canary practice, the introduction of Kruise Rollout is essential.

Kruise Rollout[1] is an open-source progressive delivery framework offered by OpenKruise. It is designed to provide a standardized set of bypass Kubernetes release components that combine traffic release with instance canary release, support various release methods (such as canary, blue-green, and A/B testing), and facilitate automatic release processes that are unaware and easy to scale based on custom metrics (such as Prometheus Metrics).

6

As shown in the preceding figure, OpenKruise Rollout can automate the complex canary release, greatly reducing the implementation cost of canary release. Customers only need to configure the CRD of Kruise Rollout and then directly publish the application to achieve canary release.

Implementing End-to-end Canary Release by Using Kruise and Istio

We have discussed the details of implementation technologies. Now, let's practice the canary release based on Kruise Rollout and Istio.

Configure Services

Begin by deploying two services: mocka and mockb. The service mocka will call the service mockb, ⚠️and the service will only retain the header my-request-id in traffic, discarding other headers. You can also employ OpenTelemetry to facilitate dynamic traffic label pass-through.

7

The following is the configuration file for the service:

apiVersion: v1
kind: Service
metadata:
  name: mocka
  namespace: e2e
  labels:
    app: mocka
    service: mocka
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mocka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-base
  namespace: e2e
  labels:
    app: mocka
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
  template:
    metadata:
      labels:
        app: mocka
        version: base
    spec:
      containers:
      - name: default
        image: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/go-http-sample:1.0
        imagePullPolicy: Always
        env:
        - name: version
          value: base
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: mockb
  namespace: e2e
  labels:
    app: mockb
    service: mockb
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mockb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-base
  namespace: e2e
  labels:
    app: mockb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
  template:
    metadata:
      labels:
        app: mockb
        version: base
    spec:
      containers:
      - name: default
        image: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/go-http-sample:1.0
        imagePullPolicy: Always
        env:
        - name: version
          value: base
        - name: app
          value: mockb
        ports:
        - containerPort: 8000

The following is the code for processing headers:

// All URLs will be handled by this function
m.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {

    requestId := r.Header.Get("my-request-id")

    fmt.Printf("receive request: my-request-id: %s\n", requestId)

    response := fmt.Sprintf("-> %s(version: %s, ip: %s)", app, version, ip)

    if url != "" {
        // Send only my-request-id for new requests.
        content := doReq(url, requestId)
        response = response + content

    }
    w.Write([]byte(response))
})

Then, deploy Istio Gateway, DestinationRule, and VirtualService to configure the routing and external access:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gateway
  namespace: e2e
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - "*"
      port:
        name: http
        number: 80
        protocol: HTTP
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: dr-mocka
  namespace: e2e
spec:
  host: mocka
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
    - labels:
        version: base
      name: version-base
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: dr-mockb
  namespace: e2e
spec:
  host: mockb
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
    - labels:
        version: base
      name: version-base
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-mocka
  namespace: e2e
spec:
  gateways:
  - simple-gateway
  hosts:
    - "*"
  http:
  - route:
      - destination:
          host: mocka
          subset: version-base
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-mockb
  namespace: e2e
spec:
  hosts:
    - mockb
  http:
    - route:
        - destination:
            host: mockb
            subset: version-base

The following figure shows the entire structure after deployment:

8

In local clusters, you can run the following command to obtain the gateway ingress IP and port to access the service:

kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}'
kubectl get po -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].status.hostIP}'

Run curl http://GATEWAY_IP:PORT. The following output is returned:

-> mocka(version: base, ip: 10.244.1.36)-> mockb(version: base, ip: 10.244.1.37)

Configuration of Rollout and TrafficRouting

Then, deploy Rollout to control mocka and mockb. The following are the policies for configuring Rollout and TrafficRouting:

• A header rule that matches my-request-id=canary is added. Traffic that contains the specified header is routed to the canary environment.
• label istio.service.tag=gray and version=canary are added for the released pod.

⚠ What are the Rules for Adding Pod Labels?

Two labels are added to the pod of the new version in the configuration. The purpose of istio.service.tag=gray is to specify the pod containing the label as a subset in the DestinationRule. The lua script automatically adds the subset to the DestinationRule. version=canary is added to overwrite version=baselabel in the original version. If this label is not overwritten, the original DestinationRule will also import the traffic of the stable version to the pod of the new version.

apiVersion: rollouts.kruise.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-a
  namespace: demo
  annotations:
    rollouts.kruise.io/rolling-style: canary
    rollouts.kruise.io/trafficrouting: mocka-tr
spec:
  disabled: false
  objectRef:
    workloadRef:
      apiVersion: apps/v1
      kind: Deployment
      name: mocka-base
  strategy:
    canary:
      steps:
      - replicas: 1
      patchPodTemplateMetadata:
        labels:
          istio.service.tag: gray
          version: canary
---
apiVersion: rollouts.kruise.io/v1alpha1
kind: TrafficRouting
metadata:
  name: mocka-tr
  namespace: demo
spec:
  strategy:
    matches:
      - headers:
        - type: Exact
          name: my-request-id
          value: canary
  objectRef:
  - service: mocka
    customNetworkRefs:
    - apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      name: vs-mocka
    - apiVersion: networking.istio.io/v1beta1
      kind: DestinationRule
      name: dr-mocka
---
apiVersion: rollouts.kruise.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-b
  namespace: demo
  annotations:
    rollouts.kruise.io/rolling-style: canary
    rollouts.kruise.io/trafficrouting: mockb-tr
spec:
  disabled: false
  objectRef:
    workloadRef:
      apiVersion: apps/v1
      kind: Deployment
      name: mockb-base
  strategy:
    canary:
      steps:
      - replicas: 1
      patchPodTemplateMetadata:
        labels:
          istio.service.tag: gray
          version: canary
---
apiVersion: rollouts.kruise.io/v1alpha1
kind: TrafficRouting
metadata:
  name: mockb-tr
  namespace: demo
spec:
  strategy:
    matches:
      - headers:
        - type: Exact
          name: my-request-id
          value: canary
  objectRef:
  - service: mockb
    customNetworkRefs:
    - apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      name: vs-mockb
    - apiVersion: networking.istio.io/v1beta1
      kind: DestinationRule
      name: dr-mockb

Start a Canary Release

Modify the environment variables in mocka and mockb to version=canary to start the release. Kruise Rollout automatically obtains gateway resources and modifies them. You can find in VirtualService and DestinationRule that VirtualService defines routing rules to route traffic with the header of my-request-id=canary to the canary version, and DestinationRule adds a new subset that contains label istio.service.tag=gray.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1beta1","kind":"VirtualService","metadata":{"annotations":{},"name":"vs-mocka","namespace":"demo"},"spec":{"gateways":["simple-gateway"],"hosts":["*"],"http":[{"route":[{"destination":{"host":"mocka","subset":"version-base"}}]}]}}
    rollouts.kruise.io/origin-spec-configuration: '{"spec":{"gateways":["simple-gateway"],"hosts":["*"],"http":[{"route":[{"destination":{"host":"mocka","subset":"version-base"}}]}]},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.istio.io/v1beta1\",\"kind\":\"VirtualService\",\"metadata\":{\"annotations\":{},\"name\":\"vs-mocka\",\"namespace\":\"demo\"},\"spec\":{\"gateways\":[\"simple-gateway\"],\"hosts\":[\"*\"],\"http\":[{\"route\":[{\"destination\":{\"host\":\"mocka\",\"subset\":\"version-base\"}}]}]}}\n"}}'
  creationTimestamp: "2023-09-12T07:49:15Z"
  generation: 40
  name: vs-mocka
  namespace: demo
  resourceVersion: "98670"
  uid: c7da3a99-789c-4f1e-93a4-caaee41cbe06
spec:
  gateways:
  - simple-gateway
  hosts:
  - '*'
  http:
  # -- Rules automatically added by the lua script.
  - match:
    - headers:
        my-request-id:
          exact: canary
    route:
    - destination:
        host: mocka
        subset: canary
  # --
  - route:
    - destination:
        host: mocka
        subset: version-base
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1beta1","kind":"VirtualService","metadata":{"annotations":{},"name":"vs-mockb","namespace":"demo"},"spec":{"hosts":["mockb"],"http":[{"route":[{"destination":{"host":"mockb","subset":"version-base"}}]}]}}
    rollouts.kruise.io/origin-spec-configuration: '{"spec":{"hosts":["mockb"],"http":[{"route":[{"destination":{"host":"mockb","subset":"version-base"}}]}]},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.istio.io/v1beta1\",\"kind\":\"VirtualService\",\"metadata\":{\"annotations\":{},\"name\":\"vs-mockb\",\"namespace\":\"demo\"},\"spec\":{\"hosts\":[\"mockb\"],\"http\":[{\"route\":[{\"destination\":{\"host\":\"mockb\",\"subset\":\"version-base\"}}]}]}}\n"}}'
  creationTimestamp: "2023-09-12T07:49:16Z"
  generation: 40
  name: vs-mockb
  namespace: demo
  resourceVersion: "98677"
  uid: 7c96ee2b-96ce-48e4-ba6d-cf94171ed854
spec:
  hosts:
  - mockb
  http:
  # -- Rules automatically added by the lua script.
  - match:
    - headers:
        my-request-id:
          exact: canary
    route:
    - destination:
        host: mockb
        subset: canary
  # -- 
  - route:
    - destination:
        host: mockb
        subset: version-base
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1beta1","kind":"DestinationRule","metadata":{"annotations":{},"name":"dr-mocka","namespace":"demo"},"spec":{"host":"mocka","subsets":[{"labels":{"version":"base"},"name":"version-base"}],"trafficPolicy":{"loadBalancer":{"simple":"ROUND_ROBIN"}}}}
    rollouts.kruise.io/origin-spec-configuration: '{"spec":{"host":"mocka","subsets":[{"labels":{"version":"base"},"name":"version-base"}],"trafficPolicy":{"loadBalancer":{"simple":"ROUND_ROBIN"}}},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.istio.io/v1beta1\",\"kind\":\"DestinationRule\",\"metadata\":{\"annotations\":{},\"name\":\"dr-mocka\",\"namespace\":\"demo\"},\"spec\":{\"host\":\"mocka\",\"subsets\":[{\"labels\":{\"version\":\"base\"},\"name\":\"version-base\"}],\"trafficPolicy\":{\"loadBalancer\":{\"simple\":\"ROUND_ROBIN\"}}}}\n"}}'
  creationTimestamp: "2023-09-12T07:49:15Z"
  generation: 12
  name: dr-mocka
  namespace: demo
  resourceVersion: "98672"
  uid: a6f49044-e889-473c-b188-edbdb8ee347f
spec:
  host: mocka
  subsets:
  - labels:
      version: base
    name: version-base
    # -- Rules automatically added by the lua script.
  - labels:
      istio.service.tag: gray
    name: canary
    # --
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1beta1","kind":"DestinationRule","metadata":{"annotations":{},"name":"dr-mockb","namespace":"demo"},"spec":{"host":"mockb","subsets":[{"labels":{"version":"base"},"name":"version-base"}],"trafficPolicy":{"loadBalancer":{"simple":"ROUND_ROBIN"}}}}
    rollouts.kruise.io/origin-spec-configuration: '{"spec":{"host":"mockb","subsets":[{"labels":{"version":"base"},"name":"version-base"}],"trafficPolicy":{"loadBalancer":{"simple":"ROUND_ROBIN"}}},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.istio.io/v1beta1\",\"kind\":\"DestinationRule\",\"metadata\":{\"annotations\":{},\"name\":\"dr-mockb\",\"namespace\":\"demo\"},\"spec\":{\"host\":\"mockb\",\"subsets\":[{\"labels\":{\"version\":\"base\"},\"name\":\"version-base\"}],\"trafficPolicy\":{\"loadBalancer\":{\"simple\":\"ROUND_ROBIN\"}}}}\n"}}'
  creationTimestamp: "2023-09-12T07:49:15Z"
  generation: 12
  name: dr-mockb
  namespace: demo
  resourceVersion: "98678"
  uid: 4bd0f6c5-efa1-4558-9e31-6f7615c21f9a
spec:
  host: mockb
  subsets:
  - labels:
      version: base
    name: version-base
    # -- Rules automatically added by the lua script.
  - labels:
      istio.service.tag: gray
    name: canary
    # --
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN

Run curl http://GATEWAY_IP:PORT. The following output is returned. All traffic passes through the base version service.

-> mocka(version: base, ip: 10.244.1.36)-> mockb(version: base, ip: 10.244.1.37)

Run curl -H http://GATEWAY_IP:PORT -Hmy-request-id:canary. The following output is returned. All traffic passes through the canary version service.

-> mocka(version: canary, ip: 10.244.1.41)-> mockb(version: canary, ip: 10.244.1.42)

The following figure shows the traffic of the entire service. Traffic that contains the headermy-request-id=canary is routed to the canary environment, and other traffic to the base environment:

9

(Optional) Use EnvoyFilter for Traffic Dyeing

We can further simplify the traffic dyeing by writing EnvoyFilter. In the preceding example, the request contains headermy-request-id that can be passed through. If you want to implement more general header request rules, for example, all traffic that contains agent=pc is routed to the grayscale environment, and other traffic to the baseline environment, you can use EnvoyFilter to dye the traffic at the ingress gateway. The following is an EnvoyFilter that defines a lua script to dye traffic with agent=pc and add my-request-id=canary to it while adding my-request-id=base to other traffic.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: http-request-labelling-according-source
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: istio-ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
            subFilter:
              name: "envoy.filters.http.router"
    patch:
      operation: INSERT_BEFORE
      value:
       name: envoy.lua
       typed_config:
         "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
         inlineCode: |
           function envoy_on_request(request_handle)
           local header = "agent"
            headers = request_handle:headers()
             version = headers:get(header)
             if (version ~= nil) then
              if (version == "pc") then
                headers:add("my-request-id","canary")
              else
                headers:add("my-request-id","base")
              end
             else
              headers:add("my-request-id","base")
             end
           end

With other configurations unchanged, run curl http://GATEWAY_IP:PORT. The following output is returned. All traffic passes through the base version service.

-> mocka(version: base, ip: 10.244.1.36)-> mockb(version: base, ip: 10.244.1.37)

Run curl -H http://GATEWAY_IP:PORT -Hagent:pc. The following output is returned. EnvoyFilter automatically adds headermy-request-id=canary for the traffic, thus all traffic passes through the new version of the service.

-> mocka(version: canary, ip: 10.244.1.41)-> mockb(version: canary, ip: 10.244.1.42)

Complete End-to-end Canary Release Solution

In addition to the open source Istio, Kruise Rollout also supports a complete end-to-end canary release solution with MSE. You can perform the following operations[2] to quickly implement a systematic end-to-end canary release.

10

The end-to-end canary release of MSE can effectively control the traffic flow across front-end, gateway, back-end microservices, and other components. Apart from RPC/HTTP traffic, asynchronous calls like MQ traffic also comply with end-to-end "traffic lane" call rules. You can use Kruise Rollout and MSE to easily implement end-to-end canary release of microservices and improve the efficiency and stability of microservice release.

References

[1] Kruise Rollout
https://openkruise.io/rollouts/introduction
[2] Operation document
https://www.alibabacloud.com/help/en/mse/user-guide/implement-mse-based-end-to-end-canary-release-by-using-kruise-rollouts

Best practices Kubernetes Istio cloud native OpenKruise Canary Release Traffic Routing Kruise Rollout
0 1 0
Share on

Read previous post:

Upgrade Spring Boot Applications to Spring Cloud

Read next post:

Cloud-native Offline Workflow Orchestration: Kubernetes Clusters for Distributed Argo Workflows

Alibaba Cloud Native Community

508 posts | 48 followers

You may also like

Comments

Alibaba Cloud Native Community

508 posts | 48 followers

Related Products

More Posts by Alibaba Cloud Native Community

See All