Pass through baggage headers in traces - Alibaba Cloud Service Mesh

You can use traffic lanes in permissive mode to isolate versions of an application and route traffic to different lanes based on a specific ratio. Your application must add headers that can be transparently transmitted throughout the corresponding traces in requests and such headers have different values. These request headers are called the end-to-end (E2E) pass-through request headers. In this example, baggage headers are used as the E2E pass-through request headers. When services in a lane call each other, if the service to be called does not exist in the lane, requests are forwarded to the same service in the baseline lane. This feature ensures the integrity of traces and simplifies traffic management.

Important

Before you get started, make sure that you have read and understood the Use traffic lanes in permissive mode to manage end-to-end traffic topic and related content.

Scenario description

In this example, three services (mocka, mockb, and mockc) and three lanes (s1, s2, and s3) are used to simulate traces. s1 is the baseline lane that contains all three services of v1. s2 contains only mocka and mockc of v2. s3 contains only mockb of v3. First, use the auto-instrumentation capability of OpenTelemetry to enable the pods for services to pass through baggage headers. Then, create three lanes in permissive mode and configure traffic routing weights to route traffic at a specific ratio.

Step 1: Deploy sample services

Enable automatic sidecar proxy injection for the default namespace. For more information, see Manage global namespaces.
Note
For more information about automatic sidecar proxy injection, see Enable automatic sidecar proxy injection.

Create a mock.yaml file that contains the following content:

Show the mock.yaml file

apiVersion: v1
kind: Service
metadata:
  name: mocka
  labels:
    app: mocka
    service: mocka
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mocka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-v1
  labels:
    app: mocka
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: v1
      ASM_TRAFFIC_TAG: v1
  template:
    metadata:
      labels:
        app: mocka
        version: v1
        ASM_TRAFFIC_TAG: v1
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v1
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: mockb
  labels:
    app: mockb
    service: mockb
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mockb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-v1
  labels:
    app: mockb
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: v1
      ASM_TRAFFIC_TAG: v1
  template:
    metadata:
      labels:
        app: mockb
        version: v1
        ASM_TRAFFIC_TAG: v1
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v1
        - name: app
          value: mockb
        - name: upstream_url
          value: "http://mockc:8000/"
        ports:
        - containerPort: 8000

---
apiVersion: v1
kind: Service
metadata:
  name: mockc
  labels:
    app: mockc
    service: mockc
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mockc
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockc-v1
  labels:
    app: mockc
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockc
      version: v1
      ASM_TRAFFIC_TAG: v1
  template:
    metadata:
      labels:
        app: mockc
        version: v1
        ASM_TRAFFIC_TAG: v1
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v1
        - name: app
          value: mockc
        ports:
        - containerPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-v2
  labels:
    app: mocka
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: v2
      ASM_TRAFFIC_TAG: v2
  template:
    metadata:
      labels:
        app: mocka
        version: v2
        ASM_TRAFFIC_TAG: v2
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v2
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-v2
  labels:
    app: mockb
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: v2
      ASM_TRAFFIC_TAG: v2
  template:
    metadata:
      labels:
        app: mockb
        version: v2
        ASM_TRAFFIC_TAG: v2
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v2
        - name: app
          value: mockb
        - name: upstream_url
          value: "http://mockc:8000/"
        ports:
        - containerPort: 8000

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockc-v2
  labels:
    app: mockc
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockc
      version: v2
      ASM_TRAFFIC_TAG: v2
  template:
    metadata:
      labels:
        app: mockc
        version: v2
        ASM_TRAFFIC_TAG: v2
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v2
        - name: app
          value: mockc
        ports:
        - containerPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-v3
  labels:
    app: mocka
    version: v3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: v3
      ASM_TRAFFIC_TAG: v3
  template:
    metadata:
      labels:
        app: mocka
        version: v3
        ASM_TRAFFIC_TAG: v3
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v3
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-v3
  labels:
    app: mockb
    version: v3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: v3
      ASM_TRAFFIC_TAG: v3
  template:
    metadata:
      labels:
        app: mockb
        version: v3
        ASM_TRAFFIC_TAG: v3
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v3
        - name: app
          value: mockb
        - name: upstream_url
          value: "http://mockc:8000/"
        ports:
        - containerPort: 8000

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockc-v3
  labels:
    app: mockc
    version: v3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockc
      version: v3
      ASM_TRAFFIC_TAG: v3
  template:
    metadata:
      labels:
        app: mockc
        version: v3
        ASM_TRAFFIC_TAG: v3
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v3
        - name: app
          value: mockc
        ports:
        - containerPort: 8000

Annotations instrumentation.opentelemetry.io/inject-java: "true" and instrumentation.opentelemetry.io/container-names: "default" are added to each service pod to declare that the corresponding service is implemented in Java, and the OpenTelemetry Operator is required to auto-instrument the container named default.

Run the following command to deploy the services:
```
kubectl apply -f mock.yaml
```
Based on the auto-instrumentation mechanism of OpenTelemetry, pods for services can automatically pass through baggage headers in traces.

Step 2: Create a lane group and corresponding lanes

Create a lane group.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Traffic Management Center > Traffic Lane.

On the Traffic Lane page, click Create Swimlane Group. In the Create Swimlane Group panel, configure the required parameters and click OK.

Parameter	Description
Name of swim lane group	For this example, enter test.
Istio gateway for an ingress gateway	Select ingressgateway.
Lane Mode	Select Permissive Mode.
Pass-through Mode of Trace Context	Select Pass Through Baggage Header.
Routing Request Header	Enter x-asm-prefer-tag.
Swimlane Services	Select the cluster in which the mocka, mockb, and mockc services reside from the Kubernetes Clusters drop-down list and default from the Namespace drop-down list. Select the mocka, mockb, and mockc services in the list, and click the icon to add these services to the selected section.

Create lanes named s1, s2, and s3 and bind the s1 lane to the v1 version of the sample services, the s2 lane to the v2 version of the sample services, and the s3 lane to v3 version of the sample services.

In the Traffic Rule Definition section of the Traffic Lane page, click Create swimlanes.
In the Create swimlanes dialog box, configure the required parameters and click OK.

Parameter	Description
Swimlane Name	Name the three lanes as s1, s2, and s3 respectively.
Configure Service Tag	Label Key: Set it to ASM_TRAFFIC_TAG. Label Value: Set it to v1 for the s1 lane, v2 for the s2 lane, and v3 for the s3 lane.
Add Service	For the s1 lane, select mocka(default), mockb(default), and mockc(default). For the s2 lane, select mocka(default) and mockc(default). For the s3 lane, select mockb(default).

The following figure shows the configurations of the s1 lane.

After the three lanes are created, you can view them in the Traffic Rule Definition section, as shown in the following figure.

By default, the first lane you create in a lane group is set as the baseline lane. However, you can change the baseline lane. When traffic is destined for services that do not exist in the other lanes, requests are forwarded to the baseline lane according to the fallback mechanism. For more information about how to change the baseline lane, see Change the baseline lane in permissive mode.

In the left-side navigation pane of the ASM console, choose Traffic Management Center > DestinationRule or VirtualService to view the destination rule or virtual service that is automatically generated for each service in the lane group. For example, the following destination rule and virtual service are automatically created for the mocka service.

Show the destination rule and virtual service that are automatically created for the mocka service

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  labels:
    asm-system: 'true'
    provider: asm
    swimlane-group: test
  name: trafficlabel-dr-test-default-mocka
  namespace: istio-system
spec:
  host: mocka.default.svc.cluster.local
  subsets:
    - labels:
        ASM_TRAFFIC_TAG: v1
      name: s1
    - labels:
        ASM_TRAFFIC_TAG: v2
      name: s2

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  labels:
    asm-system: 'true'
    provider: asm
    swimlane-group: test
  name: trafficlabel-vs-test-default-mocka
  namespace: istio-system
spec:
  hosts:
    - mocka.default.svc.cluster.local
  http:
    - match:
        - headers:
            x-asm-prefer-tag:
              exact: s1
      route:
        - destination:
            host: mocka.default.svc.cluster.local
            subset: s1
          fallback:
            target:
              host: mocka.default.svc.cluster.local
              subset: s1
    - match:
        - headers:
            x-asm-prefer-tag:
              exact: s2
      route:
        - destination:
            host: mocka.default.svc.cluster.local
            subset: s2
          fallback:
            target:
              host: mocka.default.svc.cluster.local
              subset: s1
    - match:
        - headers:
            x-asm-prefer-tag:
              exact: s3
      route:
        - destination:
            host: mocka.default.svc.cluster.local
            subset: s3
          fallback:
            target:
              host: mocka.default.svc.cluster.local
              subset: s1

Create a unified weight-based traffic routing rule.
1. In the Traffic Rule Definition section of the Traffic Lane page, click Weight-based Routing in the Traffic Routing Rule section.
2. In the Set Unified Routing Rules dialog box, configure the required parameters and click OK. The following example assumes that the inbound request path of all services in the lanes is /mock, and the same traffic routing rule is configured for each lane.
  Parameter
  Description
  realm name
  Set it to *.
  Matching request URI
  Set the Method parameter to Prefix and the Content parameter to /.
  The following figure shows an example of configuring a unified traffic routing rule:

Set the traffic routing weights for the three lanes. The weights determine the proportions of traffic sent to each lane.

In the Traffic Rule Definition section of the Traffic Lane page, click the button next to the number in the Traffic Routing Weight column. In the Edit Traffic Routing Weight dialog box, configure the required parameters and click OK.

Parameter	Description
Ingress service	Set it to mocka.default.svc.cluster.local for the three lanes.
Weight Value	For the s1 lane, set the value to 60. For the s2 lane, set the value to 20. For the s3 lane, set the value to 20.

The following figure shows an example of configuring traffic routing weights.

Step 3: Verify that the end-to-end canary release feature takes effect

Obtain the public IP address of the ingress gateway. For more information, see Step 2: Obtain the IP address of the ASM ingress gateway.
Run the following command to configure environment variables. xxx.xxx.xxx.xxx is the IP address obtained in the previous step.
```
export ASM_GATEWAY_IP=xxx.xxx.xxx.xxx
```

Verify that the end-to-end canary release feature takes effect.

Run the following command to view the calls of services in the three lanes:

for i in {1..100};  do curl http://${ASM_GATEWAY_IP}/ ;  echo ''; sleep 1; done;

Expected output:

-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v3, ip: 192.168.0.2)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v2, ip: 192.168.0.184)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v2, ip: 192.168.0.189)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v2, ip: 192.168.0.184)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v2, ip: 192.168.0.189)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v3, ip: 192.168.0.2)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v2, ip: 192.168.0.184)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v2, ip: 192.168.0.189)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v2, ip: 192.168.0.184)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v2, ip: 192.168.0.189)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v3, ip: 192.168.0.2)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v2, ip: 192.168.0.184)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v2, ip: 192.168.0.189)
-> mocka(version: v2, ip: 192.168.0.184)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v2, ip: 192.168.0.189)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v3, ip: 192.168.0.2)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v3, ip: 192.168.0.2)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v2, ip: 192.168.0.184)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v2, ip: 192.168.0.189)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v3, ip: 192.168.0.2)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v3, ip: 192.168.0.2)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v2, ip: 192.168.0.184)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v2, ip: 192.168.0.189)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v3, ip: 192.168.0.2)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v2, ip: 192.168.0.184)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v2, ip: 192.168.0.189)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)

The output indicates that traffic is sent to the s1, s2, and s3 lanes at a ratio of about 6:2:2, and s1 is used as the baseline lane. If a specific version of a service does not exist in the trace, the corresponding service in the s1 lane is called.

Parameter	Description
realm name	Set it to *.
Matching request URI	Set the Method parameter to Prefix and the Content parameter to /.