Alibaba Cloud Service Mesh:Use traffic mirroring across clusters at the service mesh layer

What is traffic mirroring?

The microservice architecture makes application development and deployment faster, but risks exist in changing service versions. Service Mesh (ASM) provides the traffic mirroring feature to mitigate the risks. The feature, also called traffic shadowing, sends production traffic to a mirrored service in real time. The mirrored traffic happens out of band of the critical request path for the production service. When traffic is mirrored, the requests that are sent to the mirrored service version have their Host/Authority headers appended with -shadow. This distinguishes production traffic and mirrored traffic. You can use the feature to mirror production traffic to a test cluster or test service version before the cluster or service version is running in the production environment. This mitigates risks involved in version changes.

Benefits

Scenarios

Traffic mirroring allows you to test a service that is running in the production environment without affecting the end users. You can perform benchmark testing for two versions of a service to determine whether the new version can process inbound requests in the same way as the existing version.

The following table describes typical scenarios where you can use traffic mirroring.

Sample code for using traffic mirroring

The following example YAML file shows how to use traffic mirroring in Istio. In the example, VirtualService routes all traffic to the v1 subset and mirrors the traffic to the v1-mirroring subset. When requests are sent to the v1 subset, the requests are copied and sent to the v1-mirroring subset.

After the v1-mirroring subset sends requests to the v1 version of the application, you can view the application logs. You can see that when the application is invoked, the response is from the v1 subset. You can also see that requests are mirrored to the v1-mirroring subset.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp-traffic-mirroring
spec:
  hosts:
    - myapp
  http:
    - route:
        - destination:
            host: myapp.default.svc.cluster.local
            port:
              number: 8000
            subset: v1
          weight: 100
      mirror:
        host: myapp.default.svc.cluster.local
        port:
          number: 8000
        subset: v1-mirroring

Enable traffic mirroring across clusters

Step 1: Deploy a sample application service in Cluster B

1. Create an httpbin.yaml file that contains the following content:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: httpbin
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: httpbin
     labels:
       app: httpbin
       service: httpbin
   spec:
     ports:
     - name: http
       port: 8000
       targetPort: 80
     selector:
       app: httpbin
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: httpbin-v1
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: httpbin
         version: v1
     template:
       metadata:
         labels:
           app: httpbin
           version: v1
       spec:
         serviceAccountName: httpbin
         containers:
         - image: docker.io/kennethreitz/httpbin
           imagePullPolicy: IfNotPresent
           name: httpbin
           ports:
           - containerPort: 80
   ---

2. Run the following command to deploy the httpbin application service of the v1 version:

   kubectl apply -f httpbin.yaml

Step 2: Configure a routing rule for the ingress gateway in Cluster B

1. Create an httpbin-gateway.yaml file that contains the following content:

   apiVersion: networking.istio.io/v1alpha3
   kind: Gateway
   metadata:
     name: httpbin-gateway
   spec:
     selector:
       istio: ingressgateway
     servers:
     - port:
         number: 80
         name: http
         protocol: HTTP
       hosts:
       - "*"
   ---
   apiVersion: networking.istio.io/v1alpha3
   kind: VirtualService
   metadata:
     name: httpbin
   spec:
     hosts:
     - "*"
     gateways:
     - httpbin-gateway
     http:
     - match:
       - uri:
           prefix: /headers
       route:
       - destination:
           host: httpbin
           port:
             number: 8000

2. Run the following command to deploy the routing rule:

   kubectl apply -f httpbin-gateway.yaml

3. Run the following command to access the ingress gateway in Cluster B and check whether the service works as expected:

   curl http://{IP address of the ingress gateway in Cluster B}/headers

   Sample output:

   {
     "headers": {
       "Accept": "*/*",
       "Host": "47.99.XX.XX",
       "User-Agent": "curl/7.79.1",
       "X-Envoy-Attempt-Count": "1",
       "X-Envoy-External-Address": "120.244.XXX.XXX",
       "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/httpbin;Hash=158e4ef69876550c34d10e3bfbd8d43f5ab481b16ba0e90b4e38a2d53ac****;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
     }
   }

   If the preceding result is returned, it indicates that the service works as expected.

Step 3: Configure an external access rule in the service mesh of Cluster A

The host of the mirrored service uses an external domain name. You need to create a service entry to specify the DNS resolution method of the host.

1. Create an httpbin-cluster-b.yaml file that contains the following content:

   apiVersion: networking.istio.io/v1alpha3
   kind: ServiceEntry
   metadata:
     name: httpbin-cluster-b
   spec:
     hosts:
     - httpbin.mirror.cluster-b
     location: MESH_EXTERNAL
     ports:
     - number: 80   # Specifies the port of the ingress gateway in Cluster B. 
       name: http
       protocol: HTTP
     resolution: STATIC
     endpoints:
     - address: 47.95.XX.XX # Specifies the IP address of the ingress gateway in Cluster B.

2. Run the following command to create the service entry:

   kubectl apply -f httpbin-cluster-b.yaml

3. Create an httpbin-gateway.yaml file that contains the following content:

   The YAML configurations route all traffic to the v1 version of the httpbin service in Cluster A, and mirror the traffic to the httpbin service in Cluster B. httpbin.mirror.cluster-b is the address used to access the external service.

   apiVersion: networking.istio.io/v1alpha3
   kind: Gateway
   metadata:
     name: httpbin-gateway
   spec:
     selector:
       istio: ingressgateway
     servers:
     - port:
         number: 80
         name: http
         protocol: HTTP
       hosts:
       - "*"
   ---
   apiVersion: networking.istio.io/v1alpha3
   kind: VirtualService
   metadata:
     name: httpbin
   spec:
     gateways:
       - httpbin-gateway
     hosts:
       - '*'
     http:
       - match:
           - uri:
               prefix: /headers
         mirror:
           host: httpbin.mirror.cluster-b
           port:
             number: 80
         mirrorPercentage:
           value: 50
         route:
           - destination:
               host: httpbin
               port:
                 number: 8000
               subset: v1

   Note: The traffic destined for httpbin.mirror.cluster-b is the same as the traffic destined for the original destination. The only difference is that the Host/Authority header is suffixed with -shadow. In the preceding YAML configurations, the Host/Authority header of the mirrored traffic is not httpbin.mirror.cluster-b, but the original request header with the -shadow suffix. The host field in the mirror section is used to only find the destination address to which traffic is forwarded and does not change the original Host header.

4. Run the following command to deploy a routing rule:

   kubectl apply -f httpbin-gateway.yaml

5. View the Envoy config dump of the ingress gateway pod in Cluster A.

   "routes": [
            {
             "match": {
              "prefix": "/headers",
              "case_sensitive": true
             },
             "route": {
              "cluster": "outbound|8000|v1|httpbin.default.svc.cluster.local",
              "timeout": "0s",
              "retry_policy": {
               "retry_on": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
               "num_retries": 2,
               "retry_host_predicate": [
                {
                 "name": "envoy.retry_host_predicates.previous_hosts",
                 "typed_config": {
                  "@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
                 }
                }
               ],
               "host_selection_retry_max_attempts": "5",
               "retriable_status_codes": [
                503
               ]
              },
              "request_mirror_policies": [
               {
                "cluster": "outbound|80||httpbin.mirror.cluster-b",
                "runtime_fraction": {
                 "default_value": {
                  "numerator": 500000,
                  "denominator": "MILLION"
                 }
                },
                "trace_sampled": false
               }
              ],

   In the preceding sample code, the request_mirror_policies field specifies the policy for request traffic mirroring, the cluster field specifies the service to which mirrored traffic is sent, and the runtime_fraction field specifies the ratio of traffic to be mirrored. The numerator field is set to 500000 and the denominator field to MILLION, which indicates 50%.