Alibaba Cloud Service Mesh:Configure an outlier detection policy

Prerequisites

The preparations are completed, and the HTTPBin and sleep services are deployed. For more information, see Preparations.

Feature description

The following section describes the process of outlier detection in Service Mesh:

1. When a request is sent to a service, a Service Mesh proxy records the number of errors responded by each endpoint of the service over a period of time. HTTP responses with a status code of 5xx are considered errors. The length of this period is specified by the interval parameter.

2. When the number of errors responded by an endpoint exceeds a certain threshold, the endpoint is marked as unhealthy and is ejected from the service's list of available endpoints. This threshold is specified by the consecutiveErrors parameter.

   Note

   If the proportion of available endpoints to the total service endpoints is less than a given proportion after an endpoint is removed, the endpoint is not removed from the list of available endpoints. The proportion is specified by the maxEjectionPercent parameter.

3. After a service endpoint is ejected, it will not receive any more requests for a given base duration. After this duration, the service endpoint is added back to the list of available service endpoints. If this service endpoint is ejected for the second time because it returns 5xx errors consecutively, the duration of the second ejection will be twice the base duration. The duration of the third ejection will be three times the base duration, and so on. The base duration is specified by the baseEjectionTime parameter.

Procedure

1. Create a destination rule for the HTTPBin service to implement outlier detection.

   1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

   2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Traffic Management Center > DestinationRule.

   3. On the DestinationRule page, click Create. On the Create page, configure the following parameters, and click Preview. Confirm that the YAML content is correct, click Submit, and then click Create.

      The following code shows the YAML file for preview.

      apiVersion: networking.istio.io/v1beta1
      kind: DestinationRule
      metadata:
        name: httpbin
        namespace: default
        labels: {}
      spec:
        host: httpbin.default.svc.cluster.local
        trafficPolicy:
          outlierDetection:
            consecutiveErrors: 1
            interval: 1s
            baseEjectionTime: 15s
            maxEjectionPercent: 100
      

2. Check whether the outlier detection policy takes effect.

   1. Use kubectl to connect to the Container Service for Kubernetes (ACK) cluster based on the information in the kubeconfig file, and then run the following command:

      kubectl -n legacy exec -it deploy/sleep -- curl httpbin.legacy:8000/status/502 -I

      Expected output:

      HTTP/1.1 502 Bad Gateway
      server: envoy
      date: xxx, xx xxx 202x xx:xx:xx GMT
      content-type: text/html; charset=utf-8
      access-control-allow-origin: *
      access-control-allow-credentials: true
      content-length: 0
      x-envoy-upstream-service-time: 4

      According to the configuration, a service endpoint is ejected if the endpoint returns an error response within 1 second, and the value of maxEjectionPercent is 100%. At this time, the only endpoint of the HTTPBin service is ejected, and the service becomes unavailable.

   2. Run the following command again within 15 seconds:

      kubectl -n legacy exec -it deploy/sleep -- curl httpbin.legacy:8000/status/502 -I

      Expected output:

      HTTP/1.1 503 Service Unavailable
      content-length: 19
      content-type: text/plain
      date: xxx, xx xxx 202x xx:xx:xx GMT
      server: envoy

      The expected output indicates that when a request is sent to the /status/502 path of the HTTPBin service, the service returns a 503 Service Unavailable response, instead of an HTTP 502 error. This indicates that all endpoints of the HTTPBin service have been ejected and the outlier detection policy takes effect.