How to use the route-level circuit breaking feature of ASM - Alibaba Cloud Service Mesh

Service Mesh (ASM) allows you to configure resources such as virtual services and destination rules to implement non-intrusive traffic governance for microservices. For example, you can use features such as traffic routing, throttling, circuit breaking, and traffic mirroring. This topic describes how to use the route-level circuit breaking feature of ASM.

Prerequisites

The configuration file required in this topic is downloaded.
An ASM instance of Enterprise Edition or Ultimate Edition is created, and the version of the ASM instance is V1.13.4 or later. For more information, see Create an ASM instance.
The cluster is added to the ASM instance. For more information, see Add a cluster to an ASM instance.
An ingress gateway is deployed. For more information, see Create an ingress gateway.
The Bookinfo and NGINX services are created. For more information, see Deploy an application in an ASM instance.
kubectl is connected to the ASM instance. For more information, see Use kubectl on the control plane to access Istio resources.

An Istio gateway is deployed. For more information, see Manage Istio gateways.

Show the YAML code of the Istio gateway

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: bookinfo-gateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - bf2.example.com
    port:
      name: http
      number: 80
      protocol: http

A virtual service is created. For more information, see Manage virtual services.

Show the YAML code of the virtual service

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: bookinfo
  namespace: default
spec:
  gateways:
  - bookinfo-gateway
  hosts:
  - bf2.example.com
  http:
  - match:
    - uri:
        exact: /productpage
    - uri:
        prefix: /static
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    name: productpage-route-name1
    route:
    - destination:
        host: productpage
        port:
          number: 9080
  - match:
    - uri:
        prefix: /httpbin
    name: httpbin-route-name1
    rewrite:
      uri: /
    route:
    - destination:
        host: httpbin.foo.svc.cluster.local
        port:
          number: 80

The traffic generation tool hey is installed. For more information, visit hey at GitHub.

Background information

ASM allows you to configure the circuit breaking feature in the trafficPolicy field. If the number of access requests on a network reaches the circuit breaking threshold, new access requests are denied. When you configure a destination rule, you can set the following two configuration items of the circuit breaking feature in the trafficPolicy field:

ConnectionPoolSettings: specifies the maximum number of connections to a service. If the number of requests reaches this value, new requests enter the pending state, time out, or are retried.
OutlierDetection: specifies the rule for removing unhealthy instances from a load balancing pool.

Istio provides the following circuit breaking configurations for a destination rule. For more information, see Destination Rule.

Show the YAML code of the destination rule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      consecutive5xxErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100

The circuit breaking feature of open source Istio works in the following way: Envoy scans hosts in upstream clusters periodically to check whether exceptions occur on the hosts. Hosts on which exceptions occur are ejected from the connection pool. This way, circuit breaking is implemented on the upstream hosts. This working mechanism has the following disadvantages:

Service-based circuit breaking is implemented instead of API-based circuit breaking.
The feature takes effect for traffic intended for a service only after routing has occurred.

To overcome the preceding disadvantages, ASM extends Envoy filter chains on the data plane and provides com.aliyun.break filter to support route-level circuit breaking. On the control plane, ASM allows you to use the ASMCircuitBreaker CustomResourceDefinition (CRD) to implement circuit breaking without the need to focus on the underlying implementation.

In this example, the Bookinfo and HTTPBin services are used to simulate slow requests and error requests. The Bookinfo service is deployed in the default namespace. The HTTPBin service is deployed in the foo namespace. The ingress gateway is deployed in the istio-system namespace. The HTTPBin service is deployed in the backend test service of the foo namespace to verify the scope of circuit breaking. 场景示例

Scenarios

You can enable the route-level circuit breaking feature for an ASM gateway. The feature supports HTTP and Google Remote Procedure Call (gRPC) protocols. You can add the Istio gateway configurations to the YAML file of an ASMCircuitBreaker CRD. In this example, a circuit breaking rule is configured for the httpbin-route-name1 route that directs requests to /httpbin.

apiVersion: istio.alibabacloud.com/v1beta1
kind: ASMCircuitBreaker
metadata:
  name: ingressgateway
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: istio-ingressgateway
  isGateway: true
  configs:
    - match:
        vhost:
          name: "bf2.example.com"
          port: 80
          route:
            name_match: nginx-route-name1
      breaker_config:
        slow_request_rt: 0.1s
        break_duration: 90s
        window_size: 10s
        max_slow_requests: 10
        min_request_amount: 3
        error_percent:
          value: 60
        custom_response:
          header_to_add:
            x-envoy-circuitbreak: "true"
          body: "hello, break!"
          status_code: 499

The following table describes some of the fields. For the descriptions of more fields in the ASMCircuitBreaker CRD, see Description of ASMCircuitBreaker fields.

Field	Description
`isGateway: true`	Specifies whether to apply the configurations to gateways. Default value: `false`.
`value: 60`	The maximum percentage of error requests that are allowed. If the percentage exceeds this value of 60% and the number of requests is equal to or greater than three, new requests are denied.
`max_slow_requests: 10`	The maximum number of slow requests that are allowed. If the number exceeds this value of 10, new requests are denied. Note If the response time of a request exceeds the value of the `slow_request_rt` field, the request is considered a slow request.
`break_duration: 90s`	The circuit breaking period during which new requests are denied. Unit: seconds. In this example, the circuit breaking period is set to 90 seconds.

Configure and verify a circuit breaking rule

In this example, a circuit breaking rule is configured for the httpbin-route-name1 route of the bf2.example.com:80 virtual host to implement throttling.

Create an asmcircuitbreaker-test-gw.yaml file and add the following content to the file:

apiVersion: istio.alibabacloud.com/v1beta1
kind: ASMCircuitBreaker
metadata:
  name: ingressgateway
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: istio-ingressgateway
  isGateway: true
  configs:
    - match:
        vhost:
          name: "bf2.example.com"
          port: 80
          route:
            name_match: httpbin-route-name1
      breaker_config:
        slow_request_rt: 0.1s
        break_duration: 90s
        window_size: 10s
        max_slow_requests: 10
        min_request_amount: 3
        error_percent:
          value: 60
        custom_response:
          header_to_add:
            x-envoy-overload: "true"
          body: "hello, break!"
          status_code: 499

Run the following command to create an ASMCircuitBreaker:
```
kubectl apply -f asmcircuitbreaker-test-gw.yaml
```
Simulate one-second latency in response to a request or a 500 response error of the Httpbin service. Repeat the operation ten times.
- Run the following command to simulate one-second latency in response to a request:
```
curl -H 'host: bf2.example.com'  http://${ASM_GATEWAY_IP}/httpbin/delay/1 -v
```
- Run the following command to simulate a 500 response error of the HTTPBin service:
```
curl -H 'host: bf2.example.com'  http://${ASM_GATEWAY_IP}/httpbin/status/500 -v
```
Expected output:
```
< HTTP/1.1 499 Unknown
< Content-Length: 12
< Content-Type: text/plain
< x-envoy-overload: true
< Date: Thu, 13 Jan 2022 03:03:09 GMT
< Server: istio-envoy
<
Hello,Break!
```
If Hello,Break! is returned, requests are denied.
Run the following command to access the /productpage interface of the Bookinfo service:
```
curl -H 'host: bf2.example.com'  http://${ASM_GATEWAY_IP}/productpage -v
```
If HTTP 200 is returned on the Productpage page, requests are not denied.

References

Description of ASMCircuitBreaker fields