Use an ASMAdaptiveConcurrency CRD to implement adaptive concurrency control - Alibaba Cloud Service Mesh

An ASMAdaptiveConcurrency CustomResourceDefinition (CRD) can dynamically adjust the maximum number of concurrent requests that are allowed for a service based on the sampled request data. If the number of concurrent requests exceeds the maximum value supported by the service, excess requests are rejected to protect the service. This topic describes how to use an ASMAdaptiveConcurrency CRD to implement adaptive concurrency control.

Prerequisites

A Service Mesh (ASM) instance whose version is 1.12.4.19 or later is created. For more information, see Create an ASM instance.
The cluster is added to the ASM instance. For more information, see Add a cluster to an ASM instance.
A kubectl client is connected to the cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Background information

Services are expected to reject excess requests if the load capacity is exceeded. This prevents other chain reactions. You can configure destination rules for an ASM instance to implement basic circuit breaking. You must specify a threshold for triggering circuit breaking, such as a specific number of pending requests. If the number of access requests on the data plane of the ASM instance exceeds the threshold, excess requests are rejected.However, it is difficult to accurately estimate the load capacity of a service in actual scenarios.

An ASMAdaptiveConcurrency CRD uses an adaptive concurrency control algorithm to dynamically adjust the concurrency limit for a service by periodically comparing the sample latency with the calculated minimum latency and performing a series of computations. This way, the concurrency limit is near the load capacity of the service, and excess requests are rejected. If a request is rejected, the HTTP status code 503 and the error message reached concurrency limit are returned.

During the periodic calculation of the minimum round-trip time (MinRTT), the number of connections is limited to a small value that is specified by the min_concurrency parameter. After you create an ASMAdaptiveConcurrency CRD for a service, we recommend that you create a destination rule to enable the retry feature for the service. This way, the requests that are rejected during MinRTT calculation can be served as much as possible based on the retries of the sidecar proxy.

Step 1: Deploy sample applications

Deploy the testserver and gotest applications. Set the load capacity of the testserver application to 500 concurrent requests. Requests that exceed the concurrency limit are queued for processing. Set the time required to process each request to 1000 ms. Set each replica of the gotest application to initiate 200 requests at a time.

Deploy a testserver application.

Create a testserver.yaml file that contains the following content:

Expand to view details

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: testserver
  name: testserver
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: testserver
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: testserver
    spec:
      containers:
      - args:
        - -m
        - "500"
        - -t
        - "1000"
        command:
        - /usr/local/bin/limited-concurrency-http-server
        image: registry.cn-hangzhou.aliyuncs.com/acs/asm-limited-concurrency-http-server:v0.1.1-gee0b08f-aliyun
        imagePullPolicy: IfNotPresent
        name: testserver
        ports:
        - containerPort: 8080
          protocol: TCP

The -m parameter specifies the maximum number of concurrent requests that the application supports. The -t parameter specifies the time required to process each request.

Run the following command to deploy the testserver application:
```
kubectl apply -f testserver.yaml
```

Deploy a Service named testserver in the testserver application.

Create a testservice.yaml file that contains the following content:

Expand to view details

apiVersion: v1
kind: Service
metadata:
  labels:
    app: testserver
  name: testserver
  namespace: default
spec:
  internalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
    - name: metrics
      port: 15020
      protocol: TCP
      targetPort: 15020
  selector:
    app: testserver
  type: ClusterIP

Run the following command to deploy the testserver Service:
```
kubectl apply -f testservice.yaml
```

Deploy a gotest application.

Create a gotest.yaml file that contains the following content:

Expand to view details

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: gotest
  name: gotest
  namespace: default
spec:
  replicas: 0
  selector:
    matchLabels:
      app: gotest
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: gotest
    spec:
      containers:
      - args:
        - -c
        - "200"
        - -n
        - "10000"
        - -u
        - testserver:8080
        command:
        - /root/go-stress-testing-linux
        image: xocoder/go-stress-testing-linux:v0.1
        imagePullPolicy: Always
        name: gotest
        resources:
          limits:
            cpu: 500m

Run the following command to deploy the gotest application:
```
kubectl apply -f gotest.yaml
```

Step 2: Create an ASMAdaptiveConcurrency CRD

Use kubectl to connect to your ASM instance. For more information. see Use kubectl on the control plane to access Istio resources.

Create an adaptiveconcurrency.yaml file that contains the following content:

Expand to view details

apiVersion: istio.alibabacloud.com/v1beta1
kind: ASMAdaptiveConcurrency
metadata:
    name: sample-adaptive-concurrency
    namespace: default
spec:
    workload_selector:
        labels:
            app: testserver
    sample_aggregate_percentile:
        value: 60
    concurrency_limit_params:
        max_concurrency_limit: 500
        concurrency_update_interval: 15s
    min_rtt_calc_params:
        interval: 60s
        request_count: 100
        jitter:
            value: 15
        min_concurrency: 50
        buffer:
            value: 25

Parameter	Type	Description	Required
workload_selector	WorkloadSelector	The workload selector that defines how to select the pod to use.	Yes
labels	map	The labels that are used to match the pod to select.	Yes
sample_aggregate_percentile	Percent	The sampling percentile. Requests are sampled based on this percentile to calculate the sample round-trip time (SampleRTT).	Yes
value	int	The percentile value. Valid values: 0 to 100.	Yes
concurrency_limit_params	ConcurrencyLimitParams	The configurations related to the concurrency limit.	Yes
max_concurrency_limit	int	The concurrency limit, which is the maximum number of concurrent requests that are allowed. Default value: 1000.	No
concurrency_update_interval	duration	The interval for updating the concurrency limit. Example: 60s.	Yes
min_rtt_calc_params	MinRTTCalcParams	The configurations related to MinRTT calculation.	Yes
interval	duration	The interval for calculating the MinRTT. Example: 120s.	No
request_count	int	The number of requests that are used to calculate the MinRTT. Default value: 50.	No
jitter	Percent	The percentage that is used to add a random jitter to the interval for calculating the MinRTT. For example, if the interval parameter is set to 120s and the jitter parameter is set to 50, the interval for calculating the MinRTT is random(120, 120 + (120 × 50%)). Default value: 15.	No
min_concurrency	int	The number of concurrent requests that are used to calculate the MinRTT. The value is also the initial concurrency limit of the concurrency controller. The value of this parameter must be much lower than the load capacity of the service. This ensures that responses are returned at the minimum latency during MinRTT calculation. Default value: 3.	No
buffer	Percent	The fluctuation range of the proper latency. The value is a percentage. For example, if a 10% fluctuation around a latency of 100 ms falls in a reasonable range, set this parameter to 10. Default value: 25.	No

Run the following command to create an ASMAdaptiveConcurrency CRD:
```
kubectl apply -f adaptiveconcurrency.yaml
```

Step 3: Enable Managed Service for Prometheus

To help you understand the running status of the concurrency controller and optimize the parameters, you can export the metrics related to adaptive concurrency control to Managed Service for Prometheus. You can view the running status of the concurrency controller in the Managed Service for Prometheus console.

Enable Managed Service for Prometheus. For more information, see Managed Service for Prometheus.
Configure ServiceMonitor in the cluster so that Managed Service for Prometheus can obtain the data of the testserver Service.
1. Create a servicemonitor.yaml file that contains the following content:
```
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: testserver-envoy-metrics
  namespace: default
spec:
  endpoints:
    - interval: 5s
      path: /stats/prometheus
      port: metrics
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      app: testserver
```
2. Use kubectl to connect to the ACK cluster based on the information in the kubeconfig file, and then run the following command to create a ServiceMonitor:
```
kubectl apply -f servicemonitor.yaml
```

Step 4: Verify whether the ASMAdaptiveConcurrency CRD takes effect

Set the number of replicas to 5 for the gotest application.
A replica of the gotest application initiates 200 requests at a time. Five replicas of the application initiate 1,000 requests at a time in total.
1. Log on to the ACK console.
2. In the left-side navigation pane of the ACK console, click Clusters.
3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
4. In the left-side navigation pane of the details page, choose Workloads > Deployments.
5. On the Deployments page, set Namespace to default and choose More > View in YAML in the Actions column of the gotest application.
6. In the Edit YAML dialog box, set replicas to 5 and click Update.

Upload the following JSON file to import a Grafana dashboard. You can view the running status of the concurrency controller on the imported dashboard. For more information, see ARMS documentation.

Expand to view the JSON file

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "description": "monitoring ASM Adaptive Concurrency",
  "editable": true,
  "gnetId": 6693,
  "graphTooltip": 0,
  "id": 3239002,
  "iteration": 1651922323976,
  "links": [],
  "panels": [
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$cluster",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "hiddenSeries": false,
      "id": 22,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_rq_blocked{service=\"$service\", pod=\"$pod\"}",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "RqBlocked",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$cluster",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 0
      },
      "hiddenSeries": false,
      "id": 24,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_burst_queue_size{service=\"$service\", pod=\"$pod\"}",
          "format": "time_series",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "HeadRoom",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$cluster",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 8
      },
      "hiddenSeries": false,
      "id": 26,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_concurrency_limit{service=\"$service\",pod=\"$pod\"}",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "ConcurrencyLimit",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$cluster",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 8
      },
      "hiddenSeries": false,
      "id": 28,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_gradient{service=\"$service\",pod=\"$pod\"}",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Gradient",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$cluster",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 16
      },
      "hiddenSeries": false,
      "id": 32,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_min_rtt_msecs{service=\"$service\",pod=\"$pod\"}",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "MinRTT(msec)",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ms",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$cluster",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 16
      },
      "hiddenSeries": false,
      "id": 34,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_sample_rtt_msecs{service=\"$service\",pod=\"$pod\"}",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "SampleRTT(msec)",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ms",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "test-adaptive-concurrency_1217520382582089",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 24
      },
      "hiddenSeries": false,
      "id": 30,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_min_rtt_calculation_active{service=\"$service\",pod=\"$pod\"}",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "MinRTTCalc",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    }
  ],
  "refresh": "5s",
  "schemaVersion": 26,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "selected": true,
          "text": "edas120_1217520382582089",
          "value": "edas120_1217520382582089"
        },
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "cluster",
        "options": [],
        "query": "prometheus",
        "queryValue": "",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "type": "datasource"
      },
      {
        "allValue": null,
        "current": {
          "isNone": true,
          "selected": false,
          "text": "None",
          "value": ""
        },
        "datasource": "$cluster",
        "definition": "label_values(envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_burst_queue_size,service)",
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "service",
        "options": [],
        "query": "label_values(envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_burst_queue_size,service)",
        "refresh": 2,
        "regex": "",
        "skipUrlSync": false,
        "sort": 1,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "datasource": "$cluster",
        "definition": "label_values(envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_concurrency_limit, pod)",
        "error": null,
        "hide": 0,
        "includeAll": true,
        "label": null,
        "multi": true,
        "name": "pod",
        "options": [],
        "query": "label_values(envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_concurrency_limit, pod)",
        "refresh": 2,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      }
    ]
  },
  "time": {
    "from": "now-15m",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "",
  "title": "ASM Adaptive Concurrency",
  "uid": "000000084",
  "version": 3
}

On the dashboard, select the cluster in which the ASMAdaptiveConcurrency CRD resides. Set the Service parameter to testserver and the Pod parameter to ALL.
The gotest application initiates 1,000 requests to the testserver Service, but the number of concurrent requests that are processed by the testserver Service is lower than 500. This indicates that the ASMAdaptiveConcurrency CRD takes effect.