Configure the periodic cleaning of metrics in an ASM instance - Alibaba Cloud Service Mesh

Service Mesh (ASM) generates metrics for all inbound traffic, outbound traffic, and service traffic within ASM to monitor service performance. These metrics include information such as total traffic, error rate, and request response time. However, long-term operation generates a large amount of metrics, significantly increasing resource consumption of Envoy and Prometheus. Therefore, ASM provides a configuration for periodic cleaning of monitoring metrics to allow you to clean up metrics cached in Envoy that have not been used for a period of time. This helps improve the memory utilization of Envoy and decrease the network load when Prometheus pulls metrics. This topic describes how to configure periodic cleaning of metrics and its usage example.

Prerequisites

A cluster has been added to the ASM instance of 1.18 or later. For more information, see Add a cluster to an ASM instance.
Automatic sidecar proxy injection is enabled.

Procedure

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.
On the Observability Settings page, click the Global tab, enter the Periodic Cleaning Time of Metrics in the Enable Periodic Cleaning of Metrics, and then click Submit.
Important
We recommend that you set the periodic cleaning time of metrics to at least twice thescrape_interval configured for Prometheus to ensure that Prometheus can successfully capture the metrics before they are cleaned up.

Example

Deploy a sample application

Run the following commands to deploy a sample application in the cluster on the data plane. For specific operations, see Create a Linux application using an orchestration template or Create using a YAML template.

Click to view details

apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/httpbin:0.1.0
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      containers:
      - name: sleep
        image: registry.cn-hangzhou.aliyuncs.com/acs/curl:8.1.2
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent

Run the following command to check the application status.

kubectl get pod

Expected output:

NAME                       READY   STATUS    RESTARTS   AGE
httpbin-846bxxx694-xxxxx   2/2     Running   0          44s
sleep-866xxx97f9-xxxxx     2/2     Running   0          44s

Enable a metric and perform verification

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.
On the Observability Settings page, click the Global tab. In the Monitoring Metrics Setting list, check the Enabled next to the REQUEST_COUNT metric in the Server-side Metrics column. Then, click Submit.

Run the following command to access an HTTPBin application.

 kubectl exec -it deploy/sleep -- sh -c 'for i in $(seq 1 10); do curl -s httpbin:8000/status/418 > /dev/null; done'

Run the following command to view the monitoring metrics of the HTTPBin application.

kubectl exec -it deploy/httpbin -c istio-proxy -- curl 0.0.0.0:15020/stats/prometheus | grep istio_requests_total{

Expected output:

istio_requests_total{reporter="destination",source_workload="sleep",source_canonical_service="sleep",...,request_protocol="http",response_code="418",grpc_response_status="",response_flags="-",connection_security_policy="mutual_tls"} 20

Configure periodic cleaning of monitoring metrics

Set the periodic cleaning time of metrics to 10s by following the steps in Procedure.
There will be a few seconds of delay for clean-up. During the verification process, you can adjust the clean-up time as needed.

After the configuration is complete, run the following command to view the monitoring metrics of the HTTPBin application.

kubectl exec -it deploy/httpbin -c istio-proxy -- sh -c ' for i in $(seq 1 2); do echo "# Current content of the istio_requests_total metric: "; curl -s 0.0.0.0:15020/stats/prometheus | grep istio_requests_total{; sleep 15; done'

Expected output:

# Current content of the istio_requests_total metric: 
istio_requests_total{reporter="destination",source_workload="sleep",source_canonical_service="sleep",...connection_security_policy="mutual_tls"} 30
# Current content of the istio_requests_total metric:

The above test commands send 2 requests in total. Before this feature is enabled, each request returns a line of metrics and a line of custom output. After this feature is enabled, the metrics are cleaned up within 15s, and each request returns a line of custom output.