Alibaba Cloud Service Mesh:Configure periodic cleanup of monitoring metrics

Step 1: Deploy the sample application

1. Deploy the httpbin and sleep workloads on the data plane cluster. For details about applying YAML templates, see Create using a YAML template.

   Click to view the YAML

   apiVersion: v1
   kind: Service
   metadata:
     name: httpbin
     labels:
       app: httpbin
       service: httpbin
   spec:
     ports:
     - name: http
       port: 8000
       targetPort: 80
     selector:
       app: httpbin
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: httpbin
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: httpbin
         version: v1
     template:
       metadata:
         labels:
           app: httpbin
           version: v1
       spec:
         containers:
         - image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/httpbin:0.1.0
           imagePullPolicy: IfNotPresent
           name: httpbin
           ports:
           - containerPort: 80
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: sleep
     labels:
       app: sleep
       service: sleep
   spec:
     ports:
     - port: 80
       name: http
     selector:
       app: sleep
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: sleep
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: sleep
     template:
       metadata:
         labels:
           app: sleep
       spec:
         terminationGracePeriodSeconds: 0
         containers:
         - name: sleep
           image: registry.cn-hangzhou.aliyuncs.com/acs/curl:8.1.2
           command: ["/bin/sleep", "infinity"]
           imagePullPolicy: IfNotPresent

2. Verify that both Pods are running with sidecar proxies injected (2/2 containers ready):

   kubectl get pod

   Expected output:

   NAME                       READY   STATUS    RESTARTS   AGE
   httpbin-846bxxx694-xxxxx   2/2     Running   0          44s
   sleep-866xxx97f9-xxxxx     2/2     Running   0          44s

Step 2: Enable the REQUEST_COUNT metric

1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.

3. On the Observability Settings page, click the Global tab. In the Monitoring Metrics Setting list, select Enabled next to REQUEST_COUNT in the Server-side Metrics column. Click Submit.

   

Step 3: Generate traffic and check metrics

Send 10 requests from sleep to httpbin:

kubectl exec -it deploy/sleep -- sh -c 'for i in $(seq 1 10); do curl -s httpbin:8000/status/418 > /dev/null; done'

Query the istio_requests_total metric from the httpbin sidecar proxy:

kubectl exec -it deploy/httpbin -c istio-proxy -- curl 0.0.0.0:15020/stats/prometheus | grep istio_requests_total{

Expected output:

istio_requests_total{reporter="destination",source_workload="sleep",source_canonical_service="sleep",...,request_protocol="http",response_code="418",grpc_response_status="",response_flags="-",connection_security_policy="mutual_tls"} 20

The metric shows accumulated request counts, confirming that server-side metrics are being recorded.

Step 4: Enable periodic cleanup and verify eviction

1. Set the Periodic Cleaning Time of Metrics to 10s by following the steps in Configure the cleanup interval.

   Cleanup takes effect after a delay of a few seconds. Adjust the cleanup time as needed during testing.

2. Run the following command to query the metric twice with a 15-second interval:

   kubectl exec -it deploy/httpbin -c istio-proxy -- sh -c 'for i in $(seq 1 2); do echo "# Current content of the istio_requests_total metric: "; curl -s 0.0.0.0:15020/stats/prometheus | grep istio_requests_total{; sleep 15; done'

   Expected output:

   # Current content of the istio_requests_total metric:
   istio_requests_total{reporter="destination",source_workload="sleep",source_canonical_service="sleep",...connection_security_policy="mutual_tls"} 30
   # Current content of the istio_requests_total metric: 

   The test command sends 2 requests in total. The first query returns the metric with the accumulated count along with a line of custom output. After 15 seconds, the second query returns only the custom output line -- the metric data is absent because the periodic cleanup has evicted the stale metric within the 15-second interval.