Merge Istio metrics with application metrics - Alibaba Cloud Service Mesh

For an application integrated with Prometheus, you can use sidecar proxies to expose application metrics by merging Istio metrics with the application metrics. This topic describes how to merge Istio metrics with application metrics.

Prerequisites

A Service Mesh (ASM) instance whose version is 1.17.2.30 or later is created. For more information, see Create an ASM instance or Update an ASM instance.

Important

This feature is not suitable for scenarios where application metrics are sensitive. Application metrics are collected together with Istio metrics. Users who have permissions to view Istio metrics can view application metrics.

Feature description

After you enable the feature of merging Istio metrics with application metrics, ASM merges application metrics with Istio metrics. The prometheus.io annotations of the application are added to all pods on the data plane to enable the metric scraping capabilities of Prometheus. If these annotations already exist, they will be overwritten. Sidecar proxies merge Istio metrics with application metrics. Prometheus can obtain the merged metrics from the :15020/stats/prometheus endpoint.

After this feature is enabled, ASM will perform the following operations on pods:

The scraping methods annotated with prometheus.io annotations before ASM is introduced will be saved to the environment variables of sidecar proxies.
The Prometheus metrics generated by the applications that are intercepted by ASM and the metrics of sidecar proxies will be merged and exported to the :15020/stats/prometheus endpoint for Prometheus to scrape.

Step 1: Deploy a sample application

Deploy an application that contains prometheus.io annotations. For more information, see Deploy an application in an ASM instance.

The following sample code provides an example of the deployment file for the productpage application:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage-v1
  labels:
    app: productpage
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productpage
      version: v1
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9080"
        prometheus.io/path: "/metrics"
      labels:
        app: productpage
        version: v1
    spec:
      ......

After the productpage application is deployed, the following pod information is generated:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.aliyun.com/pod-ips: 172.17.0.132
    kubectl.kubernetes.io/default-container: productpage
    kubectl.kubernetes.io/default-logs-container: productpage
    kubernetes.io/psp: ack.privileged
    prometheus.io/path: /metrics
    prometheus.io/port: '9080'
    prometheus.io/scrape: 'true'

In the preceding YAML template, you can see the following prometheus.io annotations:

prometheus.io/path: /metrics
prometheus.io/port: '9080'
prometheus.io/scrape: 'true'

Step 2: Enable the feature of merging Istio metrics with application metrics

By default, this feature is disabled for ASM instances. You can configure this feature for an ASM instance globally or for each workload by adding the prometheus.istio.io/merge-metrics: "true" annotation to the pods of the application. In this example, this feature is configured for an ASM instance.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.
In the Config Info section, click Enable next to Merge Istio's Metrics with Application Metrics. In the Enable Merging of Istio's Metrics with Application Metrics message, click OK.

Note

If you no longer need to use this feature, you can disable this feature by performing one of the following operations:

If this feature is enabled for an ASM instance, click Disable next to Merge Istio's Metrics with Application Metrics in the Config Info section. In the Disable Merging of Istio's Metrics with Application Metrics message, click OK.
If this feature is enabled for a workload, add the prometheus.istio.io/merge-metrics: "false" annotation to the corresponding pod.

Step 3: View the result

After you enable the feature of merging Istio metrics with application metrics, view the pod information after you restart the pod.

The following sample code provides an example of the YAML template of the pod:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.aliyun.com/pod-ips: 172.17.0.126
    kubectl.kubernetes.io/default-container: productpage
    kubectl.kubernetes.io/default-logs-container: productpage
    kubernetes.io/psp: ack.privileged
    prometheus.io/path: /stats/prometheus
    prometheus.io/port: '15020'
    prometheus.io/scrape: 'true'
    ......
spec:
  containers:
    - args:
        - proxy
        - sidecar
        ......
      env:
        ......
        - name: ISTIO_PROMETHEUS_ANNOTATIONS
          value: '{"scrape":"true","path":"/metrics","port":"9080"}'

The preceding YAML template shows that the prometheus.io annotations are updated to the following content. In addition, a new environment variable named ISTIO_PROMETHEUS_ANNOTATIONS is added. The following sample code shows that the value of the environment variable is the same as that of the prometheus.io annotations configured for the application.

prometheus.io/path: /stats/prometheus
prometheus.io/port: '15020'
prometheus.io/scrape: 'true'