Application Real-Time Monitoring Service:Use Kubernetes labels to dispatch alerts

How it works

ARMS enriches alerts with Kubernetes labels through the following flow:

1. kube-state-metrics exposes label metrics for each Kubernetes resource, such as kube_pod_labels and kube_node_labels.

2. You add a special annotation (_aliyun_arms_enrich_desc) to a Prometheus alert rule. The annotation value is a PromQL statement that queries label metrics for the alerting pod.

3. When the alert fires, ARMS runs the PromQL statement, retrieves the pod labels, and attaches them to the alert. Notification policies then match on these labels and route the alert to the appropriate contacts.

The following diagram summarizes the data flow:

Pod labels ──► kube-state-metrics ──► kube_pod_labels metric
                                            │
Alert fires ──► ARMS runs PromQL ───────────┘
                   │
                   ▼
         Alert enriched with pod labels
                   │
                   ▼
         Notification policy matches on labels ──► Contacts notified

Prerequisites

Before you begin, make sure that you have:

• A Container Service for Kubernetes (ACK) cluster with kube-state-metrics

• A Prometheus instance monitoring the ACK cluster. For setup instructions, see Create a Prometheus instance to monitor an ACK cluster

Label a pod in the ACK cluster

Add a custom label to the pod template of your target application:

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, find your cluster and click Applications in the Actions column.

3. On the Deployments page, find the target application and choose More > View in YAML in the Actions column.

4. In the pod template section of the YAML, add a custom label. For example, add app: arms-prom-ack-arms-prometheus.

   

5. Click Update.

Create a Prometheus alert rule

Create an alert rule that triggers on the condition you want to monitor:

1. Log on to the ARMS console. In the left-side navigation pane, choose Managed Service for Prometheus > Prometheus Alert Rules.

2. In the upper-right corner of the Prometheus Alert Rules page, click Create Prometheus Alert Rule.

3. Set Alert Name to EnrichDemo and set Alert Condition to When Container CPU Usage Greater than 2%, Meet the alarm conditions.

4. Configure the remaining parameters and click Save. For details, see Create an alert rule for a Prometheus instance.

Enrich alerts with pod labels

While still on the Create Prometheus Alert Rule page, expand Advanced Settings and add an annotation with the following values:

Example annotation value:

kube_pod_labels{pod_name=${pod_name}}

In this statement, ${pod_name} is replaced at runtime with the pod name from the firing alert. ARMS runs the query, retrieves all labels for that pod, and attaches them to the alert.

Configure a notification policy

Add a label-based matcher to a notification policy so that enriched alerts are routed to the correct contacts:

1. In the ARMS console, open the Notification Policies page.

2. In the Match Alert Event Rules section, add a condition. For example: This condition matches all alerts from pods with the label app: arms-prom-ack-arms-prometheus and routes them to the contacts defined in this policy.

      label_app Equal To arms-prom-ack-arms-prometheus

For details on creating and managing notification policies, see Notification policies.

Verify alert dispatch

After you configure the alert rule and notification policy, verify that alerts are dispatched correctly:

1. In the ARMS console, go to the Alert Sending History page. For details, see View historical alerts.

2. Confirm that alerts carry the expected labels (for example, label_app: arms-prom-ack-arms-prometheus) and are routed to the correct contacts.

Limits

What's next

• Create and manage notification policies

• Create alert rules for Prometheus instances

• View historical alerts