Configure a notification template and a webhook template - Application Real-Time Monitoring Service

If the alert rules of the specified notification policy are triggered, ARMS sends alert notifications by DingTalk, Lark, WeCom, text message, phone call, email, or webhook. ARMS provide preset notification templates for each notification method. If a preset notification template cannot meet your requirements, you can modify the notification template. This topic describes the Go template syntax, preset notification templates, and variables in the templates.

Go template syntax

ARMS uses the Go template syntax to render notification templates. When the Go template syntax is used to render a template, the template is filled based on the context.

Note

The alert parameters in a notification template consist of multiple labels. You can use labels to customize the alert parameters that you need. For example, the alertname label in this example indicates the alert name.

The following sample code describes the syntax of common Go templates:

{
  "alerts": [
    {
      "annotations": {
        "message": "Namespace: arms-prom / Pod: kube-state-metrics-ccb59dbff-jljg4 / Container: kube-state-metrics. The memory usage of the container has exceeded 80%. Current value: 15.52%.",
        "value": "15.521240234375"
      },
      "endsAt": "2023-02-22T07:27:15.404000000Z",
      "fingerprint": "bec72890cc2c7b4a027e008df0cd1013",
      "labels": {
        "container": "kube-state-metrics",
        "severity": "warning",
        "instance": "10.0.80.186:10255",
        "clustername": "klyz1688-kubernetes-1",
        "alertname": "The memory usage of the container has exceeded 80%.",
        "_aliyun_arms_involvedObject_name": "klyz1688-kubernetes-1",
        "pod_name": "kube-state-metrics-ccb59dbff-jljg4",
        "_aliyun_arms_involvedObject_kind": "cluster",
        "name": "k8s_kube-state-metrics_kube-state-metrics-ccb59dbff-jljg4_arms-prom_359508f3-7e76-4740-b915-41ea48849641_0",
        "namespace": "arms-prom"
      },
      "startsAt": "2023-02-22T07:18:15.578000000Z",
      "status": "firing"
    }
  ],
  "commonAnnotations": {
    "message": "Namespace: arms-prom / Pod: kube-state-metrics-ccb59dbff-jljg4 / Container: kube-state-metrics. The memory usage of the container has exceeded 80%. Current value: 15.52%.",
    "value": "15.521240234375"
  },
  "commonLabels": {
    "container": "kube-state-metrics",
    "severity": "warning",
    "instance": "10.0.80.186:10255",
    "clustername": "klyz1688-kubernetes-1",
    "alertname": "The memory usage of the container has exceeded 80%.",
    "_aliyun_arms_involvedObject_name": "klyz1688-kubernetes-1",
    "pod_name": "kube-state-metrics-ccb59dbff-jljg4",
    "_aliyun_arms_involvedObject_kind": "cluster",
    "name": "k8s_kube-state-metrics_kube-state-metrics-ccb59dbff-jljg4_arms-prom_359508f3-7e76-4740-b915-41ea48849641_0",
    "namespace": "arms-prom"
  },
  "groupLabels": {
    "alertname": "The memory usage of the container has exceeded 80%."
  },
  "status": "firing",
  "startTime":"2023-02-22 07:18:15",
  "endTime":"The time when the alert was cleared",
  "level":"error",
  "dispatchRuleName":"Notification policy name"
  "alarmId":"123456"
}

Dot (`.`)

A dot is used to indicate the context in the current scope.

Example 1: Obtain all the context in the top-level scope that is shown in the preceding sample code:

{{ . }}

Output: The context in the preceding sample code is returned.

Example 2: Obtain the level field in the context:

{{ .level }}

Output:

error

Example 3: Obtain the commonLabels.alertname field in the context:

{{ .commonLabels.alertname }}

Output:

The memory usage of the container has exceeded 80%.

Variables

The data passed to a Go template can be stored in the variables of the template and accessed. For example, if you use the $context variable in the { {$context := .}} action and save passed data in the variable, you can use the {{$context}} action to access the variable.

Example 1: Assign an alert name to the $alertname variable. Then, the value of the $alertname variable is returned:

{{ $alertname := .commonLabels.alertname }} {{$alertname}}

Output:

The memory usage of the container has exceeded 80%.

Example 2: Assign the first element in the alerts list to the $alert0 variable. Then, the alertname of the first alert (element) is returned:

{{ $alert0 := index .alerts 0 }} {{$alert0.labels.alertname}}

Output:

The memory usage of the container has exceeded 80%.

if and else actions

You can execute an if action to check data. If the if action cannot meet your business requirements, you can execute an else action. For null values, 0, nil pointer, empty strings, or zero-length strings, false is returned.

Example 1: If the value of the severity tag in an alert event is warning, a P3 alert is generated:

{{if eq "warning" .commonLabels.severity }} P3 alert {{ end }}

Output:

P3 alert

Example 2: If the value of the severity tag in an alert event is critical, a P1 alert is generated. Otherwise, a P2 alert is generated:

{{if eq "critical" .commonLabels.severity }} P1 alert {{ else }} P2 alert {{ end }}

Output:

P2 alert

for action

The for action is used to repeatedly execute code.

Example 1: Traverse the alerts list and obtain the alertname of each alert:

{{ for .alerts}} {{.labels.alertname}} \n {{end}}

Output:

The memory usage of the container has exceeded 80%.

range action

The range action is used to traverse the specified content.

Example 1: Traverse integers between 0 and 9:

{{ $ran := range 0 10 }} {{ for $ran }} {{.}} {{end}}

Output:

 0 1 2 3 4 5 6 7 8 9

index function

The index function is used to obtain the elements in an array.

Example 1: Obtain the first element in the alerts list and assign a value to the $alert0 variable:

{{ $alert0 := index .alerts 0 }} {{$alert0.labels.alertname}}

Output:

 The memory usage of the container has exceeded 80%.

Comparison functions

Common comparison functions:

eq: equal to
ne: not equal to
lt: less than
le: less than or equal to
gt: greater than
ge: greater than or equal to

Example: If the value of the severity tag in the commonLabels section is equal to warning, a P3 alert is generated:

{{if eq "warning" .commonLabels.severity }} P3 alert {{ end }}

Output:

  P3 alert

Logical functions and, or, and not

Example: If the value of the severity tag in the commonLabels section is equal to warning or warnning, a P3 alert is generated:

{{if eq "warning" .commonLabels.severity or eq "warning" .commonLabels.severity }} P3 alert {{ end }}

Output:

  P3 alert

Common built-in functions

printf: formats the output data.

Example: The value of the first alert in the alerts list is rounded to two decimal places and returned:

{{ $alert0 := index .alerts 0 }} {{ printf "%.2f" $alert0.annotations.value }}

Output:

  15.52

humanizePercentage: formats percentages.

Example: Convert the string 0.95332123123124 to a more readable percentage:

{{ "0.95332123123124" | humanizePercentage }}

Output:

95.33%

humanizeDate: formats a date by converting a millisecond timestamp to a date. The default date format is yyyy-MM-dd HH:mm:ss.

Example: Convert the timestamp 1671074058234 to a date:

{{ "1671074058234" | humanizeDate }}

Output:

2022-12-15 11:14:18

len: obtains the length of an array.

Example: Obtain the length of the alerts list:

{{ len .alerts }}

Output:

Preset notification templates

Email notification template

Alert name: {{ .commonLabels.alertname }}
{{if .commonLabels.clustername }}
Cluster name: {{ .commonLabels.clustername }}
{{ end }}
{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}
Application name: {{ .commonLabels._aliyun_arms_involvedObject_name }}
{{ end }}
Notification policy: {{ .dispatchRuleName }}
Alert triggered at: {{ .startTime }}
Alert content: {{ if .newIncidentIn }} (New event) {{ end }} {{ for .alerts }} {{.annotations.message}} {{ if .generatorURL }} <a href="{{.generatorURL}}" >Details URL</a> {{end}} {{end}}

Template content	Description
`Alert name: {{ .commonLabels.alertname }}`	The name of the alert. By default, notification policies are grouped by `alertname`. Therefore, all events related to an alert have the same alert name.
`{{if .commonLabels.clustername }} Cluster name: {{ .commonLabels.clustername }} {{ end }}`	If the `clustername` label exists in the event, the value of the `clustername` label is rendered as the cluster name field.
`{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }} Application name: {{ .commonLabels._aliyun_arms_involvedObject_name }} {{ end }}`	If the type of the object that triggers the alert is `app`, the name of the corresponding application is returned.
`Notification policy: {{ .dispatchRuleName }}`	The name of the notification policy that matches the alert.
`Alert triggered at: {{ .startTime }}`	The time when the alert was triggered.
`Alert content:{{ if .newIncidentIn }} (New event) {{end}} {{ for .alerts }} {{.annotations.message}} {{if .generatorURL }} <a href="{{.generatorURL}}" >Details URL</a> {{end}} {{end}}`	The alert event list is traversed to render `message` values. If an event contains the `generatorUrl` field, the value of the `generatorUrl` field is rendered as the URL of the event details. Note The HTML syntax `<a></a>` is used in the preset template to render URLs. `<a href="{{.generatorUrl}}">Details URL</a>` `{{ if .newIncidentIn }} (New event) {{end}}`: If a new event occurs in the alert group, the current notification is marked and `New event` is returned.

After the sample code is used for rendering, the following results are returned:

Alert name: The memory usage of the container has exceeded 80%.
Cluster name: klyz1688-kubernetes-1
Notification policy: Notification policy name
Alert triggered at: 2023-02-22 07:18:15
Alert content: Namespace: arms-prom / Pod: kube-state-metrics-ccb59dbff-jljg4 / Container: kube-state-metrics. The memory usage of the container has exceeded 80%. Current value: 15.52%.

Text message and phone call notification templates

Text message and phone call notification templates have the same content as the email template except that they do not contain the URL of alert details.

{{ .level }} alert occurred
Alert name: {{ .commonLabels.alertname }}
{{if .commonLabels.clustername }}
Cluster name: {{ .commonLabels.clustername }}
{{ end }}
{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}
Application name: {{ .commonLabels._aliyun_arms_involvedObject_name }}
{{ end }}
Notification policy: {{ .dispatchRuleName }}
Alert triggered at: {{ .startTime }}
Alert content: {{ if .newIncidentIn }} (New event)
 {{end}} {{ for .alerts }} {{ .annotations.message }} {{ end }}

Chatbot notification template

{{ if .newIncidentIn }} (New event) {{end}}
{{if .commonLabels.clustername }}
 >  Cluster name: {{ .commonLabels.clustername }}
 {{ end }}
{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}
 >  Application name: {{ .commonLabels._aliyun_arms_involvedObject_name }}
 {{ end }}
{{ for .alerts }} >  {{ .annotations.message }} {{if .generatorURL }} [Details URL]({{.generatorURL}})  {{end}}{{if .annotations._aliyun_arms_insights_analyze_link }}[<font color='#ff0000'>Root cause</font>]({{ .annotations._aliyun_arms_insights_analyze_link}}){{ end }}{{if  eq "1" .labels._aliyun_arms_denoise_code }} (Important: {{.labels._aliyun_arms_important_reason }}) {{end}}

{{end}}

The robot rendering result is returned in Markdown. Different chatbots support different Markdown features.

Reference: >
Font color:
```
<font color='#ff0000'>Root cause</font>
```
Note
Only DingTalk and WeCom allow you to change font colors. Lark does not allow you to change font colors.
Line break:
- DingTalk line breaks: Press Enter twice in the template.
- Lark line breaks: Press Enter only once in the template.
- WeCom line breaks: Press Enter only once in the template.

Webhook template

{
  "Alert name":"{{ .commonLabels.alertname }}
{{if .commonLabels.clustername }}", "Cluster name":"{{ .commonLabels.clustername }} {{ end }}
{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}", "Application name":"{{ .commonLabels._aliyun_arms_involvedObject_name }} {{ end }}",
  "Notification Policy":"{{ .dispatchRuleName }}",
  "Alert triggered at":"{{ .startTime }}",
  "Alert content":"{{ for .alerts }} {{ .annotations.message }} {{ end }}"
}

After the sample code is used for rendering, the following results are returned:

{
  "Alert name": "The memory usage of the container has exceeded 80%.",
  "Cluster name": "klyz1688-kubernetes-1",
  "Notification policy": "Notification policy name",
  "Alert triggered at":"{{ .startTime }}",
  "Alert content": "Namespace: arms-prom / Pod: kube-state-metrics-ccb59dbff-jljg4 / Container: kube-state-metrics. The memory usage of the container has exceeded 80%. Current value: 15.52%."
}

Variables

The following table lists the variables that can be used in a notification template.

Variable	Description
alarmId	The ID of the original alert.
alerts	The original alert event. The data type is List<Alert>.
alerts.annotations	The annotations of the original alert. The data type is Map<String, String>.
alerts.endsAt	The time when the original alert was cleared. The time when the alert was cleared is the time when the alert was triggered plus the period of time consumed to clear the alert.
alerts.startsAt	The time when the original alert was triggered.
alerts.fingerprint	The fingerprint of the original alert. Alerts of one label group have the same fingerprint.
alerts.labels	The labels of the original alert.
alerts.status	The status of the original alert. Valid values: firing: The alert is being triggered. resolved: The alert is cleared.
commonAnnotations	The same annotations for a group of original alerts.
commonLabels	The same labels for a group of original alerts.
groupLabels	The group labels. Alerts are grouped based on the labels that are specified in the group condition of the notification policy.
status	The status of a group of alerts. Valid values: firing: The alerts are being triggered. resolved: The alerts are cleared.
startTime	The time when the alert group was created.
endTime	The time when the group of alerts was cleared, which is the time when the last original alert in the group was cleared.
level	The severity level of the alert. Valid values: critical: P1 error: P2 warning: P3 page: P4
dispatchRuleName	The name of the notification policy for the alert group.

Go template syntax

Dot (.)

Variables

if and else actions

for action

range action

index function

Comparison functions

Logical functions and, or, and not

Common built-in functions

Preset notification templates

Email notification template

Text message and phone call notification templates

Chatbot notification template

Webhook template

Variables

Dot (`.`)