Reference log content in alert notifications - Simple Log Service

You can use the labels, annotations, fire_results, and results variables in an alert template to reference log content in alert notifications.

Simple Log Service can display the details of an alert in alert notifications. This way, you can view alert details when you receive an alert notification. To include alert details in an alert notification, you must configure the required parameters when you create an alert monitoring rule and add the labels, annotations, fire_results, and results variables to the alert template that is used. For more information, see Variables in new alert templates.

labels: the labels of an alert
annotations: the annotations of an alert
fire_results: the data records for which an alert is triggered
results: the query parameters and intermediate results

For example, if you monitor access to NGINX, you can use the following query statement to obtain information such as the status code that is returned and indicates an access error, the IP address of the machine on which the error occurred, and the number of times that the error occurred. Then, you can create an alert monitoring rule based on the information and reference the information in the alert template. If a status code is 400 or a larger number, Simple Log Service sends you an alert notification that includes the status code, IP address of the machine on which the error occurred, and the number of times that the error occurred.

Query statement

status >=400 | SELECT status, __source__ AS ip, count(*) AS cnt GROUP BY status,ip

Query results

Use the labels variable

You can use the labels variable in an alert template to reference log content in alert notifications.

Alert monitoring rule
You can add labels or configure group evaluation when you create an alert monitoring rule. After an alert is triggered, information about labels is generated and included in the alert message.
- Configure group evaluation: After you configure group evaluation, the fields that you select for group evaluation are automatically added to the information about labels. For more information, see Use the group evaluation feature.
  Important
  When you select fields for group evaluation, select a field that can identify a monitored entity and whose values can be enumerated. For example, if you select the status field for group evaluation, different status codes trigger different alerts when error logs are recorded. Do not select the fields that cannot identify monitored entities. Using such fields result in numerous groups. If the trigger condition in a group is met, an alert is triggered. If a large number of groups exist, alert storms may occur. In this case, you may miss out on important alert information.
- Add labels: Labels are identifying attributes. For more information, see Labels.
For example, if you want to add information such as the log source, environment, and status code to alert messages, you can add the app:nginx and env:prod labels and select the status field for group evaluation.
Alert message
After an alert is triggered, Simple Log Service generates an alert message that contains the following information:
```
{
    "labels": {
        "app": "nginx",
        "env": "prod",
        "status": "404"
    }
}
```
Alert template
You can add the labels variable to the alert template to reference specified log content in alert notifications. For more information about how to create an alert template, see Create an alert template.

Use the annotations variable

You can use the annotations variable in an alert template to reference log content in alert notifications.

Alert monitoring rule
You can manually add information about annotations or turn on Auto-Add Annotations to automatically add information about annotations when you create an alert monitoring rule. After an alert is triggered, information about annotations is generated and included in the alert message. For more information, see Annotations.
Note
Simple Log Service provides two annotations by default.
For example, if you want to add information such as the status code that is returned and indicates an access error, IP address of the machine on which the error occurred, and number of times that the error occurred to alert messages, you can add annotations such as status:${status}, ip:${ip}, and cnt:${cnt}.
- Manually add information about annotations
- Turn on Auto-Add Annotations to automatically add information about annotations

Alert message

After an alert is triggered, Simple Log Service generates an alert message that contains the following information:

{
    "annotations": {
        "title": "Alert is triggered by an NGINX access error",
        "desc": "The error that is indicated by the status code 400 occurred 15 times"
        "status": "400"
        "ip": "127.0.0.1"
        "cnt": "15"
        "__count__": "1"
    }
}

Alert template
You can add the annotations variable to the alert template to reference specified log content in alert notifications. For more information about how to create an alert template, see Create an alert template.

Use the fire_results variable

You can use the fire_results variable in an alert template to reference log content in alert notifications.

Alert monitoring rule
Simple Log Service generates alert messages based on the alert monitoring rule that you create.

Alert message

After an alert is triggered, Simple Log Service generates an alert message that contains the following information. The information includes the status code that is returned and indicates an access error, IP address of the machine on which the error occurred, and number of times that the error occurred.

{
    "fire_results": [
        { "status": "401", "ip": "127.0.0.1", "cnt": "3" },
        { "status": "400", "ip": "127.0.0.1", "cnt": "7" },
        { "status": "501", "ip": "127.0.0.1", "cnt": "4" },
        { "status": "404", "ip": "127.0.0.1", "cnt": "4" },
        { "status": "402", "ip": "127.0.0.1", "cnt": "6" },
        null
    ]
}

Alert template
You can add the fire_results variable to the alert template to reference specified log content in alert notifications. For more information about how to create an alert template, see Create an alert template.

Use the results variable

You can use the results variable in an alert template to reference log content in alert notifications.

Alert monitoring rule
Simple Log Service generates alert messages based on the alert monitoring rule that you create.

Alert message

Note

If an alert monitoring rule contains multiple query statements, the results array contains multiple items. Each item corresponds to a query statement.

{
    "results": [{
        "store_type": "log",
        "region": "cn-hangzhou",
        "project": "test-alert",
        "store": "nginx-access-log",
        "query": "status >= 400 | select status, __source__ as ip, count(*) as cnt group by status, ip",
        "start_time": 1640006894,
        "end_time": 1640007014,
        "dashboard_id": "",
        "raw_results": [
            { "status": "401", "ip": "127.0.0.1", "cnt": "3" },
            { "status": "400", "ip": "127.0.0.1", "cnt": "7" },
            { "status": "501", "ip": "127.0.0.1", "cnt": "4" },
            { "status": "404", "ip": "127.0.0.1", "cnt": "4" },
            { "status": "402", "ip": "127.0.0.1", "cnt": "6" },
            null
        ],
        "raw_result_count": 6,
        "fire_result": {
            "status": "401",
            "ip": "127.0.0.1",
            "cnt": "3"
        },
        "has_sql": true,
        "truncated": false,
        "role_arn": ""
    }]
}

Alert template
You can add the results variable to the alert template to reference specified log content in alert notifications. For more information about how to create an alert template, see Create an alert template.

Template variables

If you reference a variable in the JSON format in an alert template, such as {{ alert.results[0].raw_results | to_json }}, the content of the variable is displayed as a JSON string in alert notifications. You can also use a cycle method when you configure the alert template to display the content in alert notifications by line. This way, the content is easy to read. For more information, see Syntax for new alert templates. Example:

{%- for result in alert.fire_results %}
- status: {{ result.status }}, count: {{ result.cnt }}
{%- endfor %}

Simple Log Service allows you to use built-in functions to format the content of alert notifications. For more information, see Syntax for new alert templates and Built-in functions in alert templates.