You can configure the alert ingestion system of Log Service as an Alertmanager component in Prometheus. After you complete the configuration, Prometheus sends alerts to the alerting system of Log Service. Then, the alerting system denoises the alerts and sends alert notifications.

Prerequisites

An alert ingestion service and an alert ingestion application are created. For more information, see Configure webhook URLs for alert ingestion.

Configure Prometheus

In the Prometheus configuration file, add the settings of the alertmanagers parameter. The following code shows the settings. For more information, see the alertmanager_config section in the Configuration topic of the Prometheus official website. 

# Alertmanager configuration
alerting:
  alertmanagers:
  - path_prefix: /event/webhook/RAMAK_{ACCESS_KEY_ID}/{WEBHOOK_APP_ID}
    api_version: v2
    static_configs:
    - targets:
      - {ALIYUN_SLS_ENDPOINT}
Parameter Description Example
path_prefix The path. Set this parameter to the subpath of the webhook URL that is generated after you create an alert ingestion service and an alert ingestion application in the alert ingestion system of Log Service. For more information, see Obtain webhook URLs. 
-path_prefix: event/webhook/RAMAK_WEDC***YEBD/Prometheus-alert01_k8s
targets The destination to which you want to send alerts. Set the value to an endpoint for Log Service, for example, cn-heyuan-intranet.log.aliyuncs.com. For more information, see Obtain webhook URLs.
Note If your Prometheus server is deployed on an Elastic Compute Service (ECS) instance, we recommend that you select the region where the ECS instance resides and use an internal endpoint that can be accessed over a LAN or virtual private cloud (VPC). If your Prometheus server is not deployed on an ECS instance, you can select a region and use the public endpoint that can be accessed over the Internet. 
-targets:
        -cn-heyuan-intranet.log.aliyuncs.com

Prometheus alerts

The following example shows the content of a Prometheus alert.

Note If a Prometheus alert contains the severity field, Log Service maps the severity of the alert to the corresponding severity after the alert is ingested into Log Service. If a Prometheus alert does not contain the severity field, the alert severity is mapped to Medium by default. For more information, see Severity levels. 
[
    {
        "annotations": {
            "description": "description info",
            "summary": "High request latency"
        },
        "endsAt": "2020-10-28T12:28:52.710Z",
        "startsAt": "2020-10-28T12:23:37.710Z",
        "generatorURL": "http://127.0.0.1:9090/graph?g0.expr=go_threads%7Binstance%3D%22localhost%3A9090%22%2Cjob%3D%22prometheus%22%7D+%3E+0\\u0026g0.tab=1",
        "labels": {
            "alertname": "HighErrorRate",
            "instance": "localhost:9090",
            "job": "prometheus",
            "severity": "page"
        }
    }
]

Field mappings

After a Prometheus alert is ingested into Log Service, the alert is converted to an alert that is supported by Log Service by using field mapping. The following example shows the fields of a Log Service alert:
{
    "aliuid": "{The ID of the Alibaba Cloud account to which the alert ingestion application belongs}",
    "alert_instance_id": "{The alert instance ID that is automatically generated}",
    "project": "{The project to which Alert Center belongs}",
    "region": "{The region of the endpoint to which the alert is sent}",
    "alert_id": "HighErrorRate",
    "alert_type": "sls_pub",
    "alert_name": "HighErrorRate",
    "next_eval_interval": 78,
    "alert_time": 1603859020,
    "fire_time": 1603859017,
    "resolve_time": 0,
    "status": "firing",
    "labels": {
        "alertname": "HighErrorRate",
        "instance": "localhost:9090",
        "job": "prometheus"
    },
    "annotations": {
        "__pub_alert_region__": "{The region of the endpoint to which the alert is sent}",
        "__config_app__": "sls_pub_alert",
        "__pub_alert_service__": "{The ID of the alert ingestion service}",
        "__pub_alert_app__": "{The ID of the alert ingestion application}",
        "__pub_alert_protocol__": "prometheus",
        "severity": "page"
    },
    "severity": 2,
    "policy": {
        "alert_policy_id": "{The alert policy that is specified for the alert ingestion application}",
        "action_policy_id": "{The action policy that is specified for the alert ingestion application}",
        "repeat_interval": "{The cycle that is specified for the alert ingestion application}"
    },
    "drill_down_query": "http: //127.0.0.1:9090/graph?g0.expr=go_threads%7Binstance%3D%22localhost%3A9090%22%2Cjob%3D%22prometheus%22%7D+%3E+0\\u0026g0.tab=1"
}

The following table describes the mappings between the alert attributes of Log Service and the alert fields of Prometheus.

Alert attribute Prometheus field Description
aliuid None The ID of the Alibaba Cloud account to which the alert ingestion application belongs.
alert_id alertname The ID of the alert monitoring rule.
alert_type None The type of the alert. Valid value: sls_pub.
alert_name alertname The name of the alert monitoring rule.
status None The status of the alert. Valid values: firing and resolved.
  • If the value of the endsAt field in the Prometheus alert is greater than the value of the alert_time field, the status is firing.
  • If the value of the endsAt field in the Prometheus alert is less than the value of the alert_time field, the status is resolved.
next_eval_interval None The interval at which the alert is evaluated.

If the current alert status is firing, the value of this parameter is calculated by using the following formula: Evaluation interval = (endsAt - alert_time)/4.

alert_time None The time when Log Service receives the Prometheus alert.
fire_time startsAt The time when the alert is triggered. The fire_time field is mapped to the startsAt field in the Prometheus alert.
resolve_time None The time when the alert is cleared.
  • If the value of the endsAt field in the Prometheus alert is greater than the value of the alert_time field, the value of the resolve_time is 0.
  • If the value of the endsAt field in the Prometheus alert is less than the value of the alert_time field, the value of the resolve_time field is a timestamp. The timestamp is the same as the timestamp of the endsAt field.
labels labels The labels of the alert.
  • If the severity field exists in the labels field of the Prometheus alert, the severity field is added to the annotations field after the alert is ingested into Log Service.
  • If you add a label on the Enrichment tab when you create the alert ingestion application, the label is added to the labels field.
    Note If the key of the specified label on the Enrichment tab is the same as a subfield in the labels field of the Prometheus alert, the label on the Enrichment tab prevails.
annotations annotations After the Prometheus alert is ingested into Log Service, the following fields are added to the annotations field of the corresponding Log Service alert:
  • __config_app__: "sls_pub_alert"
  • __pub_alert_service__: {The ID of the alert ingestion service}
  • __pub_alert_app__: {The ID of the alert ingestion application}
  • __pub_alert_protocol__: "prometheus"
  • __pub_alert_region__: {The region of the endpoint to which the alert is sent}
If you add an annotation on the Enrichment tab when you create the alert ingestion application, the annotation is added to the annotations field.
Note If the key of the specified annotation on the Enrichment tab is the same as a subfield in the labels field of the Prometheus alert, the annotation on the Enrichment tab prevails.
severity severity The severity of the alert. For more information, see Severity levels.
policy None The alert policy that is specified for the alert ingestion application. For more information, see Description of the policy variable.
project None The project to which Alert Center belongs. For more information, see Project.
drill_down_query generatorURL The value of the generatorURL field in the Prometheus alert is displayed.