Application Real-Time Monitoring Service:Integrate a self-managed Prometheus instance into Alert Management

Step1: Create an integration to integrate a self-managed Prometheus instance

1. Log on to the ARMS console. In the left-side navigation pane, choose Alert Management > Integrations.

2. On the Alert Integration tab, click Prometheus.

3. In the Create Prometheus Integration dialog box, enter a name and a description, specify the automatic recovery time of alert events, and then click Save.

   Note If an alert event is not triggered again within the specified period of time, the alert event is automatically cleared.

4. Integrate a Prometheus instance based on the type of the instance and click OK.

   Prometheus instances managed by using Prometheus-Operator

   1. Create a file named arms-alertmanager-config.yaml on an on-premises server, and then paste the following content to the file:

      - scheme: https
        bearer_token: ******
        static_configs:
          - targets: ['alerts.aliyuncs.com']

      Note

      You can obtain the value of bearer_token in the ARMS console.

   2. Create a secret named arms-am-config by using the arms-alertmanager-config.yaml file.

      kubectl create secret generic arms-am-config --from-file=arms-alertmanager-config.yaml

   3. Create a file named prometheus-patch.yaml on the on-premises server, and then paste the following content to the file:

      spec:
        additionalAlertManagerConfigs:
          key: arms-alertmanager-config.yaml
          name: arms-am-config

   4. Run the following command to obtain the Prometheus object:

      kubectl get prometheus

      Sample response:

      NAME                                                VERSION   REPLCAS   AGE
      prometheus-operator-160879-prometheus               v2.18.2   1         4d1h

   5. Run the following command to update the content of the prometheus-patch.yaml file to the Prometheus object:

      kubectl patch prometheus prometheus-operator-160879-prometheus --patch "$(cat prometheus-patch.yaml)" --type=merge

   Self-managed Prometheus instances created by using other methods

   1. Modify the prometheus.yaml configuration file.

      In this example, a Prometheus 2.35 instance is integrated. For information about other versions of Prometheus instances, see Prometheus documentation.

      alerting:
        alertmanagers:
        - api_version: v1
          authorization:
            type: Bearer
            credentials: ${token}
          scheme: https
          static_configs:
          - targets: ["alerts.aliyuncs.com"]

      Note

      You can obtain the value of bearer_token in the ARMS console.

   2. Visit the configuration page of the Prometheus instance in your browser and check whether the prometheus.yaml file is modified.

   Other Prometheus observability systems

   Thanos uses the --alertmanagers.config parameter of the thanos rule command to specify the Alertmanager configuration file. Perform the preceding steps to configure the address and token for the integration. For more information, see Thanos documentation.

   VictoriaMetric uses the -notifier.config parameter of the ./bin/vmalert command to specify the notifier configuration file. Perform the preceding steps to configure the address and token for the integration. For more information, see VictoriaMetric documentation.

(Optional) Step 2: Modify the integration

Alert Management provides field mappings between Prometheus alert sources and ARMS alert events. You can also add or modify the mappings between fields on the Edit Integration page.

1. On the Alert Integration tab, find the integration that you want to manage and click Edit in the Actions column.
2. In the Event Mapping section, click Send Test Data.
3. In the Send Test Data dialog box, enter the alert content of a third-party alert source in the JSON format and click Send.
   Note

   • If the message Uploaded. No events are generated. Configure mappings based on the original data. appears, the fields of the alert source are not mapped to the fields of ARMS alert events. The data that is sent is displayed in the left-side box. This allows you to select the source fields when you configure mappings.
   • If the message Uploaded. appears, the alert content is reported to the Alert Event History page. For more information, see View historical alert events.
4. In the Send Test Data dialog box, click Disable.
5. In the left part of the Event Mapping section, click the data records for which you want to configure mappings to view the details.

6. In the right part of the Event Mapping section, configure field mappings between the alert source and ARMS alert events.

   1. Optional:In the Select Root Node section, specify whether to enable batch processing.
      If an array node exists in the alert data, you can specify the array node as the root node. The data that belongs to the root node is processed in batches.

      After you select Use Batch Processing, select the array node to be batch processed as the root node.

      Note If multiple array nodes exist in the alert data, you can select only one of the array nodes for batch processing.
   2. In the Map Source Fields to Target Fields section, map the fields of the alert source to the alert fields of ARMS.
      Click the mapping icon to change the method for field mapping.

      • Direct: The specified field of the alert source is directly mapped to the specified alert field of ARMS.
      • Series: You can use delimiters to concatenate multiple fields of the alert source into one field, and then map this field to the specified alert field of ARMS. Only special characters can be used as delimiters.
      • Conditional: The specified alert source fields are mapped to the ARMS alert fields only when the field values meet the specified conditions.
      • Mapping table: The alert source severity is mapped to the ARMS alert severity. The mapping table contains only the severity field.

      The following table describes the alert fields of ARMS.

      Alert field	Description
      alertname	The name of the custom alert.
      severity	The severity level of the alert. You must configure mappings for this field. The mapping method must be set to Direct.
      message	The description of the alert. The description is used as the content of the alert message. The description cannot exceed 15,000 characters in length.
      value	The sample value of a metric.
      imageUrl	The URL of the line chart that contains Grafana metrics. The URL is used to map the line chart.
      check	The check item of the alert. Examples: CPU, JVM, Application Crash, and Deployment.
      source	The source of the alert.
      class	The type of the object that triggers the alert event, for example, host.
      service	The source service of the alert. Example: Login Service.
      startat	The timestamp that represents the start time of the event.
      endat	The timestamp that represents the end time of the event.
      generatorUrl	The URL of the event details.

7. Configure field deduplication for alert events.
   To reduce duplicate data, the system uses relevant fields as the basis for deduplication. ARMS Alert Management allows you to preview the deduplication grouping results of historical event data that is displayed in the Event Mapping section. You can change the fields to be deduplicated.

   Note You can configure deduplication only for events that are not cleared.
   1. In the Event Deduplication section on the Integration Details page, select the fields that are used for deduplication.
      If multiple events have the same value for a specified field, the events are merged into one alert notification.
   2. Click Deduplication Test to preview the alert group after deduplication.
      Note The deduplication test takes effect only on the latest 10 data records that are uploaded in the left part of the Event Mapping section.
8. After you configure the settings, click Save.

View the details about an alert event

1. In the left-side navigation pane of the ARMS console, choose Alert Management > Alert Event History.
2. On the Alert Event History page, click the name of the alert event to view the event details. For more information, see View historical alert events.

Manage the integration

In the left-side navigation pane, choose Alert Management > Integrations. On the Alert Integration tab, you can perform the following operations on the integrations that you created:

• View the details of an integration: Find the integration and then click the row. On the Integration Details page, view the integration details.

• Update a key: Find the integration and then choose More > Update Key in the Actions column. In the message that appears, click OK.

  Important

  After the key is updated, you must reconfigure the self-managed Prometheus instance. For more information, see Configure Prometheus instances.

• Modify an integration: Find the integration and then click Edit in the Actions column. On the Integration Details page, modify the integration information and then click Save.
• Enable or disable an integration: Find the integration and then click Disable or Enable in the Actions column.
• Delete an integration: Find the integration and then click Delete in the Actions column. In the message that appears, click OK.
• Add an event processing flow to an integration: Find the integration and click Add Event Processing Flow in the Actions column. For more information, see Work with event processing flows.
• Create a notification policy: Find the integration for which you want to create a notification policy, and click More in the Actions column. In the list that appears, click Create Notification Policy. For more information, see Create and manage a notification policy.