The alert management system sends notifications for alerts that match the conditions specified in a route and merge policy.
Route and merge policy
You can configure merge settings for alerts on the Route and Merge Policy tab. The related parameters are Merge by, Action Policy, Group Wait, Group Interval, and Repeat Interval. Alerts are merged into one merge set only when the values of the parameters configured between route and merge policies are the same.
For example, a service is deployed on Host 1 and Host 2. Host 1 and Host 2 trigger a high-CPU alert at 1-minute intervals from 20:00 and 20:01, respectively. In this case, you can create a route and merge policy based on the service name to merge alerts. After alert notifications for the first and new alerts are sent, alert notifications for duplicate alerts in the merge set are delayed.
Merge by
You can specify merge conditions based on alert attributes and labels to merge alerts. You can use built-in merge conditions or specify custom merge conditions.
Merge condition type | Description |
Built-in | Simple Log Service provides the following built-in merge conditions:
|
Custom | You can specify custom merge conditions based on alert attributes and labels to merge alerts.
|
Action Policy
An action policy defines how Simple Log Service sends alert notifications. You can associate an action policy when you create a route and merge policy or an alert rule. If you set Action Policy to Dynamic Action Policy, the action policy that you specify when you create an alert rule is used. If you set Action Policy to a specific action policy, the specified action policy is used.
Group Wait, Group Interval, and Repeat Interval
Scenario 1: Only Alert A is generated in the period that is specified by Group Wait.
In this scenario, Group Wait is set to 5 Seconds, Group Interval is set to 1 Minutes, and Repeat Interval is set to 4 Hours. In the following figure, Alert A is represented in orange, and Alert B is represented in blue.
Alert A is generated at 00:00:00 and a merge set is created at the same time. However, the alert notification is not immediately sent because a value is specified for Group Wait.
When the value of Group Wait is reached at 00:00:05, Simple Log Service sends the first alert notification.
After the first alert notification is sent, the system periodically checks the data in the merge set based on the value of Group Interval. Alert B is generated and added to the merge set during the first period of Group Interval. Therefore, Simple Log Service sends the second alert notification at 00:01:05.
Then, the system continues to check the data in the merge set based on the value of Group Interval. Only Alert A and Alert B exist in the merge set. When the value of Repeat Interval is reached at 04:01:05, Simple Log Service sends the third alert notification because no data changes in the merge set after the second notification is sent.
Scenario 2: Alert A and Alert B are generated in the period that is specified by Group Wait.
In this scenario, Group Wait is set to 5 Seconds, Group Interval is set to 1 Minutes, and Repeat Interval is set to 4 Hours. In the following figure, Alert A is represented in orange, and Alert B is represented in blue.
Alert A and Alert B are generated in the time range from 00:00:00 to 00:00:05 and a merge set is created within the same time range. However, the alert notification is not immediately sent because a value is specified for Group Wait.
When the value of Group Wait is reached at 00:00:05, Simple Log Service sends the first alert notification.
After the first alert notification is sent, the system periodically checks the data in the merge set based on the value of Group Interval. The system finds that only Alert A and Alert B exist in the merge set from 00:00:05 to 04:01:05. When the value of Repeat Interval is reached at 04:01:05, Simple Log Service sends the second alert notification because no data changes in the merge set after the first notification is sent.
Parameter | Description |
Group Wait | The interval after which Simple Log Service sends an alert notification when the merge set is created. We recommend that you set the unit to Seconds. |
Group Interval | The interval after which Simple Log Service sends an alert notification when the data in the merge set changes. If new alerts are added to a merge set or the status of an alert changes, the system considers that the data in the merge set changes. We recommend that you set the unit to Minutes. You can also set the unit to Seconds to receive alert notifications at the earliest opportunity. |
Repeat Interval | The interval after which Simple Log Service sends an alert notification when the data in the merge set remains unchanged. If no alerts are added to a merge set or alert status does not change, the system considers that the data in the merge set remains unchanged. We recommend that you set the unit to Hours. Note If you set Action Policy to Dynamic Action Policy when you create an alert rule, you do not need to configure Repeat Interval in the associated alert policy. By default, the system uses the repeat interval specified in your alert rule to overwrite the repeat interval specified in your route and merge policy. |
Examples
When you create an alert rule, you can configure alert policies to merge or not merge alerts.
Scenario 1: Merge alerts
Merge alerts based on the project to which your alert rule belongs, the env label, and the service label.
Alerts
// Alert A { "alert_name": "Alert1", "project": "Project1", "labels": { "env": "test", "service": "service1" } } // Alert B { "alert_name": "Alert2", "project": "Project1", "labels": { "env": "prod", "service": "service2" } } // Alert C { "alert_name": "Alert3", "project": "Project1", "labels": { "env": "test", "service": "service1" } } // Alert D { "alert_name": "Alert4", "project": "Project1", "labels": { "env": "prod", "service": "service2" } }Configuration
Merge result
Alert A and Alert C are merged into one merge set. Alert B and Alert D are merged into another merge set.
Scenario 2: Do not merge alerts
If you set Merge by to Alert ID and All Labels when you configure a route and merge policy, alerts that are triggered by the same alert rule and have the same labels are merged into one merge set. In the following example, two alert rules are used.
In the Alert Rule 1, Group Evaluation is set to Custom Label, and Alert Policy is not set to Advanced Mode. In the route and merge policy, Merge by is set to Alert ID and All Labels. In this case, the alert management system separately sends alert notifications for alerts triggered by Host 1, Host 2, and Host 3.
In the Alert Rule 2, Group Evaluation is set to No Grouping and Alert Policy is not set to Advanced Mode. In the route and merge policy, Merge by is set to Alert ID and All Labels. In this case, the alert management system sends one alert notification for the merge set that includes all alerts triggered by Host 1, Host 2, and Host 3.
