Simple Log Service:Monitor data transformation jobs (new version)

Item

Description

Purpose

A rule of this type monitors the latency that occurs when data is consumed from shards in data transformation jobs. If the latency during data transformation exceeds the threshold in the rule, an alert is triggered.

Associated dashboard

For more information, see Shard details analysis.

Sample SQL

In the following template, replace {job_name} with the name of the data transformation job that you want to monitor.

__topic__: etl_metrics and 
job_name: {job_name} and
"_etl_:connector_meta.action": ingest
| select
  split_part(
    "_etl_:connector_meta.task_name",
    '#',
    2
  ) as shard,
  max_by("_etl_:connector_metrics.lags", __time__) as lags
group by
  shard
having
  shard is not null
limit
  all

Monitoring rule

• For Trigger Condition, set When to the query result contains and set the expression to lags > 120, which means that the threshold is when the latency reaches or exceeds 120 seconds.

• Set the query interval to 5 minutes.

• Set the query frequency to 5 minutes.

Handling method

You can clear triggered alerts based on the following rules:

1. If the data transformation job is newly created, check for one hour whether the metric value drops below the threshold. A newly created job requires some time to process historical data. If the job is not newly created, proceed to the next step.

2. If the data volume in the source Logstore significantly increases, perform the following operations based on your business requirements:

   • If Transforming Speed (events/s) increases and Shard Consuming Latency (s) decreases, the job is automatically scaling out resources to respond to the increasing data volume in the source Logstore. In this case, wait five minutes and then check whether the latency drops below the threshold. If the latency does not, proceed to the next step.

   • If Transforming Speed (events/s) does not increase or Shard Consuming Latency (s) still increases, the job fails to scale out resources due to shard shortage in the source Logstore. In this case, you must manually split shards in the source Logstore. For more information, see Split a shard. After you split the shards, wait five minutes and then check whether the latency drops below the threshold. If the latency does not, proceed to the next step.

3. If data transformation exceptions exist, troubleshoot the exceptions first. After troubleshooting, wait five minutes and then check whether the latency drops below the threshold. If the latency does not, proceed to the next step.

4. If you are unable to clear the alert, submit a ticket with project, Logstore, and job information for technical support.

Item

Description

Purpose

A rule of this type monitors exceptions in data transformation jobs. If an exception occurs during data transformation, an alert is triggered.

Associated dashboard

For more information, see Transformation errors.

Sample SQL

In the following template, replace {job_name} with the name of the data transformation job that you want to monitor.

__topic__: etl_metrics and 
job_name: {job_name} and 
"_etl_:connector_metrics.error": *
| select
  distinct "_etl_:connector_metrics.error" as errors

Monitoring rule

• For Trigger Condition, set When to data is returned.

• Set the query interval to 10 minutes.

• Set the query frequency to 10 minutes.

Handling method

Fix exceptions based on the error message.

• If the error message contains Invalid SPL query, the Processing Language (SPL) rule configured for the job has a syntax error and needs to be modified based on the error message. For more information, see SPL syntax.

• If the error message contains Unauthorized, InvalidAccessKeyId, or SignatureNotMatch, the data transformation job does not have the required permissions to read data from the source Logstore or write data to the destination Logstore. For more information, see Authorization overview.

• If the error message contains ProjectNotExist or LogStoreNotExist, the related project or Logstore of the data transformation job does not exist. In this case, log on to the Simple Log Service console to troubleshoot the issue.

• If you are unable to clear the alert, submit a ticket with project, Logstore, and job information for technical support.

Item

Description

Purpose

A rule of this type monitors the comparisons between the ratios of data records written to the destination Logstore with those of the same period the previous day or week. An alert is triggered if the value goes beyond the specified threshold for increase or drops below the specified threshold for decrease.

Associated dashboard

Event Delivered Ratio: the ratio of the number of data records that were successfully delivered to the destination Logstore to the number of data records that were read from the source Logstore. The default statistical window is one day.

Sample SQL

In the dialog box for creating the alert rule, enter the following SQL statement for queries:

In the following template, replace {job_name} with the name of the data transformation job that you want to monitor.

__topic__: etl_metrics and 
job_name: {job_name}
| select
  round(diff [1], 1) as percent,
  round(coalesce(diff [3], 0), 1) as ratio_1d,
  round(coalesce(diff [5], 0), 1) as ratio_1w
from(
    select
      compare(percent, 86400, 604800) as diff
    FROM      (
        select
          deliver /(ingest + 0.0001) as percent
        from(
            select
              sum(
                if(
                  "_etl_:connector_meta.action" = 'ingest',
                  "_etl_:connector_metrics.native_bytes",
                  0
                )
              ) as ingest,
              sum(
                if(
                  "_etl_:connector_meta.action" = 'deliver',
                  "_etl_:connector_metrics.native_bytes",
                  0
                )
              ) as deliver
            FROM              log
          )
      )
  )

Monitoring rule

• For Trigger Condition, set When to the query result contains and set the expression to (ratio_1d>120 || ratio_1d<80) && (ratio_1w>120 || ratio_1w<80), which means that the threshold is 20% for both increases and decreases.

• Set the query interval to 1 hour.

• Set the query frequency to 1 hour.

Handling method

You can clear triggered alerts based on the following rules:

1. If the data volume in the source Logstore changes, check whether a new data pattern is added or whether an added data pattern is interrupted. If such circumstances exist and the volume changes caused thereby match the metric changes, the metric changes are caused by the data pattern alterations. If such circumstances do not exist, proceed to the next step.

2. If transformation latencies or exceptions exist, troubleshoot these issues first. After you troubleshoot these issues, wait 15 minutes and check whether the latency is less than 1 minute. If so, check whether the changes in processed data volume match the data volume changes in the source Logstore. If they do not match, proceed to the next step.

3. If you are unable to clear the alert, submit a ticket with project, Logstore, and job information for technical support.

Item

Description

Purpose

A rule of this type monitors the comparisons between the ratios of data records read from the source Logstore with those of the same period the previous day or week. An alert is triggered if the value goes beyond the specified threshold for increase or drops below the specified threshold for decrease.

Associated dashboard

Total Events Read: the total number of data records that were read from each shard of the source Logstore. The default statistical window is one day.

Sample SQL

In the dialog box for creating the alert rule, enter the following SQL statement for queries:

In the following template, replace {job_name} with the name of the data transformation job that you want to monitor.

__topic__: etl_metrics and 
job_name: {job_name} and
"_etl_:connector_meta.action": ingest
| select
  diff [1] as events,
  round(coalesce(diff [3], 0),  1) as ratio_1d,
  round(coalesce(diff [5], 0),  1) as ratio_1w
from(
    select
      compare(events, 86400, 604800) as diff
    FROM      (
        select
          sum("_etl_:connector_metrics.events") as events
        FROM          log
      )
  )

Monitoring rule

• For Trigger Condition, set When to the query result contains and set the expression to (ratio_1d>120 || ratio_1d<80) && (ratio_1w>120 || ratio_1w<80), which means that the threshold is 20% for both increases and decreases.

• Set the query interval to 1 hour.

• Set the query frequency to 1 hour.

Handling method

You can clear triggered alerts based on the following rules:

1. If the value changes are consistent with the changes in the data volume in the source Logstore, the value changes are caused by the data volume changes in the source Logstore. If they are not, proceed to the next step.

2. If transformation latencies or exceptions exist, troubleshoot these issues first. After you troubleshoot these issues, wait 15 minutes and check whether the latency is less than 1 minute. If so, check whether the changes in processed data volume match the data volume changes in the source Logstore. If they do not match, proceed to the next step.

3. If you are unable to clear the alert, submit a ticket with project, Logstore, and job information for technical support.

Item

Description

Purpose

A rule of this type monitors the comparisons between the number of data records written to the destination Logstore with those of the same period the previous day or week. An alert is triggered if the value goes beyond the specified threshold for increase or drops below the specified threshold for decrease.

Associated dashboard

Dashboard > Overall metrics > Total Events Delivered

Sample SQL

In the dialog box for creating the alert rule, enter the following SQL statement for queries:

In the following template, replace {job_name} with the name of the data transformation job that you want to monitor.

__topic__: etl_metrics and 
job_name: {job_name} and
"_etl_:connector_meta.action": deliver
| select
  diff [1] as events,
  round(coalesce(diff [3], 0),  1) as ratio_1d,
  round(coalesce(diff [5], 0),  1) as ratio_1w
from(
    select
      compare(events, 86400, 604800) as diff
    FROM      (
        select
          sum("_etl_:connector_metrics.events") as events
        FROM          log
      )
  )

Monitoring rule

• For Trigger Condition, set When to the query result contains and set the expression to (ratio_1d>120 || ratio_1d<80) && (ratio_1w>120 || ratio_1w<80), which means that the threshold is 20% for both increases and decreases.

• Set the query interval to 1 hour.

• Set the query frequency to 1 hour.

Handling method

You can clear triggered alerts based on the following rules:

1. If the value changes are consistent with the changes in the data volume in the source Logstore, the value changes are caused by the data volume changes in the source Logstore. If they are not, proceed to the next step.

2. If transformation latencies or exceptions exist, troubleshoot these issues first. After you troubleshoot these issues, wait 15 minutes and check whether the latency is less than 1 minute. If so, check whether the changes in processed data volume match the data volume changes in the source Logstore. If they do not match, proceed to the next step.

3. If you are unable to clear the alert, submit a ticket with project, Logstore, and job information for technical support.