Simple Log Service:What do I do if the destination Logstores contain no data?

Scenario 1: The storage destinations that are specified in the transformation statement are inconsistent with the storage destinations that are specified in the Create Data Transformation Job panel

The source Logstore is website_log. It contains 1,000 logs whose SourceIP is 192.0.2.54, 1,000 logs whose SourceIP is 192.0.2.28, 1,000 logs whose SourceIP is 192.0.2.136, and 2,000 logs whose SourceIP is different from these IP addresses. The logs are transformed. Then, the logs that meet specific conditions are distributed to the following destination Logstores: 54_log_target, 28_log_target, and 136_log_target.

• Transformation requirements

  • Distribute logs whose SourceIP is 192.0.2.54 to the 54_log_target Logstore.

  • Distribute logs whose SourceIP is 192.0.2.28 to the 28_log_target Logstore.

  • Distribute logs whose SourceIP is 192.0.2.136 to the 136_log_target Logstore.

  • Discard all other logs.

• Transformation statement

  The three destination Logstores are 54_log, 28_log, and 136_log.

  e_if(e_search("SourceIP==192.0.2.54"),    
    e_output(name="54-target",
               project="sls-test",
               logstore="54_log"))
  e_if(e_search("SourceIP==192.0.2.28"),
      e_output(name="28-target",
               project="sls-test",
               logstore="28_log"))
  e_if(e_search("SourceIP==192.0.2.136"),
      e_output(name="136-target",
               project="sls-test",
               logstore="136_log"))
  e_drop()

• Storage destinations

  The three destination Logstores are 54_log_target, 28_log_target, and 136_log_target.

  

• Transformation results

  The 54_log_target, 28_log_target, and 136_log_target destination Logstores contain no data.

• Cause

  In the e_output functions of the transformation statement, the configurations of the project and logstore parameters are different from those of the Destination Project and Target Store parameters in the Create Data Transformation Job panel. In this case, the settings of the transformation statement take precedence.

  As a result, the logs are distributed to the 54_log, 28_log, and 136_log destination Logstores that are specified in the transformation statement.

• Solution

  • If you use the e_output or e_coutput function and configure only the name parameter, the transformation results are distributed to the storage destinations that are specified in the Create Data Transformation Job panel.

  • If you use the e_output or e_coutput function and configure the project and logstore parameters, make sure that the storage destination settings that are specified in the transformation statement are consistent with the storage destination settings that are specified in the Create Data Transformation Job panel. For more information, see e_output and e_coutput.

    Note

    If no e_drop() function is included in the transformation statement and multiple storage destinations are specified in the transformation statement, the system distributes all logs that do not meet the conditions in the statement and are not discarded to the destination Logstore in the storage destination that is numbered 1.

    e_if(e_search("SourceIP==192.0.2.54"),    
      e_output(name="54-target",
                 project="sls-test",
                 logstore="54_log_target"))
    e_if(e_search("SourceIP==192.0.2.28"),
        e_output(name="28-target",
                 project="sls-test",
                 logstore="28_log_target"))
    e_if(e_search("SourceIP==192.0.2.136"),
        e_output(name="136-target",
                 project="sls-test",
                 logstore="136_log_target"))
    e_drop()

Scenario 2: No transformation statements are specified or the specified transformation statement contains only the e_drop() function

• Transformation statement

  None

• Storage destinations

• Transformation results

  Only the 54_log_target Logstore contains data.

• Cause

  • No transformation statements are specified on the data transformation page. However, the following operations are performed in the Create Data Transformation Job panel:

    • Specify a storage destination. The system copies data in the source Logstore to the destination Logstore in the storage destination.

    • Specify multiple storage destinations. The system distributes all logs to the destination Logstore in the storage destination that is numbered 1.

  • A transformation statement that contains only the e_drop() function is specified on the data transformation page. The system discards all logs. The destination Logstore contains no data.

Scenario 3: Latency exists during data transformation

If you confirm that the settings of your data transformation job are valid but the destination Logstores contain no data, latency may exist during data transformation.