Simple Log Service:Advanced preview

Procedure

1. Log on to the Simple Log Service console.

2. Go to the data transformation page.

   1. In the Projects section, click the project that you want to manage.

   2. On the Log Storage > Logstores tab, click the Logstore that you want to manage.

   3. On the query and analysis page, click Data Transformation.

3. In the upper-right corner of the page that appears, specify the time range of the data that you want to manage.

   After you specify the time range, verify that logs appear on the Raw Logs tab.

4. In the edit box, enter a data transformation statement.

   For more information, see Data processing syntax.

   Note

   You can add comments to the data transformation statement to debug the statement line by line.

5. Preview data.

   1. In the upper-right corner of the page, click Advanced.

   2. Click Preview Data.

   3. In the Add Preview Settings panel, set the parameters and click OK. The following table describes the parameters.

      When you preview data for the first time, you must set the parameters. After you set the parameters, you can click Modify Preview Settings to modify the parameters.

      Parameter	Description
      Authorization Method	You can authorize a data transformation task to read data from the current source Logstore by using one of the following methods: Default Role: The data transformation job assumes the AliyunLogETLRole system role to read data from the source Logstore. Custom Role: The data transformation job assumes a custom role to read data from the source Logstore. You must grant the custom role the permissions to read from the source Logstore. Then, you must enter the Alibaba Cloud Resource Name (ARN) of the custom role in the Role ARN field. For more information, see Access data by using a custom role. AccessKey Pair: The data transformation job uses the AccessKey pair of an Alibaba Cloud account or a RAM user to read data from the source Logstore. Alibaba Cloud account: The AccessKey pair of an Alibaba Cloud account has permissions to read from the source Logstore. You can directly enter the AccessKey ID and AccessKey secret of the Alibaba Cloud account in the AccessKey ID and AccessKey Secret fields. For more information about how to obtain an AccessKey pair, see AccessKey pair. RAM user: You must grant the RAM user the permissions to read from the source Logstore. Then, you can enter the AccessKey ID and AccessKey secret of the RAM user in the AccessKey ID and AccessKey Secret fields. For more information, see Access data by using AccessKey pairs.
      Advanced Parameter Settings	Simple Log Service allows you to set the passwords that are required in the transformation statement in the key-value pair format. For example, you can set a password that is used to connect to a database in the key-value pair format. You can reference passwords in the transformation statement by using the ${key} variable. You can click the + icon to add more key-value pairs. For example, you can add config.vpc.vpc_id.test1:vpc-uf6mskb0b****n9yj to indicate the ID of the virtual private cloud (VPC) to which an ApsaraDB RDS instance belongs.

      After you configure the preview settings, you can preview the data transformation results on the Transformation Results tab.

      • If the data fails to be transformed because the syntax of the transformation statement or the permissions are invalid, troubleshoot the failure as prompted.

      • If the data is transformed as expected, you can save the transformation statement as a rule. For more information, see Create a data transformation job.

Advanced preview example

• Transformation statement

  # e_set("insert_field", "test_value")
  e_table_map(
      res_rds_mysql(
          address="rm-uf6wjk5****.mysql.rds.aliyuncs.com",
          username="test_username",
          password="****",
          database="test_db",
          table="test_table",
      ),
      "ip",
      ["country", "province"],
  )

• Raw log entry

  {
    "__source__": "192.0.2.0",
    "__time__": 1624523917,
    "__topic__": "topic",
    "id": "1001",
    "ip": "127.0.0.1"
  }

• Advanced preview settings

• Transformation result