How to access data by using AccessKey pairs - Simple Log Service

For a data transformation job, you can specify the AccessKey pairs of Alibaba Cloud accounts or Resource Access Management (RAM) users to authorize Simple Log Service to read data from a source Logstore and write transformed data to one or more destination Logstores. The AccessKey pair of an Alibaba Cloud account has access permissions on Logstores and can be directly used. If you use a RAM user, you must grant the RAM user the access permissions on Logstores. For more information, see the following sections.

Prerequisites

RAM users are created. For more information, see Create a RAM user.

Important

When you create the RAM users, select OpenAPI Access for Access Mode. Then, record the AccessKey pairs of the RAM users.
The AccessKey secret of a RAM user is displayed only when you create the AccessKey pair for the RAM user. You cannot query the AccessKey secret after you create the AccessKey pair. Store your AccessKey secret in a secure location.

Grant a RAM user the permissions to read data from a source Logstore

After you use an Alibaba Cloud account to grant the read permissions to the RAM user, the RAM user can read data from the source Logstore. When you create a data transformation job, you can enter the AccessKey pair of the RAM user. For more information, see Create a data transformation job.

Log on to the RAM console by using an Alibaba Cloud account.

Create a policy.

The policy allows the RAM user to read data from a source Logstore.

In the left-side navigation pane, choose Permissions > Policies.
On the Policies page, click Create Policy.

On the Create Policy page, click the JSON tab, replace the existing script in the code editor with the one of the following policy documents, and then click Next to edit policy information.

Policy that uses exact match

In this example, the source project name is log-project-prod, and the source Logstore name is access_log. Replace the project and Logstore names based on your business requirements.

{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:ListShards",
        "log:GetCursorOrData",
        "log:GetConsumerGroupCheckPoint",
        "log:UpdateConsumerGroup",
        "log:ConsumerGroupHeartBeat",
        "log:ConsumerGroupUpdateCheckPoint",
        "log:ListConsumerGroup",
        "log:CreateConsumerGroup"
      ],
      "Resource": [
        "acs:log:*:*:project/log-project-prod/logstore/access_log",
        "acs:log:*:*:project/log-project-prod/logstore/access_log/*"
      ],
      "Effect": "Allow"
    }
  ]
}

Policy that uses fuzzy match

In this example, the source project name can be log-project-dev-a, log-project-dev-b, or log-project-dev-c, and the source Logstore name can be app_a_log, app_b_log, or app_c_log. Replace the project and Logstore names based on your business requirements.

{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:ListShards",
        "log:GetCursorOrData",
        "log:GetConsumerGroupCheckPoint",
        "log:UpdateConsumerGroup",
        "log:ConsumerGroupHeartBeat",
        "log:ConsumerGroupUpdateCheckPoint",
        "log:ListConsumerGroup",
        "log:CreateConsumerGroup"
      ],
      "Resource": [
        "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log",
    "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log/*"
      ],
      "Effect": "Allow"
    }
  ]
}

For more information about authorization scenarios, see Use custom policies to grant permissions to a RAM user.

Configure the Name parameter for the policy. In this example, enter log-etl-source-reader-1-policy. Then, click OK.

Attach the policy to the RAM user.
1. In the left-side navigation pane, choose Identities > Users.
2. On the Users page, find the RAM user and click Add Permissions in the Actions column.
3. In the Grant Permission panel, go to the Policy section and select Custom Policy from the filter drop-down list. Then, click the policy that you created in Step 2 and click Grant permissions. In this example, click the log-etl-source-reader-1-policy policy.
4. Confirm the authorization results. Then, click Close.

Grant a RAM user the permissions to write data to destination Logstores

After you use an Alibaba Cloud account to grant the write permissions to the RAM user, the RAM user can write data to the destination Logstores. When you create a data transformation job, you can enter the AccessKey pair of the RAM user. For more information, see Create a data transformation job.

Log on to the RAM console by using an Alibaba Cloud account.
Create a policy.
The policy allows the RAM user to write data to the destination Logstores.
1. In the left-side navigation pane, choose Permissions > Policies.
2. On the Policies page, click Create Policy.
3. On the Create Policy page, click the JSON tab, replace the existing script in the code editor with the one of the following policy documents, and then click Next to edit policy information.
  - Policy that uses exact match
    In this example, the destination project name is log-project-prod, and the destination Logstore name is access_log_output. Replace the project and Logstore names based on your business requirements.
    { "Version": "1", "Statement": [ { "Action": [ "log:Post*", "log:BatchPost*" ], "Resource": "acs:log:*:*:project/log-project-prod/logstore/access_log_output", "Effect": "Allow" } ] }
  - Policy that uses fuzzy match
    In this example, the destination project name can be log-project-dev-a, log-project-dev-b, or log-project-dev-c, and the destination Logstore name can be app_a_log_output, app_b_log_output, or app_c_log_output. Replace the project and Logstore names based on your business requirements.
    { "Version": "1", "Statement": [ { "Action": [ "log:Post*", "log:BatchPost*" ], "Resource": "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log_output", "Effect": "Allow" } ] }
    For more information about authorization scenarios, see Use custom policies to grant permissions to a RAM user.
4. Configure the Name parameter for the policy. In this example, enter log-etl-target-writer-1-policy. Then, click OK.
Attach the policy to the RAM user.
1. In the left-side navigation pane, choose Identities > Users.
2. On the Users page, find the RAM user and click Add Permissions in the Actions column.
3. In the Grant Permission panel, go to the Policy section and select Custom Policy from the filter drop-down list. Then, click the policy that you created in Step 2 and click Grant permissions. In this example, click the log-etl-target-writer-1-policy policy.
4. Confirm the authorization results. Then, click Close.

What to do next

You can specify the AccessKey pairs of the RAM users for a data transformation job. For more information, see Create a data transformation job.

In Section 1, enter the AccessKey pair of the RAM user that is granted the permissions to read data from a source Logstore. For more information, see Grant a RAM user the permissions to read data from a source Logstore.
In Section 2, enter the AccessKey pair of the RAM user that is granted the permissions to write data to destination Logstores. For more information, see Grant a RAM user the permissions to write data to destination Logstores.

修改加工规则