All Products
Search
Document Center

Simple Log Service:Write data to MaxCompute by using a custom RAM role

Last Updated:Nov 28, 2024

When a data shipping job of the new version is running to ship data to MaxCompute, the data that is read from a Logstore must be written to a MaxCompute table. To meet this requirement, you can authorize the data shipping job to assume a custom Resource Access Management (RAM) role. This topic describes how to authorize a MaxCompute data shipping job of the new version to ship data by using a custom RAM role.

Prerequisites

  • A RAM role is created. In this example, the RAM role is named MaxComputeShipRole. For more information, see Create a RAM role for a trusted Alibaba Cloud service.

    Note that the RAM role must meet the following requirements:

    • When you create the RAM role, select Alibaba Cloud Service as the trusted entity.

    • In the trust policy of the RAM role, the "Service" field must include "log.aliyuncs.com", as illustrated below.

      {
        "Statement": [
          {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Principal": {
              "Service": [
                "log.aliyuncs.com"
              ]
            }
          }
        ],
        "Version": "1"
      }

      image

  • If you use a RAM user, make sure that the RAM user has the permissions to manage RAM roles.

Ship data within an Alibaba Cloud account

After you grant a RAM role the permissions to write data to MaxCompute, a MaxCompute data shipping job can assume the RAM role to write data of a Logstore to a MaxCompute table. To complete the authorization, you must add the RAM role as a workspace member.

You can use the GUI or the CLI to grant permissions to the RAM role.

Use the GUI to grant permissions to the RAM role

  1. Modify the trust policy of the RAM role.

    1. Log on to the RAM console.

    2. In the left-side navigation pane, choose Identities > Roles.

    3. On the Roles page, click the RAM role that you want to manage.

    4. On the Trust Policy tab, click Edit Trust Policy.

    5. Replace the content in the code editor with the following policy document and click Save trust policy document:

      {
        "Statement": [
          {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Principal": {
              "Service": [
                "log.aliyuncs.com",
                "dataworks.aliyuncs.com"
              ]
            }
          }
        ],
        "Version": "1"
      }
  2. Add the RAM role as a workspace member.

    1. Log on to the DataWorks console.

    2. In the upper-left corner of the page that appears, select a region.

    3. In the left-side navigation pane, click Workspaces. image

    4. On the Workspaces page, find the workspace that you want to manage and click Manage in the Actions column.

    5. On the Workspace Members tab of the Workspace page, click Add Members. image.png

    6. In the Add Members dialog box, select and add the current logon account and the RAM role.

      In the Batch Assign Roles section, select Development. For more information, see Grant permissions to a RAM user.

  3. Grant the RAM role the permissions to manage a MaxCompute table.

    1. On the Workspace page, choose Compute Engine Information > MaxCompute > Go to the MaxCompute Console. image.png

    2. On the MaxCompute project management page, click Role Permissions. image.pngIf the following error is reported, perform the following operations in the role list: Find the admin role and click Manage Members in the Actions column. In the Manage Members dialog box, select and add the current logon account. image.png

    3. In the role list, find the role_project_admin role and click Manage Members in the Actions column.

    4. In the Manage Members dialog box, select and add the current logon account and the RAM role.

    5. In the role list, find the role_project_admin role and click Manage Members in the Actions column.

    6. On the Table tab of the Edit Role dialog box, find the MaxCompute table that you want to manage and select Describe, Alter, and Update. image.png

Important

The preceding authorization takes effect only on the specified MaxCompute table. If you want to grant the RAM role the permissions to manage all tables in the current MaxCompute project, you can grant the permissions of the admin role to the current logon account and the target role. In the role list, find the admin role and click Manage Members in the Actions column. In the Manage Members dialog box, select and add the current logon account and the target role.

After you configure the settings, you can assign the RAM role to a MaxCompute data shipping job of the new version to ship data to the specified MaxCompute table. When you create the data shipping job, select Custom Role for Write Permissions on MaxCompute and specify the Alibaba Cloud Resource Name (ARN) of the RAM role. Example: acs:ram::10**12:role/maxcomputeshiprole. For more information about how to obtain the ARN of a RAM role, see View the information about a RAM role. MaxCompute授权

Use the CLI to grant permissions to the RAM role

  1. Log on to the DataWorks console.

  2. In the upper-left corner of the page that appears, select a region.

  3. In the left-side navigation pane, click Workspaces. image

  4. On the Workspaces page, find the workspace that you want to manage, move the pointer over Shortcuts, and then click Data Development.

  5. Create a workflow.

    1. On the Scheduled Workflow page, choose Create > Create Workflow.

    2. In the Create Workflow dialog box, configure Workflow Name and click Create.

  6. Create a node.

    1. On the Scheduled Workflow page, choose Create > Create Node > MaxCompute > ODPS SQL.

    2. In the Create Node dialog box, configure Name and Path, and click Confirm.

      You must set Path to the workflow that you created in Step 5.

  7. In the code editor of the node, run the required commands to complete the authorization. The following table describes the commands.

    MaxCompute授权

    Command

    Description

    USE project-name;

    Specifies a MaxCompute project. The MaxCompute project must be the same as the MaxCompute project that you specified when you created your data shipping job of the new version. For more information, see Create a data shipping job of the new version to ship data to MaxCompute.

    ADD USER RAM$****.aliyunid.com:`role/maxcomputeshiprole`;

    Adds the RAM role as a user to the MaxCompute project.

    • ****.aliyunid.com specifies the Alibaba Cloud account to which the MaxCompute project belongs. You can run the list users; command to view the Alibaba Cloud account.

    • maxcomputeshiprole specifies the name of the custom RAM role. The name must be in lowercase letters.

    GRANT CreateInstance ON PROJECT project-name TO USER RAM$*****:`role/maxcomputeshiprole`;

    Grants the RAM role the permission to create instances in the specified MaxCompute project.

    GRANT DESCRIBE, ALTER, UPDATE ON TABLE table-name to user RAM$****.aliyunid.com:`role/maxcomputeshiprole`;

    Grants the RAM role the permissions to view, modify, and update the specified MaxCompute table.

    Note

    The preceding authorization takes effect only on the specified MaxCompute table. If you want to grant the RAM role the permissions to manage all tables in the current MaxCompute project, run the GRANT admin to user RAM$****.aliyunid.com:`role/maxcomputeshiprole`; command to complete the authorization.

    SHOW GRANTS FOR `RAM$****.aliyunid.com:role/maxcomputeshiprole`;

    Checks whether the authorization is successful.

    If information similar to the following code is returned, the authorization is successful:

    Authorization Type: ACL
    [user/RAM$****.aliyunid.com:role/maxcomputeshiprole]
    A       projects/default_project_****: CreateInstance
    A       projects/default_project_****/tables/****: Describe | Alter | Update

    The following error messages may appear during the authorization process:

After you configure the settings, you can assign the RAM role to a MaxCompute data shipping job of the new version to ship data to the specified MaxCompute table. When you create the data shipping job, select Custom Role for Write Permissions on MaxCompute and specify the ARN of the RAM role. Example: acs:ram::10**12:role/maxcomputeshiprole. For more information about how to obtain the ARN of a RAM role, see View the information about a RAM role. MaxCompute授权

Ship data across Alibaba Cloud accounts

If Simple Log Service is activated for Alibaba Cloud Account A and MaxCompute is activated for Alibaba Cloud Account B, you must grant RAM Role B that belongs to Alibaba Cloud Account B the permissions to write data to MaxCompute. After the authorization is complete, a MaxCompute data shipping job of the new version can assume the RAM role to write data of a Logstore to a MaxCompute table.

  1. Modify the trust policy of RAM Role B that belongs to Alibaba Cloud Account B.

    1. Log on to the RAM console by using Alibaba Cloud Account B.

    2. In the left-side navigation pane, choose Identities > Roles.

    3. In the role list, click RAM Role B.

    4. On the Trust Policy tab, click Edit Trust Policy.

    5. Modify the trust policy and click Save trust policy document.

      Add ID of Alibaba Cloud Account A@log.aliyuncs.com and dataworks.aliyuncs.com to the Service element. Replace ID of Alibaba Cloud Account A with the actual ID. You can view the ID of an Alibaba Cloud account in the Account Center console.

      The following policy allows Alibaba Cloud Account A to obtain a temporary Security Token Service (STS) token to manage the cloud resources of Alibaba Cloud Account B:

      {
          "Statement": [
              {
                  "Action": "sts:AssumeRole",
                  "Effect": "Allow",
                  "Principal": {
                      "Service": [
                          "log.aliyuncs.com",
                          "dataworks.aliyuncs.com",
                          "ID of Alibaba Cloud Account A@log.aliyuncs.com"
                      ]
                  }
              }
          ],
          "Version": "1"
      }
  2. Add RAM Role B that belongs to Alibaba Cloud Account B as a workspace member.

    You can use the GUI or the CLI to grant permissions to the RAM role. To complete the authorization, log on to the MaxCompute console by using Alibaba Cloud Account B. For more information, see Use the GUI to grant permissions to the RAM role or Use the CLI to grant permissions to the RAM role.

After you configure the settings, you can assign the RAM role of Alibaba Cloud Account B to a MaxCompute data shipping job of the new version to ship data to the specified MaxCompute table. When you create the data shipping job, select Custom Role for Write Permissions on MaxCompute and enter the ARN of the RAM role that belongs to Alibaba Cloud Account B. Example: acs:ram::11**13:role/maxcomputeshiprole. For more information about how to obtain the ARN of a RAM role, see View the information about a RAM role. MaxCompute授权