All Products
Search
Document Center

Simple Log Service:Write data to MaxCompute by using a default role

Last Updated:May 24, 2024

When a data shipping job of the new version is running to ship data to MaxCompute, the data that is read from a Logstore must be written to a MaxCompute table. To meet the requirement, you can authorize the data shipping job to assume a default role.

Prerequisites

If you are using a Resource Access Management (RAM) user, make sure that the RAM user has the permissions to manage RAM roles.

Ship data within an Alibaba Cloud account

If Simple Log Service and MaxCompute belong to the same Alibaba Cloud account, you must complete authorization on the Cloud Resource Access Authorization page and create the AliyunLogDefaultRole role. After the AliyunLogDefaultRole role is created, you must add the AliyunLogDefaultRole role as a workspace member.

You can use the GUI or the CLI to grant permissions to the RAM role.

Use the GUI to grant permissions to the RAM role

  1. Modify the trust policy of the RAM role.

    1. Log on to the RAM console.

    2. In the left-side navigation pane, choose Identities > Roles.

    3. On the Roles page, find the AliyunLogDefaultRole role and click the role name.

    4. On the Trust Policy tab, click Edit Trust Policy.

    5. Replace the content in the code editor with the following policy document and click Save trust policy document.

      {
        "Statement": [
          {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Principal": {
              "Service": [
                "log.aliyuncs.com",
                "dataworks.aliyuncs.com"
              ]
            }
          }
        ],
        "Version": "1"
      }
  2. Add the RAM role as a workspace member.

    1. Log on to the DataWorks console.

    2. In the upper-left corner of the page that appears, select a region.

    3. In the left-side navigation pane, click Workspaces.

    4. On the Workspaces page, find the workspace that you want to manage and click Manage in the Actions column.

    5. On the Space members tab of the Work space page, click Add Members. image.png

    6. In the Add Members dialog box, select the current logon account and the AliyunLogDefaultRole role, and then add the logon account and the AliyunLogDefaultRole role as prompted.

      In the Batch Setting Roles section, select Development. For more information, see Grant permissions to a RAM user.

  3. Grant the AliyunLogDefaultRole role the permissions to manage a MaxCompute table.

    1. On the Work space page, choose Compute engine information > MaxCompute > Go to the MaxCompute Console. image.png

    2. On the MaxCompute project management page, click Role Permissions. image.png

      If the following error is reported, perform the following operations in the role list: Find the admin role and click Manage Members in the Actions column. In the Manage Members dialog box, select the current logon account and add the logon account as prompted. image.png

    3. In the role list, find the role_project_admin role and click Manage Members in the Actions column.

    4. In the Manage Members dialog box, select the current logon account and the AliyunLogDefaultRole role, and then add the logon account and the AliyunLogDefaultRole role as prompted.

    5. In the role list, find the role_project_admin role and click Edit Role in the Actions column.

    6. On the Table tab of the Edit Role dialog box, find the MaxCompute table that you want to manage and select Describe, Alter, and Update. image.png

    Important

    The preceding authorization takes effect only on the specified MaxCompute table. If you want to grant the AliyunLogDefaultRole role the permissions to manage all tables in the current MaxCompute project, you can grant the permissions of the admin role to the current logon account and the AliyunLogDefaultRole role. In the role list, find the admin role and click Manage Members in the Actions column. In the Manage Members dialog box, select the current logon account and the AliyunLogDefaultRole role, and then add the logon account and the AliyunLogDefaultRole role as prompted.

    After you configure the settings, you can assign the AliyunLogDefaultRole role to a MaxCompute data shipping job of the new version to ship data to the specified MaxCompute table. When you create the data shipping job, select Default Role for Write Permissions on MaxCompute. For more information, see Create a data shipping job of the new version to ship data to MaxCompute. MaxCompute投递授权

Use the CLI to grant permissions to the RAM role

  1. Log on to the DataWorks console.

  2. In the upper-left corner of the page that appears, select a region.

  3. In the left-side navigation pane, click Workspaces.

  4. On the Workspaces page, find the workspace that you want to manage, move the pointer over Shortcuts, and then click Data Development.

  5. Create a workflow.

    1. On the Scheduled Workflow page, choose Create > Create Workflow.

    2. In the Create Workflow dialog box, configure the Workflow Name parameter and click Create.

  6. Create a node.

    1. On the Scheduled Workflow page, choose Create > Create Node > MaxCompute > ODPS SQL.

    2. In the Create Node dialog box, configure the Name and Path parameters, and click Confirm.

      You must set the Path parameter to the workflow that you created in Step 5.

  7. In the code editor of the node, run the required commands to complete the authorization. The following table describes the commands.

    MaxCompute授权

    Command

    Description

    USE project-name;

    Specifies a MaxCompute project. The MaxCompute project must be the same as the MaxCompute project that you specified when you created your data shipping job of the new version. For more information, see Create a data shipping job of the new version to ship data to MaxCompute.

    ADD USER RAM$****.aliyunid.com:`role/aliyunlogdefaultrole`;

    Adds the RAM role as a user to the MaxCompute project.

    • ****.aliyunid.com specifies the Alibaba Cloud account to which the MaxCompute project belongs. You can run the list users; command to view the Alibaba Cloud account.

    • aliyunlogdefaultrole specifies the name of the AliyunLogDefaultRole role. The name must be in lowercase letters.

    GRANT CreateInstance ON PROJECT project-name TO USER RAM$*****:`role/aliyunlogdefaultrole`;

    Grants the RAM role the permission to create instances in the specified project.

    GRANT DESCRIBE, ALTER, UPDATE ON TABLE table-name TO user RAM$****.aliyunid.com:`role/aliyunlogdefaultrole`;

    Grants the RAM role the permissions to view, modify, and update the specified MaxCompute table.

    Note

    The authorization procedure applies only to the specified MaxCompute table. If you want to grant the RAM role the permissions to manage all tables in the current MaxCompute project, run the GRANT admin to user RAM$****.aliyunid.com:`role/aliyunlogdefaultrole`; command to complete the authorization.

    SHOW GRANTS FOR `RAM$****.aliyunid.com:role/aliyunlogdefaultrole`;

    Checks whether the authorization is successful.

    If information similar to the following code is returned, the authorization is successful:

    Authorization Type: ACL
    [user/RAM$****.aliyunid.com:role/aliyunlogdefaultrole]
    A       projects/default_project_****: CreateInstance
    A       projects/default_project_****/tables/****: Describe | Alter | Update

    The following error messages may appear during the authorization process:

After you configure the settings, you can assign the AliyunLogDefaultRole role to a MaxCompute data shipping job of the new version to ship data to the specified MaxCompute table. When you create the data shipping job, select Default Role for Write Permissions on MaxCompute. For more information, see Create a data shipping job of the new version to ship data to MaxCompute. MaxCompute投递授权

Ship data across Alibaba Cloud accounts

If Simple Log Service is activated for Alibaba Cloud Account A and MaxCompute is activated for Alibaba Cloud Account B, you must complete authorization on the Cloud Resource Access Authorization page and create the AliyunLogDefaultRole role in each account. Then, perform the following operations:

  1. Modify the trust policy of the AliyunLogDefaultRole role that belongs to Alibaba Cloud Account B.

    1. Log on to the RAM console by using Alibaba Cloud Account B.

    2. In the left-side navigation pane, choose Identities > Roles.

    3. In the role list, click AliyunLogDefaultRole.

    4. On the Trust Policy tab, click Edit Trust Policy.

    5. Replace the content in the code editor with the following policy document and click Save trust policy document.

      Add ID of Alibaba Cloud Account A@log.aliyuncs.com and dataworks.aliyuncs.com to the Service element. Replace ID of Alibaba Cloud Account A with the actual ID. You can view the ID of Alibaba Cloud account A in the Account Management console.

      The following policy allows Alibaba Cloud Account A to obtain a temporary Security Token Service (STS) token to manage the cloud resources of Alibaba Cloud Account B:

      {
        "Statement": [
         {
           "Action": "sts:AssumeRole",
           "Effect": "Allow",
           "Principal": {
             "Service": [
               "ID of Alibaba Cloud Account A@log.aliyuncs.com",
               "dataworks.aliyuncs.com",
               "log.aliyuncs.com"
             ]
           }
         }
        ],
        "Version": "1"
      }
  2. Add the AliyunLogDefaultRole role that belongs to Alibaba Cloud Account B as a workspace member.

    You can use the GUI or the CLI to grant permissions to the RAM role. To complete the authorization, log on to the DataWorks console by using Alibaba Cloud Account B. For more information, see Use the GUI to grant permissions to the RAM role or Use the CLI to grant permissions to the RAM role.

After you configure the settings, you can assign the AliyunLogDefaultRole role of Alibaba Cloud Account B to a MaxCompute data shipping job of the new version to ship data to the specified MaxCompute table. When you create the data shipping job, select Custom Role for Write Permissions on MaxCompute and enter the Alibaba Cloud Resource Name (ARN) of the AliyunLogDefaultRole role that belongs to Alibaba Cloud Account B. In this example, acs:ram::11**13:role/aliyunlogdefaultrole is used. For more information about how to obtain the ARN of a RAM role, see View the information about a RAM role. MaxCompute授权