E-MapReduce:ECS application role (used in EMR V3.32.0 or an earlier minor version, or EMR V4.5.0 or an earlier minor version)

Prerequisites

This role is authorized. For more information, see Assign roles to an Alibaba Cloud account.

Background information

MetaService allows you to access only Object Storage Service (OSS), Log Service, and Message Service (MNS) without an AccessKey pair.

Permissions

Data sources that support MetaService

MetaService allows you to access OSS, Log Service, and MNS. You can use an EMR SDK in your EMR cluster to read data from and write data to the preceding data sources without an AccessKey pair.

By default, only access to OSS is enabled. If you want to read data from and write data to Log Service and MNS, log on to the RAM console and configure the required permissions for the AliyunEmrEcsDefaultRole role. For more information, see RAM console.

For more information about how to authorize a RAM role, see Grant permissions to a RAM role.

Use MetaService

• Brings the following benefits for services in your EMR cluster:

  The jobs that you run in the services can access Alibaba Cloud resources (OSS, Log Service, and MNS) without an AccessKey pair.

  Comparison of operations before and after MetaService is used:

  • Run the hadoop fs -ls command to view OSS data.
    • MetaService is not used:

      hadoop fs -ls oss://ZaH******As1s:Ba23N**************sdaBj2@bucket.oss-cn-hangzhou-internal.aliyuncs.com/a/b/c

    • MetaService is used:

      hadoop fs -ls oss://bucket/a/b/c

  • Create an external table in Hive.
    • MetaService is not used:

      CREATE EXTERNAL TABLE test_table(id INT, name string)
              ROW FORMAT DELIMITED
              FIELDS TERMINATED BY '/t'
              LOCATION 'oss://ZaH******As1s:Ba23N**************sdaBj2@bucket.oss-cn-hangzhou-internal.aliyuncs.com/a/b/c';

    • MetaService is used:

      CREATE EXTERNAL TABLE test_table(id INT, name string)
              ROW FORMAT DELIMITED
              FIELDS TERMINATED BY '/t'
              LOCATION 'oss://bucket/a/b/c';

  • Use Spark to view OSS data.
    • MetaService is not used:

      val data = sc.textFile("oss://ZaH******As1s:Ba23N**************sdaBj2@bucket.oss-cn-hangzhou-internal.aliyuncs.com/a/b/c")

    • MetaService is used:

      val data = sc.textFile("oss://bucket/a/b/c")

• Brings the following benefits for self-deployed services:
  MetaService is an HTTP service. You can access the URL of this HTTP service to obtain a Security Token Service (STS) temporary credential. Then, you can use the STS temporary credential to access Alibaba Cloud resources without an AccessKey pair in self-managed systems.

  Important A new STS temporary credential is generated 30 minutes before the current one expires. Both STS credentials can be used within the 30 minutes.

  For example, you can run curl http://localhost:10011/cluster-region to obtain the region where your cluster resides.

  You can use MetaService to obtain the following information:

  • Region: /cluster-region
  • Role name: /cluster-role-name
  • AccessKey ID: /role-access-key-id
  • AccessKey secret: /role-access-key-secret
  • Security token: /role-security-token
  • Network type: /cluster-network-type