E-MapReduce:Create a cluster

Parameter

Description

Region

The geographic location where the Elastic Compute Service (ECS) instances of the cluster are located. To ensure minimal network latency, select a region that is close to your geographical location. After the cluster is created, you cannot change the region.

Select a region from the drop-down list.

Business Scenario

Select a business scenario based on your business requirements. Valid values:

• Data Lake: provides a big data compute engine that allows you to analyze data in a flexible, reliable, and efficient manner.

  • Supports the data lake architecture and accelerates data queries in data lakes based on JindoFS.

  • Supports the OSS-HDFS (fully managed HDFS) service for storage, which helps you reduce O&M costs. You are charged based on actual usage of the OSS-HDFS service.

  For more information, see DataLake cluster.

• Data Analytics: provides efficient, real-time, and flexible data analytics capabilities to meet the requirements of various business scenarios, such as user profiling, recipient selection, BI reports, and business analytics. You can write data to online analytical processing (OLAP) engines such as ClickHouse and StarRocks for analysis by importing data or using external tables.

• Real-time Data Streaming: provides an end-to-end (E2E) real-time computing solution. Dataflow clusters incorporate Kafka, a distributed message system with high throughput and scalability, and the commercial Flink kernel provided by Apache Flink-powered Ververica. The clusters are used to resolve various E2E real-time computing issues and are widely used in real-time data extract, transform, and load (ETL), and log collection and analysis scenarios. You can use one of the two components or both.

• Data Service:

  • Provides a DataServing cluster that allows you to analyze data in a flexible, reliable, and efficient manner.

  • Provides semi-managed HBase clusters and can decouple computing clusters from data storage based on the OSS-HDFS (JindoFS) service.

  • Supports data caching by using JindoData to improve the read and write performance of DataServing clusters.

  For more information, see DataServing cluster.

• Custom Cluster: provides various services. You can select services based on your business requirements.

  Note

  We recommend that you do not deploy multiple storage services on the same node group in the production environment.

Product Version

The version of EMR. For more information, see Overview.

High Service Availability

By default, this switch is turned off. If you turn on the switch, multiple master nodes are created in the cluster to ensure the high availability of the ResourceManager and NameNode processes. In addition, EMR distributes the master nodes across different underlying hardware devices to reduce the risk of failures.

Optional Services (Select One At Least)

The services that you can select for the cluster. You can select services based on your business requirements. The processes related to the services that you select are automatically started.

Collect Service Operational Logs

Specifies whether to enable log collection for all services. By default, this switch is turned on to collect the service operational logs of your cluster. The logs are used only for cluster diagnostics.

After you create a cluster, you can modify the Collection Status of Service Operational Logs parameter on the Basic Information tab.

Metadata

The method for storing and managing metadata. Valid values:

• DLF Unified Metadata: Metadata is stored in Data Lake Formation (DLF). We recommend that you select this method.

  After you activate DLF, the system selects a DLF catalog for you to store metadata. The ID of your account is used by default. If you want different clusters to be associated with different DLF catalogs, you can perform the following operations to create DLF catalogs:

  1. Click Create Catalog. In the popover that appears, enter a catalog ID and click OK.

  2. Select the catalog that you created from the DLF Catalog drop-down list.

• Self-managed RDS: Metadata is stored in a self-managed or Alibaba Cloud ApsaraDB RDS database.

  If you select Self-managed RDS, you must configure the parameters of the existing ApsaraDB RDS database. For more information, see Configure a self-managed ApsaraDB RDS for MySQL database.

• Built-in MySQL: Metadata is stored in the local MySQL database of your cluster. This method is not recommended.

  Note

  • Test environment: We recommend that you select Built-in MySQL.

  • Production environment: We recommend that you select DLF Unified Metadata or Self-managed RDS.

Root Storage Directory of Cluster

The root storage directory of cluster data. This parameter is required only if you select the OSS-HDFS service.

The first time you use OSS-HDFS, you must complete authorization as prompted. If you use a RAM user, you must attach the AliyunEMRDlsFullAccess policy and assign the AliyunOSSDlsDefaultRole and AliyunEMRDlsDefaultRole roles to the RAM user by using your Alibaba Cloud account. For more information, see Grant permissions to RAM users. Select a bucket for which OSS-HDFS is enabled in the same region, or click Create OSS-HDFS Bucket to create an OSS-HDFS bucket as the root storage path of the cluster.

Parameter

Description

Billing Method

The billing method of the cluster. Subscription is selected by default. EMR supports the following billing methods:

• Pay-as-you-go: a billing method that allows you to pay for an instance after you use the instance. The system charges you for a cluster based on the hours the cluster is actually used. Bills are generated on an hourly basis at the top of every hour. We recommend that you use pay-as-you-go clusters for short-term test jobs or dynamically scheduled jobs.

• Subscription: a billing method that allows you to use an instance only after you pay for the instance.

  Note

  • We recommend that you create a pay-as-you-go cluster for a test run. If the cluster passes the test, you can create a subscription cluster for production.

  • If you select Subscription for Billing Method, you must also configure the Subscription Duration and Auto-renewal parameters. By default, the subscription period is six months and the Auto-renewal switch is turned on. If you turn on Auto-renewal, the system renews your subscription for one more month seven days before the expiration date. For more information, see Renewal policy.

Zone

The zone where you want to create a cluster. A zone in a region is a physical area with independent power supplies and network facilities. Clusters in zones within the same region can communicate with each other over an internal network. In most cases, you can use the zone that is selected by default.

VPC

The virtual private cloud (VPC) where you want to deploy the cluster. A VPC is a logically isolated network on which you have full control.

You can select an existing VPC or click create a VPC to create a VPC in the VPC console. For more information, see Create and manage a VPC.

vSwitch

The vSwitch of the cluster. vSwitch is a basic component of VPCs. vSwitches can be used to establish network communication between cloud resources.

You can select an existing vSwitch or click Create vSwitch to create a vSwitch in the VPC console. For more information, see Create and manage a vSwitch.

Default Security Group

The security group of the cluster. A security group is a virtual firewall that is used to control the inbound and outbound traffic of instances in the security group. For more information, see Overview.

You can select an existing security group or click create a new security group to create a security group in the ECS console. For more information, see Create a security group.

Node Group

The node groups of the cluster. You can select instance types based on your business requirements. For more information, see Instance families.

• Master node group: runs control processes, such as ResourceManager and NameNode.

• Core node group: stores all the data of a cluster. You can add core nodes based on your business requirements after a cluster is created.

• Task node group: stores no data and is used to adjust the computing capabilities of clusters. No task node group is configured by default. You can configure a task node group based on your business requirements.

• Add to Deployment Set: If you turn on the High Service Availability switch, the master nodes are added to a deployment set by default. A deployment set is used to control the distribution of ECS instances. For more information, see Overview.

• System Disk: You can select a standard SSD, enhanced SSD, or ultra disk based on your business requirements. You can adjust the size of the system disk based on your business requirements.

• Data Disk: You can select standard SSDs, enhanced SSDs, or ultra disks based on your business requirements. You can adjust the size of the data disks based on your business requirements.

  Note

  If you select enhanced SSDs, you can specify different performance levels (PLs) for the enhanced SSDs based on the disk capacity to meet different cluster performance requirements. The default performance level is PL1. When you configure the system disk, you can select an enhanced SSD of the following performance levels: PL0, PL1, and PL2. When you configure data disks, you can select enhanced SSDs of the following performance levels: PL0, PL1, PL2, and PL3. For more information, see Disks.

• Instances: One master node is configured by default. If you turn on the High Service Availability switch, multiple master nodes can be configured.

  Two core nodes are configured in the core node group by default. You can change the number of core nodes based on your business requirements.

• Additional Security Group: An additional security group allows interactions between different external resources and applications. You can associate a node group with up to two additional security groups.

• Assign Public Network IP: specifies whether to associate an EIP address with the cluster. This switch is turned off by default. You can assign public IP addresses only to the node groups of DataLake clusters.

  Note

  If you do not turn on this switch but want to access the cluster over the Internet after you create the cluster, you must apply for a public IP address on ECS. For information about how to apply for an EIP address, see Elastic IP addresses.

Parameter

Description

Cluster Name

The name of the cluster. The name must be 1 to 64 characters in length and can contain only letters, digits, hyphens (-), and underscores (_).

Identity Credentials

The credentials that are used to log on to the master node of the cluster. For more information, see Log on to a cluster. Valid values:

• Key Pair (default): Select an existing key pair or click Create Key Pair to create a key pair.

  Key pairs are a secure and convenient authentication method provided for ECS instance logons. Only Linux instances support key pair-based authentication. For information about how to use a key pair, see SSH key pair overview.

• Password: Configure a password for the master node and confirm the password. By default, the username is root.

  • The password must be 8 to 30 characters in length and must contain uppercase letters, lowercase letters, digits, and special characters.

  • The following special characters are supported: ! @ # $ % ^ & *