E-MapReduce:Create a cluster

Configuration

Description

Region

A region is a geographic area where a data center is located. Select a region close to you to reduce network latency. The region cannot be changed after the instance is created.

From the Region drop-down list, select the physical location for the EMR instance.

Business Scenario

Select a scenario based on your actual needs:

• Data Lake: Provides a flexible, reliable, and efficient cluster to run big data compute engines and deliver excellent data analytics capabilities.

  • Supports building a data lake architecture and uses JindoFS for data lake acceleration.

  • Supports Hadoop Distributed File System (HDFS) on OSS (a fully managed HDFS) for storage. This reduces your O&M costs and is billed based on usage.

  For more information, see DataLake cluster.

• Data Analytics (OLAP): Import or use external tables to load massive amounts of data into an online analytical processing (OLAP) engine, such as ClickHouse or StarRocks. This provides efficient, real-time, and flexible data analytics for scenarios such as user personas, audience segmentation, BI reports, and business analytics.

• Real-time Data Streaming (DataFlow): A one-stop, real-time computing solution on the EMR platform. It includes two major components: Kafka, a distributed, high-throughput, and scalable messaging system, and a commercial Flink kernel from Ververica, which is based on the official Apache Flink product. It focuses on solving various end-to-end real-time computing problems and is widely used in scenarios such as real-time extract, transform, and load (ETL) and log collection and analysis. You can also use either component individually.

• Data Service (DataServing):

  • Provides a flexible, reliable, and efficient data service cluster.

  • Provides a semi-managed HBase cluster and can decouple the compute cluster from data storage based on the OSS-HDFS (JindoFS) service.

  • Supports JindoData local cache to further improve the read and write performance of the data service cluster.

  For more information, see DataServing cluster.

• Custom Cluster: A custom cluster provides a rich combination of services. Select services as needed.

  Note

  In a production environment, avoid deploying multiple storage services on the same node group.

Product Version

The release version of the EMR product. For more information, see Release versions.

High Service Availability

This feature is disabled by default. If you enable high availability, EMR creates multiple master nodes to support high availability for ResourceManager and NameNode. EMR distributes these nodes across different underlying hardware to reduce the risk of failure.

Optional Services (Select One At Least)

Select other services as needed. The related service processes for the selected services will start by default.

Collect Service Operational Logs

You can enable or disable log collection for all services with one click. This feature is enabled by default to collect your service operational logs. These logs are used only for cluster diagnostics.

After the cluster is created, change the Collection Status of Service Operational Logs on the Basic Information page.

Metadata

The following methods are supported for storing and managing metadata:

• DLF Unified Metadata (Recommended): Metadata is stored in Data Lake Formation (DLF).

  After you activate DLF, the system selects a default DLF Catalog for you. The default catalog is your UID. If you want to use different data catalogs for different clusters, create a new catalog as follows:

  1. Click Create Catalog. In the dialog box that appears, enter a catalog ID and click OK.

  2. From the DLF Catalog drop-down list, select the data catalog that you created.

• Self-managed RDS: Select your own or an Alibaba Cloud RDS instance as the metastore.

  If you select this option, configure the RDS-related parameters. For more information, see Configure a self-managed RDS database.

• Built-in MySQL (Not recommended): Metadata is stored in the MySQL database in the cluster's local environment.

  Note

  • Test scenarios: Use DLF Unified Metadata.

  • Production scenarios: Use DLF Unified Metadata or Self-managed RDS.

Root Storage Directory of Cluster

Configure this parameter when you select the OSS-HDFS service in the optional services section. This parameter is not required if you select the HDFS service.

The first time you use the OSS-HDFS service, the Alibaba Cloud account must click here and follow the prompts to complete the authorization. For a Resource Access Management (RAM) user, the Alibaba Cloud account must grant authorization to activate the service and grant the AliyunEMRDlsFullAccess permission, and the AliyunOSSDlsDefaultRole and AliyunEMRDlsDefaultRole roles. For more information, see Grant permissions to a RAM user. Select a bucket for which the OSS-HDFS service is activated in the same region, or click Create OSS-HDFS Bucket and follow the prompts to create an OSS-HDFS instance as the cluster's root storage path.

Configuration

Description

Billing Method

The default billing method is subscription. The following billing methods are supported:

• Pay-as-you-go: A post-paid mode where you pay after use. With pay-as-you-go, you pay for the actual hours of use. Billing occurs hourly. This method is suitable for short-term test tasks or flexible, dynamic tasks.

• Subscription: A pre-paid mode where you pay before use.

  Note

  • For test scenarios, use Pay-as-you-go. After testing is successful, create a new Subscription cluster for production use.

  • For subscription instances, also select a Subscription Duration and whether to enable Auto-renewal. The default renewal duration is six months, and auto-renewal is enabled. If auto-renewal is enabled, the auto-renewal operation is performed seven days before the instance expires. For more information, see Renewal policy.

Zone

A zone is a distinct physical area within the same region. Zones within the same region can communicate with each other over the internal network. You can usually use the default zone.

VPC

A virtual private cloud (VPC) is an isolated network environment that you define in Alibaba Cloud. You have full control over your VPC.

Select an existing VPC, or click Create VPC to go to the VPC console and create a VPC. For more information, see Create and manage a VPC.

vSwitch

A vSwitch is a basic network module of a VPC that connects different cloud resources.

Select an existing vSwitch, or click . Create vSwitch to go to the VPC console and create a vSwitch. For more information, see Create and manage a vSwitch.

Default Security Group

A security group is a virtual firewall that controls the inbound and outbound traffic of instances within the security group. For more information, see Security group overview.

Select an existing security group, or click create a new security group. to go to the ECS console and create a new security group. For more information, see Create a security group.

Node Group

Select an instance type as needed. For more information, see Instance families.

• Master: Mainly responsible for deploying control processes such as ResourceManager and NameNode.

• Core: Mainly responsible for storing all data of the cluster. You can also scale out core nodes as needed after the cluster is created.

• Task: Does not store data. It is used to adjust the computing capacity of the cluster. This is disabled by default but can be configured as needed.

  Important

  The supported billing methods for task node groups are pay-as-you-go, spot instance, and subscription.

• Add to Deployment Set: After high availability is enabled, the master node is added to a deployment set by default. A deployment set is a policy that controls the distribution of instances. For more information, see Deployment sets.

• System Disk: Select a standard SSD, enterprise SSD (ESSD), or ultra disk as needed. Adjust the system disk size as needed.

• Data Disk: Select a standard SSD, ESSD, or ultra disk as needed. Adjust the data disk size as needed.

  Note

  When you select an ESSD, you can set different performance levels (PLs) based on the capacity of the selected disk to meet different cluster performance requirements. The default performance level is PL1. Based on the disk capacity range, the system disk supports PL0, PL1, and PL2 ESSD specifications, and the data disk supports PL0, PL1, PL2, and PL3 ESSD specifications. For more information about cloud disks, see Cloud disk overview.

• Instances: The master node group has one instance by default. If high availability is enabled, you can have multiple Master instances.

  The core node group has two instances by default. You can adjust this number as needed.

• Additional Security Group: An additional security group lets you flexibly customize access between different external resources or applications. You can associate up to two additional security groups with this node group.

• Assign Public Network IP: Specifies whether to assign an Elastic IP Address (EIP) to the cluster. This is disabled by default. Only DataLake clusters support assigning public IPs at the node group level.

  Note

  If you do not enable this feature but want to access the cluster using a public IP address after creation, apply for a public IP address on ECS. For more information, see Apply for an EIP.

Cluster Scaling

Select a scaling rule as needed:

• Do Not Use Auto Scaling Rules (Default).

• Custom Auto Scaling Rule: Create a custom scaling rule to enable automatic scaling based on time or load. For more information, see Create a custom auto scaling rule.

• Managed Auto Scaling Rule: When you start the cluster, EMR pre-allocates the number of task nodes based on the managed scaling rule. For more information, see Create a managed auto scaling rule.

Configuration Item

Description

Cluster Name

The name of the cluster. The name must be 1 to 64 characters in length and can contain Chinese characters, letters, digits, hyphens (-), and underscores (_).

Identity Credentials

The Identity Credentials are used to securely log on to the cluster's master node. For logon operations, see Log on to a cluster. The following identities are supported:

• Key Pair (Default): Select an existing key pair, or click Create Key Pair to create one immediately.

  A key pair is a secure and convenient logon authentication method that consists of a public key and a private key. It is supported only for Linux instances. For more information about using key pairs, see SSH key pairs.

• Password: Set the logon password for the master node and confirm the password. The default username is root.

  • Password rules: 8 to 30 characters in length, and must contain uppercase letters, lowercase letters, digits, and special characters.

  • Special characters include the following: exclamation point (!), at sign (@), pound sign (#), dollar sign ($), percent sign (%), caret (^), ampersand (&), and asterisk (*).