All Products
Search

This Product

    E-MapReduce:Manage YARN partitions in the EMR console

    Document Center

    E-MapReduce:Manage YARN partitions in the EMR console

    Last Updated:Nov 30, 2023

    E-MapReduce (EMR) allows you to manage Yet Another Resource Negotiator (YARN) partitions in the console in a visualized manner. You can establish mappings between multiple node groups and partitions at a time. You can configure partition properties for a node group. After a manual scale-out or auto scaling on the node group, EMR adds labels for the new nodes. This way, you do not need to configure the new nodes. This topic describes how to manage YARN partitions in the EMR console.

    Background information

    You can use the YARN partition management feature in the following scenarios:

    • Different types of instances exist in the environment.

      In the actual environment, you may need to deploy different types of instances. For example, some instances are compute-optimized instances, and some instances are memory-optimized instances. The newly purchased instances are compute-optimized instances with a large memory and a large number of vCPUs. The existing instances are storage-optimized instances.

    • Cluster resources are shared by multiple computing frameworks.

      The applications that are used to perform batch processing jobs, such as a Flink application on an offline cluster, use a large amount of network resources or CPU resources. In this case, sufficient resources may not be available for quasi-real-time computing.

    • The performance of important jobs must be ensured.

      If you do not want important jobs to be scheduled to elastic nodes, you can configure a separate partition for inelastic nodes and allow important jobs to use the partition. This prevents latency on these important jobs due to a scale-in activity.

    You can use the YARN partition management feature to associate nodes with specific partitions based on the roles and processing capacities of the nodes. Then, you can run jobs on nodes in the specified partition. This helps you meet your multi-dimensional business requirements and better manage and schedule different types of resources and applications.

    Prerequisites

    A cluster is created and is in the Running state. For more information, see Create a cluster.

    Limits

    • The cluster must run EMR V5.11.1 or later minor versions or EMR V3.45.1 or later minor versions.

    • The cluster must be in the Running state.

    • The capacity scheduler must be used as the YARN scheduler.

    Usage notes

    • After you add or edit a partition, you must click Deploy to make the configurations take effect. We recommend that you perform the operation during off-peak hours of your business.

    • If you use the fair scheduler as the YARN scheduler, you must turn off Node Labels.

    Add a partition

    1. Go to the Configure tab of the YARN service page.

      1. Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.

      2. In the top navigation bar, select the region in which your cluster resides and select a resource group based on your business requirements.

      3. On the EMR on ECS page, find the cluster that you want to manage and click Services in the Actions column.

      4. On the Services tab, click Configure in the YARN section.

    2. Click the Partitions tab.

    3. Click Add Partition. In the Add Partition dialog box, configure the following parameters.

      Parameter

      Description

      Partition Name

      The name of the partition to be added.

      The name must be 1 to 255 characters in length and can contain letters, digits, hyphens (-), and underscores (_).

      Note

      • You can create a partition named DEFAULT.

      • The name cannot start with a hyphen (-) or underscore (_).

      Partition Type

      • Exclusive: Containers can be allocated to a node of an exclusive partition only when the partition is specified. This is the default value.

      • Non Exclusive: Containers can be allocated to a node of a non-exclusive partition when the partition is specified. If the partition to which containers are allocated is not specified, or the DEFAULT partition is specified as the partition to which containers are allocated, containers can be allocated to a node of a non-exclusive partition that has idle resources.

      Associated Node Groups

      The node groups with which you want to associate the partition. Each node group can be associated with only one partition. You can modify the associated node groups.

    4. Click OK.

    5. Click Deploy.

      After the configurations take effect, you can view the added partition on the Partitions tab.

      By default, the configurations take effect in 10 minutes. If you want the configurations to take effect in a shorter period of time, perform the following operations: Add the yarn.nodemanager.node-labels.resync-interval-ms and yarn.nodemanager.node-labels.provider.fetch-interval-ms configuration items on the yarn-site.xml tab of the Configure tab. Change the time required for the configurations to take effect. Then, manually restart the NodeManager component on the Status tab.

    Edit a partition

    1. On the Partitions tab of the YARN service page, find the partition that you want to edit and click Edit in the Actions column.

    2. In the Edit Partition dialog box, modify the associated node groups. Other parameters cannot be modified.

    3. Click OK.

    4. Click Deploy.

    Delete a partition

    1. On the Partitions tab of the YARN service page, find the partition that you want to delete and click Delete in the Actions column.

    2. In the Confirm message, click OK.

    3. Click Deploy.

      Note

      • If you delete a partition, the node groups that are associated with the partition are disassociated.

      • If the partition is associated with a queue, you must turn off Disable Partition-queue Association Management on the Edit Resource Queue tab before you can delete the partition.

    View the partition information

    On the Partitions tab of the YARN service page, you can view the information about each partition, including the partition name, queues that can access the partition, associated node group, partition type, and total amount of resources.

    Disable the partition management feature

    If you want to switch the YARN scheduler from the capacity scheduler to the fair scheduler, you must turn off Node Labels to disable the partition management feature. Then, on the Status tab of the YARN service page, manually restart the ResourceManager component to make the configurations take effect.

    Remove the Partitions tab from the EMR console

    If you want to use a custom script to manage your partitions and disable the partition management feature of the EMR console, you can set the node_labels_managed_by_emr parameter on the yarn-operator-conf tab to false on the Configure tab of the YARN service page. After you refresh the EMR console, the Partitions tab is invisible and partition management is no more verified based on the node-labels.xml file. For more information about how to use the node label feature, see Node labels.