E-MapReduce (EMR) provides a YARN-based dashboard that allows you to view the resource utilization and auto scaling effect of clusters. You can also configure and optimize custom auto scaling rules based on historical YARN cluster metrics. This topic describes how to view the overview information about EMR cluster resources in the EMR console.
Prerequisites
A DataLake cluster or a custom cluster is created and the YARN service is selected when you create the cluster. For more information, see Create a cluster.
Procedure
Method 1: View the comparison of YARN resource utilization
Go to the Auto Scaling tab.
Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.
In the top navigation bar, select the region where your cluster resides and select a resource group based on your business requirements.
On the EMR on ECS page, find the desired cluster and click the name of the cluster in the Cluster ID/Name column.
Click the Auto Scaling tab in the upper part of the cluster details page.
On the Configure Auto Scaling subtab, view the comparison of YARN resource utilization.
You can view a trend line that shows the changes in resource utilization of the current cluster over the previous 30 days. You can also compare the trend line with the trend line that shows the changes in resource utilization of the current cluster over the previous N days and check whether cluster resources are wasted. N is an integer that is less than 30. The system checks the resource utilization of the current cluster and provides the recommended auto scaling rule. You can configure auto scaling for the cluster node group based on the recommended rule.
Method 2: View YARN metric data
Go to the Metric Monitoring subtab.
Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.
In the top navigation bar, select the region where your cluster resides and select a resource group based on your business requirements.
On the EMR on ECS page, find the desired cluster and click the name of the cluster in the Cluster ID/Name column.
Click the Monitoring and Diagnostics tab in the upper part of the cluster details page.
Click the Metric Monitoring subtab.
Select YARN-HOME from the Dashboard drop-down list. The dashboard displays the YARN metric data of the cluster. You can select a time range to view the resource utilization of the cluster over the specified period.
View the Yarn Scaling metric in the dashboard. Yarn Scaling is a metric that is used to show YARN auto scaling. This metric can help you gain insight into the resource utilization of containers and nodes that are running in a cluster, and further understand the auto scaling effect of the cluster. This allows you to accurately evaluate the auto scaling effect of the cluster.