E-MapReduce:Use Cruise Control

Prerequisites

A Dataflow cluster of EMR V3.42.0 or later is created, and the Kafka service is selected from the optional services during cluster creation. For more information about how to create a cluster, see Create a Dataflow Kafka cluster.

Procedure

1. Go to the configuration page of the Kafka service.

   1. Log on to the EMR on ECS console.

   2. In the top navigation bar, select the region where your cluster resides and select a resource group based on your business requirements.

   3. On the EMR on ECS page, find the cluster that you want to manage and click Services in the Actions column.

   4. On the Services tab, click Configure in the Kafka section.

2. Enable the Cruise Control reporter to collect the metrics of the Kafka cluster.

   1. On the Configure tab, click the server.properties tab.

   2. Change the value of the kafka.cruisecontrol.metric.reporter.enable parameter to true.

   3. Click Save.

   4. In the dialog box that appears, configure the Execution Reason parameter, and then click Save.

3. Configure the performance information of Kafka brokers.

   You can click the cruisecontrol_capacity.json tab to view the performance information of the current configuration. If the preset configurations do not match the actual performance parameters of the broker, you can modify the parameters.

4. Configure the public IP address to access the Cruise Control component.

   1. Click the cruisecontrol.properties tab.

   2. Change the value of the cruise.control.ui.access.host parameter to the public IP address of the master-1-1 node.

      You can view the public IP address of the master-1-1 node on the Nodes page in the EMR console.

   3. Click Save.

   4. In the dialog box that appears, configure the Execution Reason parameter, and then click Save.

5. Restart the Kafka broker.

   1. On the EMR on ECS page, choose More > Restart.

   2. In the dialog box that appears, configure the Execution Reason parameter and click OK.

   3. In the Confirm message, click OK.

6. Start the Cruise Control service.

   1. Log on to the cluster in SSH mode. For more information, see Log on to a cluster.

   2. Run the following command to start the Cruise Control service:

      cd $KAFKA_ROOT/cruise-control-current/
      ./emr-cruise-control-start.sh

7. View the logs of the Cruise Control service.

   Run the following command to view the logs in the directory. You can view the logs to check whether the startup process is normal.

   ll $LOG_DIR_ROOT/cruise-control

   For example, the following information is returned:

   -rw-r--r-- 1 kafka hadoop 109095 Jul 20 15:07 kafkacruisecontrol.log
   -rw-r--r-- 1 kafka hadoop      0 Jul 20 15:07 kafkacruisecontrol-operation.log
   -rw-r--r-- 1 kafka hadoop  87452 Jul 20 15:07 kafka-cruise-control.out
   -rw-r--r-- 1 kafka hadoop      0 Jul 20 15:07 kafkacruisecontrol-request.log

8. Access the Cruise Control service.

   You can use the Cruise Control service by accessing the Cruise Control user interface (UI) or calling REST API operations.

   • By default, the address of the Cruise Control UI is http://<Public IP address of the master-1-1 node>:9090/.

     Note

     Before you access the Cruise Control UI, you must add port 9090 to the security group. For more information, see Add a security group rule.

   • For more information about how to use REST API operations, see REST-APIs.