Application Real-Time Monitoring Service:How do I deploy and configure a Kafka JMX agent?

Limit

You can deploy a Kafka JMX agent only in a Prometheus instance for Container Service or a Prometheus instance for ECS.

Step 1: Deploy a Kafka JMX agent

1. Download the kafka JMX-Agent package to the pod or ECS instance where Kafka resides.

2. Add -javaagent:/{jmx-agent directory}/kafka-jmx_prometheus_javaagent-1.18.1.jar={jmx listening port} to the Java Virtual Machine (JVM) startup parameters of Kafka Producer, Broker, and Consumer.

   • jmx-agent directory: Replace the value with the actual directory where the Kafka JMX agent is stored.

   • jmx listening port: Replace the value with the actual JMX listening port.

   

3. Restart Kafka Producer, Broker, and Consumer.

4. Check whether the Kafka JMX agent is running as expected on Kafka Producer, Broker, and Consumer. Run the curl localhost:{jmx listening port}/metrics command on each pod or ECS instance to check whether normal metric data is returned. If normal metric data is returned, the Kafka JMX agent is running as expected on Kafka Producer, Broker, and Consumer.

   jmx listening port: Replace the value with the actual JMX listening port.

   

(Optional) Step 2: Configure containerPort

If you deploy a Kafka JMX agent in a Prometheus instance for Container Service, you also need to configure containerPort. This way, Managed Service for Prometheus can capture the data of the Kafka JMX agent. If you deploy a Kafka JMX agent in a Prometheus instance for ECS, skip this step.

1. Log on to the ACK console.

2. In the left-side navigation pane of the ACK console, click Clusters.

3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.

4. In the left-side navigation pane of the details page, choose Workloads > StatefulSets. Then, click a workload name. In the Port section of the kafka tab, configure the port name, container port number, and protocol.

   • Port name: the port name of the Kafka JMX agent. Example: app1-kafka-jmx.

   • Container port number: the JMX listening port defined in Step 1: Deploy a Kafka JMX agent.

   • Protocol: Select TCP.

Step 3: Attach tags to Kafka pods or ECS instances

Managed Service for Prometheus can use tags to identify pods or ECS instances to implement service discovery. Therefore, you need to attach tags to Kafka pods or ECS instances. If you have attached tags to Kafka pods or ECS instances, you can skip this step.

Configure a tag for a Prometheus instance for Container Service

Configure a tag in the format of {custom tag key}:{ custom tag value} for the pods of Kafka Producer, Broker, and Consumer. Example: arms-kafka-exporter:my-kafka1. Perform the following steps to configure the tag:

1. Log on to the ACK console.

2. In the left-side navigation pane of the ACK console, click Clusters.

3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.

4. In the left-side navigation pane, choose Workloads > Pods. On the Pods page, click the name of the workload. On the pod details page, click Edit. In the dialog box that appears, add a tag and click Update.

Configure a tag for a Prometheus instance for ECS

What to do next

After the Kafka JMX agent is configured, you can configure the agent to view monitoring data in the Managed Service for Prometheus console. For more information, see Use Managed Service for Prometheus to monitor self-managed Kafka clusters and ApsaraMQ for Kafka instances.