Enable horizontal pod autoscaling based on the metrics of ApsaraMQ for RocketMQ - Container Service for Kubernetes

You can use Kubernetes-based Event Driven Autoscaler (KEDA) to enable automated and efficient horizontal pod autoscaling for your applications based on message accumulation metrics of ApsaraMQ for RocketMQ. This prevents overloading and service interruptions due to message accumulation and improves system reliability and stability.

Overview

As the business of enterprises develops, the demand for messaging increases. As a distributed messaging middleware service that features high performance, high reliability, and high scalability, ApsaraMQ for RocketMQ is widely adopted by enterprise-class applications. You may encounter message accumulation when you use ApsaraMQ for RocketMQ, especially when the loads are high. This may further increase the system loads and even cause application break down. To resolve this issue, you can use KEDA to enable horizontal pod autoscaling for your applications based on custom message accumulation metrics of ApsaraMQ for RocketMQ.

KEDA can automatically scale applications based on the message accumulation metrics with high efficiency. This ensures the reliability and stability of the system. If you use open source Apache RocketMQ, you can enable horizontal pod autoscaling based on the metrics collected by the Java Management Extensions (JMX) Prometheus exporter. For more information, visit Apache RocketMQ.

In this topic, horizontal pod autoscaling is configured based on the metrics of ApsaraMQ for RocketMQ collected by Managed Service for Prometheus (Prometheus).

Prerequisites

The ack-keda component is deployed. For more information, see ACK KEDA.
An ApsaraMQ for RocketMQ instance is created.
A Container Service for Kubernetes (ACK) cluster is created. For more information, see Create an ACK managed cluster or Create an ACK dedicated cluster.

Step 1: Deploy an application

Log on to the ACK console. In the left-side navigation pane, click Cluster.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, choose Workloads > Deployments.
In the upper-right corner of the Deployments page, click Create from YAML.

On the Create page, add the following content to the code editor and then click Create to deploy an application named sample-app.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  namespace: default
  labels:
    app: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: sample-app
        image: nginx:1.7.9  // Specify the actual image used by the consumer application. 
        resources:
          limits:
            cpu: "500m"

Step 2: Create a ScaledObject

Log on to the ApsaraMQ for RocketMQ console. In the left-side navigation pane, click Instances.
Find the instance that you want to manage and click its name. The instance details page appears. In the left-side pane, click Topics. Copy the topic name and instance ID in the upper-right corner of the page. In this example, the topic name is keda and the instance ID is mq-cn-uax33****.
Log on to the ARMS console.
In the left-side navigation pane, choose Managed Service for Prometheus > Instances.
Find the instance named cloud-product-prometheus_{{RegionId}} and click its name. In the left-side pane, click Settings. Click Cloud Service: rocketmq and then copy an endpoint in the HTTP API Address (Grafana Read Address) section.

Create a file named ScaledObject.yaml and add the following content to the file:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: prometheus-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: sample-app
  maxReplicaCount: 10
  minReplicaCount: 2
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://cn-beijing.arms.aliyuncs.com:9090/api/v1/prometheus/8cba801fff65546a3012e9a684****/****538168824185/cloud-product-rocketmq/cn-beijing
      metricName: rocketmq_consumer_inflight_messages
      query: sum({__name__=~"rocketmq_consumer_ready_messages|rocketmq_consumer_inflight_messages",instance_id="rmq-cn-uax3xxxxxx",topic=~"keda"}) by (consumer_group)
      threshold: '30'

The following table describes the parameters.

Parameter	Description
scaleTargetRef.name	The object that you want to scale. In this example, the value is set to sample-app, which is the name of the application that you created in Step 1: Deploy an application.
maxReplicaCount	The maximum number of replicated pods.
minReplicaCount	The minimum number of replicated pods.
serverAddress	The endpoint of the Prometheus instance that stores the metrics of the ApsaraMQ for RocketMQ instance. In this example, the value is set to the endpoint that you copied in the HTTP API Address (Grafana Read Address) section in Step 4.
metricName	The Prometheus Query Language (PromQL) query data.
query	Aggregates the PromQL query data specified in the metricName parameter. In this example, the statistics of the number of accumulated messages are aggregated.
threshold	The scaling threshold. In this example, the scaling threshold is 30, which indicates that the application is scaled out if the number of accumulated messages exceeds 30.

Run the following command to create a ScaledObject:

// Deploy a ScaledObject. 
kubectl apply -f ScaledObject.yaml

scaledobject.keda.sh/prometheus-scaledobject created

// Query the status of the ScaledObject. 
kubectl get ScaledObject

NAME                      SCALETARGETKIND      SCALETARGETNAME   MIN   MAX   TRIGGERS     AUTHENTICATION   READY   ACTIVE   FALLBACK   AGE
prometheus-scaledobject   apps/v1.Deployment   sample-app        2     10    prometheus                    True    False    False      105s

// Check whether a Horizontal Pod Autoscaler (HPA) is created to scale the application. 
kubectl get hpa

NAME                               REFERENCE               TARGETS      MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-prometheus-scaledobject   Deployment/sample-app   0/30 (avg)   2         10        2          28m

Step 3: Specify the endpoint, username, and password of the ApsaraMQ for RocketMQ instance

In this example, the rocketmq-keda-sample project produces and consumes data. You must specify the endpoint, username, and password of the ApsaraMQ for RocketMQ instance in the code of the project. The information is obtained in Step 2.

Step 4: Produce and consume data to trigger application scaling

Log on to the ApsaraMQ for RocketMQ console. In the left-side navigation pane, click Instances.
Find the instance that you want to manage and click its name. On the instance details page, copy the values of the Username, Password, and Endpoint and Network Information parameters.
Run a producer program to produce data and then run the following command to query information about the HPA:
```
kubectl get hpa
```
Expected output:
```
NAME                               REFERENCE               TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-prometheus-scaledobject   Deployment/sample-app   32700m/30 (avg)   2         10        10         47m
```
The output indicates that the number of replicated pods for the sample-app application is scaled to the maximum value specified in KEDA.

Stop the producer program and run a consumer program. Then, run the following command:

kubectl get hpa -w

Expected output:

NAME                               REFERENCE               TARGETS            MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-prometheus-scaledobject   Deployment/sample-app   222500m/30 (avg)   2         10        10         50m
keda-hpa-prometheus-scaledobject   Deployment/sample-app   232400m/30 (avg)   2         10        10         51m
keda-hpa-prometheus-scaledobject   Deployment/sample-app   0/30 (avg)         2         10        10         52m
keda-hpa-prometheus-scaledobject   Deployment/sample-app   0/30 (avg)         2         10        2          57m

The output indicates that the number of replicated pods for the sample-app application is scaled to the minimum value specified in KEDA at a point in time after data consumption ends.

Reference

For more information about how to configure horizontal pod autoscaling based on the metrics of ApsaraMQ for RabbitMQ and KEDA, see Horizontal pod autoscaling based on the metrics of ApsaraMQ for RabbitMQ.