ApsaraMQ for Kafka:Service architecture

Cluster architecture

An ApsaraMQ for Kafka cluster consists of four components: producers, brokers, ZooKeeper, and consumer groups.

• Producer

  • A producer sends messages to brokers in the cluster. Messages typically contain data such as page view events, server logs, and system resource information like CPU utilization and memory usage.

• Broker

  • A broker is a server that stores and serves messages. Brokers scale out horizontally. Adding more brokers to the cluster increases throughput.

• Consumer group

  • A consumer group (referred to as "Group" in the ApsaraMQ for Kafka console) pulls messages from brokers to consume data from subscribed topics.

• ZooKeeper

  • ZooKeeper coordinates the cluster. It manages cluster configuration, elects the leader partition, and rebalances load when consumer group membership changes.

Publish-subscribe model

ApsaraMQ for Kafka uses the publish-subscribe model. Producers publish messages to topics, and consumer groups subscribe to those topics to consume messages.

Topics and consumer groups have a many-to-many relationship:

• A single consumer group can subscribe to multiple topics.

• A single topic can be consumed by multiple consumer groups.

Within a consumer group, each message from a subscribed topic is consumed by only one consumer in the group.