This topic introduces the terms that are used in ApsaraMQ for Kafka to help you better understand and use ApsaraMQ for Kafka.
Apache Kafka
A distributed and open source platform for data stream processing. This platform allows you to publish, subscribe to, store, and process data streams in real time. For more information, see Apache Kafka.
ApsaraMQ for Kafka
A fully managed Apache Kafka service provided by Alibaba Cloud. This service frees you from deployment and O&M, and offers you a cost-effective service that features high scalability, reliability, and throughput. For more information, see What is ApsaraMQ for Kafka?
ZooKeeper
A distributed and open source coordination service for applications. In ApsaraMQ for Kafka, ZooKeeper is used to manage clusters and configurations and elect leaders. ZooKeeper is part of the ApsaraMQ for Kafka architecture. You do not need to concern yourself with ZooKeeper.
broker
A server node in ApsaraMQ for Kafka. ApsaraMQ for Kafka provides a fully managed service that automatically changes the broker quantity and configurations based on the traffic specifications of instances. You do not need to concern yourself with the details of brokers.
cluster
A collection of multiple brokers.
instance
An independent resource entity that corresponds to a cluster in ApsaraMQ for Kafka.
VPC-connected instance
An instance that provides only a virtual private cloud (VPC) endpoint and can be accessed only from the VPC in which the instance is deployed.
Internet- and VPC-connected instance
An instance that provides public and VPC endpoints and can be accessed from the VPC in which the instance is deployed and over the Internet.
major version upgrade
An upgrade in which the major version changes. For example, you can upgrade an ApsaraMQ for Kafka instance from version 0.10.x to version 2.x. For more information, see Upgrade instance versions.
minor version update
An update in which the major version does not change. For example, you can update an ApsaraMQ for Kafka instance from version 0.10 to version 0.10.2, or from version 0.10.2 to the 0.10.2 kernel-optimized version. For more information, see Upgrade instance versions.
endpoint
An address that is used by producers or consumers to connect to ApsaraMQ for Kafka. This address can consist of the IP address and port number of a broker in the format of Broker IP address:Port number. This address can also consist of the domain name and port number of an ApsaraMQ for Kafka instance in the format of Instance domain name:Port number. For more information, see Comparison among endpoints.
message
A carrier for transferring information in ApsaraMQ for Kafka. Messages can be page views, server logs, and information about system resources such as CPU and memory resources. In ApsaraMQ for Kafka, a message is a byte array.
message retention duration
The maximum period of time for which messages can be retained when disk capacity is sufficient.
After the storage usage exceeds 85%, the system deletes messages in the order in which the messages are stored. The earliest message that is stored is the first message that is deleted. This helps ensure that the performance of the service is not degraded.
Valid values: 24 to 480. Default value: 72. Unit: hours.
maximum message size
The maximum size of a message that can be sent and received in ApsaraMQ for Kafka.
You can specify a message size of up to 10 MB for instances of the Standard Edition and the Professional Edition.
Before you change the maximum message size to a new value, make sure that the new value is the same as the maximum message size that you specified for message producers and message consumers in the instance.
publish-subscribe model
An asynchronous communications model between services. A publisher sends a message to a specific topic without the need to acquire information about the subscriber that subscribes to the message. A subscriber receives a message from a specific topic without the need to acquire information about the publisher that sends the message. ApsaraMQ for Kafka supports the publish-subscribe model. For more information, see Publish-subscribe model of ApsaraMQ Kafka.
subscription
The subscription relationship between consumer groups and topics. ApsaraMQ for Kafka allows you to query the status of connected consumer groups that subscribe to a specific topic. The status of disconnected consumer groups cannot be queried.
producer
An application that sends messages to ApsaraMQ for Kafka.
consumer
An application that receives messages from ApsaraMQ for Kafka.
group
A group of consumers that have the same group ID. If messages in a topic are consumed by multiple consumers in the same consumer group, each message in the topic is delivered only to one of the consumers. This balances the load among consumers and allows messages in a topic to be concurrently consumed.
topic
A message topic that is used to classify messages.
topic traffic rebalance
An operation that evenly redistributes topic traffic after an ApsaraMQ for Kafka cluster is scaled out. For more information, see Rebalance topic traffic.
partition
In ApsaraMQ for Kafka, each topic is divided into partitions. Each partition is an ordered message queue that allows independent data reads and writes. The number of partitions in a topic is determined by the throughput and concurrency of the ApsaraMQ for Kafka cluster.
replicas
ApsaraMQ for Kafka allows you to create multiple replicas for each partition to provide data redundancy and high availability. The number of replicas is the number of replicas of a partition. In most cases, the number of replicas of a partition is determined by the availability and fault tolerance capability of the ApsaraMQ for Kafka cluster. While improving availability and data redundancy, a large number of replicas increases storage and network overheads. By default, each partition has three replicas, which means the partition replication factor is 3.
partition replicas
The number of partition replicas is the total number of replicas of all partitions in a topic. The number of partition replicas is calculated by using the following formula: Number of partition replicas = Number of partitions in a topic × Partition replication factor. For example, if a topic has 10 partitions and each partition has three replicas, the number of partition replicas in the topic is 30. The number of partition replicas affects the storage and throughput of the ApsaraMQ for Kafka cluster. While improving fault tolerance and read performance, a large number of partition replicas increase network and storage overheads.
offset
A sequence number that is assigned to a message when the message arrives at a partition.
earliest offset
The offset for the first message in a partition. For information about how to query the earliest offset of a partition, see View partition status.
maximum offset
The offset for the most recent message in a partition. For information about how to query the latest offset of a partition, see View partition status.
consumer offset
The latest offset of messages that are consumed in a partition. For information about how to query a consumer offset, see View consumer information.
latest consumption time
The time when the most recent message consumed by a group is published to and stored on the ApsaraMQ for Kafka broker. If no messages are accumulated, the time is close to the message sending time.
accumulated messages
The total number of messages that are accumulated in a partition. The value is equal to the latest offset minus the consumer offset. The number of accumulated messages is a key metric in ApsaraMQ for Kafka. If a large number of messages are accumulated, consumers may be blocked, or the consumption rate cannot keep up with the production rate. In this case, you must analyze the status of consumers and improve the consumption rate. You can clear all accumulated messages, and start to consume messages from the latest offset or reset the consumer offset based on the point in time. For more information, see Reset consumer offsets.
local storage
A storage engine that uses the In-Sync Replicas (ISR) algorithm of native Apache Kafka. If you have special requirements, such as compaction, idempotence, transactions, and partitionally ordered messages, we recommend that you use local storage. For more information, see Comparison between storage engines.
cloud storage
A storage engine that uses the algorithm for Alibaba Cloud disks. Cloud storage has the benefits of the underlying storage of Alibaba Cloud. Compared with local storage, cloud storage provides better performance in auto scaling, reliability, availability, and cost-effectiveness. In most cases, we recommend that you use cloud storage. For more information, see Comparison between storage engines.
cleanup policy
A log cleanup policy. You need to configure a log cleanup policy only if you use local storage. You can specify local storage as the storage engine only when you create topics on ApsaraMQ for Kafka Professional Edition instances. ApsaraMQ for Kafka supports the following log cleanup policies:
Delete: the default log cleanup policy. If sufficient storage space is available in the system, messages are retained based on the maximum retention period. After the storage usage exceeds 85%, the system deletes the earliest stored messages to ensure service availability.
Compact: the log compaction policy that is used in Apache Kafka. Log compaction ensures that the latest values are retained for messages that have the same key. This policy is suitable for scenarios such as restoring a failed system or reloading the cache after a system restarts. For example, when you use Kafka Connect or Confluent Schema Registry, you must store the information about the system status and configurations in a log-compacted topic.
ImportantYou can use log-compacted topics only in specific cloud-native components such as Kafka Connect and Confluent Schema Registry. For more information, see aliware-kafka-demos.
normal message
By default, messages of the same key are stored in the same partition in the order in which they are sent. If broker restarts or failures occur in a cluster, a small number of messages in the cluster may be out of order. For more information, see Comparison between storage engines.
partitionally ordered message
By default, messages of the same key are stored in the same partition in the order in which they are sent. If broker failures occur in a cluster, messages in the cluster are still in order. However, messages in specific partitions cannot be sent until the failed brokers are restored. For more information, see Comparison between storage engines.
connector
An ApsaraMQ for Kafka component that synchronizes data between ApsaraMQ for Kafka and other Alibaba Cloud services. For more information, see Overview.
tag
In ApsaraMQ for Kafka, a tag is used to identify resources. You can use tags to classify ApsaraMQ for Kafka resources based on resource features for easy resource search and aggregation. For more information, see Manage tags.
RAM
A service provided by Alibaba Cloud to manage user identities and resource access permissions. Resource Access Management (RAM) is applicable only to scenarios in which you want to implement access control for users who send and receive messages in the ApsaraMQ for Kafka console or by calling API operations. For more information, see Overview.
ACL
A service provided by ApsaraMQ for Kafka to manage the permissions of Simple Authentication and Security Layer (SASL) users and clients to send and receive messages by using SDKs. This feature is consistent with that of open source Apache Kafka. The access control list (ACL) feature is applicable only to scenarios in which you want to implement access control for users who use SDKs to send and receive messages. For more information, see Overview.