ApsaraMQ for Kafka:Use ApsaraMQ for Kafka as a Logstash input in a VPC

Step 1: Get the endpoint

Logstash connects to ApsaraMQ for Kafka through a VPC endpoint. ApsaraMQ for Kafka provides two types of VPC endpoints:

For more information about endpoint differences, see Comparison among endpoints.

To get the endpoint:

1. Log on to the ApsaraMQ for Kafka console.

2. In the Resource Distribution section of the Overview page, select the region where your instance is deployed.

3. On the Instances page, click the name of the target instance.

4. On the Instance Details page, find the endpoint in the Endpoint Information section. If you plan to use the SASL endpoint, note the Username and Password values in the Configuration Information section.

Step 2: Create a topic

Create a topic to hold the messages that Logstash will consume.

1. Log on to the ApsaraMQ for Kafka console.

2. In the Resource Distribution section of the Overview page, select the region where your instance is deployed.

3. On the Instances page, click the name of the target instance.

4. In the left-side navigation pane, click Topics.

5. On the Topics page, click Create Topic.

6. In the Create Topic panel, configure the following parameters and click OK.

After the topic is created, it appears on the Topics page.

Step 3: Send test messages

Send a test message to verify the topic is ready for consumption.

1. Log on to the ApsaraMQ for Kafka console.

2. Navigate to the target instance and click Topics in the left-side navigation pane.

3. Click the topic name, then click Send Message in the upper-right corner of the Topic Details page.

4. In the Start to Send and Consume Message panel, select a Sending Method (Console, Docker, or SDK) and send a test message. If you want to send the message to a specific partition, you can specify the partition ID. For information about how to query partition IDs, see View partition status.

Step 4: Create a consumer group

Create a consumer group for Logstash to track its consumption offset.

1. Log on to the ApsaraMQ for Kafka console.

2. Navigate to the target instance and click Groups in the left-side navigation pane.

3. On the Groups page, click Create Group.

4. In the Create Group panel, enter a Group ID and Description, attach tags if needed, and click OK.

After the consumer group is created, it appears on the Groups page.

Step 5: Configure and run Logstash

1. Navigate to the Logstash installation directory.

2. Create a configuration file named input.conf in the bin directory:

input {
  kafka {
    bootstrap_servers => "<your-kafka-endpoint>"       # VPC endpoint, e.g., alikafka-pre-cn-zv****-1-vpc.alikafka.aliyuncs.com:9092
    group_id          => "<your-consumer-group-id>"     # Consumer group created in Step 4
    topics            => ["<your-topic-name>"]          # Topic created in Step 2
    consumer_threads  => 12                             # Match this to the number of topic partitions
    auto_offset_reset => "earliest"                     # Start from the earliest message
  }
}
output {
  stdout { codec => rubydebug }
}

1. Run Logstash:

./logstash -f input.conf

Verify message consumption

After Logstash starts, consumed messages print to stdout in Ruby debug format:

See also

• Kafka input plugin -- Full parameter reference for the Logstash Kafka input plugin.

• Comparison among endpoints -- Differences between ApsaraMQ for Kafka endpoint types.