Create, delete, and query Kafka topics using Terraform - Terraform

A topic is the first-level identifier for messages in ApsaraMQ for Kafka. Before your application can produce or consume messages, create a topic. This guide covers the full lifecycle of managing topics with the alicloud_alikafka_topic Terraform resource: create, update, query, and delete.

Note

Run the complete sample code directly in your browser. Open Terraform Explorer to get started without any local setup.

Before you begin

Before you begin, ensure that you have:

An Alibaba Cloud account with an AccessKey pair for a Resource Access Management (RAM) user. Avoid using your Alibaba Cloud account credentials directly. For setup instructions, see Create a RAM user and Create an AccessKey pair

RAM permissions that follow the principle of least privilege. For instructions, see Grant permissions to RAM users. The following policy grants the minimum permissions required for this guide:

  {
      "Version": "1",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": [
                  "vpc:DescribeVpcAttribute",
                  "vpc:DescribeRouteTableList",
                  "vpc:DescribeVSwitchAttributes",
                  "vpc:DeleteVpc",
                  "vpc:DeleteVSwitch",
                  "vpc:CreateVpc",
                  "vpc:CreateVSwitch"
              ],
              "Resource": "*"
          },
          {
              "Effect": "Allow",
              "Action": "bss:ModifyAgreementRecord",
              "Resource": "*"
          },
          {
              "Effect": "Allow",
              "Action": [
                  "bss:DescribeOrderList",
                  "bss:DescribeOrderDetail",
                  "bss:PayOrder",
                  "bss:CancelOrder"
              ],
              "Resource": "*"
          },
          {
              "Effect": "Allow",
              "Action": [
                  "ecs:CreateSecurityGroup",
                  "ecs:ModifySecurityGroupPolicy",
                  "ecs:DescribeSecurityGroups",
                  "ecs:ListTagResources",
                  "ecs:DeleteSecurityGroup",
                  "ecs:DescribeSecurityGroupAttribute",
                  "ecs:AuthorizeSecurityGroup",
                  "ecs:RevokeSecurityGroup"
              ],
              "Resource": "*"
          },
          {
              "Action": "alikafka:*",
              "Resource": "*",
              "Effect": "Allow"
          },
          {
              "Action": "ram:CreateServiceLinkedRole",
              "Resource": "*",
              "Effect": "Allow",
              "Condition": {
                  "StringEquals": {
                      "ram:ServiceName": [
                          "connector.alikafka.aliyuncs.com",
                          "instanceencryption.alikafka.aliyuncs.com",
                          "alikafka.aliyuncs.com",
                          "etl.alikafka.aliyuncs.com"
                      ]
                  }
              }
          }
      ]
  }

A Terraform runtime environment set up through one of the following methods:

Method	Best for
Terraform Explorer	Quick experimentation at no cost, directly in the browser
Cloud Shell	Fast setup with preinstalled Terraform and preconfigured credentials
On-premises installation	Unstable network conditions or custom development environments

Required resources

Note

The resources used in this guide incur charges. Release or unsubscribe from them when they are no longer needed.

This guide uses the following Terraform resources:

Resource	Purpose
alicloud_vpc	Creates a virtual private cloud (VPC)
alicloud_vswitch	Creates a vSwitch in the VPC
alicloud_security_group	Creates a security group
alicloud_alikafka_instance	Creates an ApsaraMQ for Kafka instance
alicloud_alikafka_topic	Creates an ApsaraMQ for Kafka topic
alicloud_alikafka_topics	Queries ApsaraMQ for Kafka topics

Topic configuration options

Before writing your Terraform configuration, review the key topic parameters:

Parameter	Description	Default in this guide
`partition_num`	Number of partitions. More partitions allow higher throughput but consume more instance resources.	`12`
`local_topic`	Set to `true` to store messages on local disks for higher throughput. Set to `false` for cloud disk storage with higher reliability.	`false`
`compact_topic`	Set to `true` to enable log compaction, which retains only the latest value for each message key. Available only when `local_topic` is set to `true`.	`false`

Note

Topic names cannot be changed after creation.

Create a topic

This example creates a topic named example-topic in the China (Shenzhen) region.

Step 1: Set up the infrastructure

Create a working directory and add a file named main.tf with the following configuration. This block provisions the VPC, vSwitch, security group, and ApsaraMQ for Kafka instance that the topic requires.

variable "region" {
  default = "cn-shenzhen"
}

variable "instance_name" {
  default = "alikafkaInstanceName"
}

variable "zone_id" {
  default = "cn-shenzhen-f"
}

provider "alicloud" {
  region = var.region
}

# Create a VPC.
resource "alicloud_vpc" "default" {
  vpc_name   = "alicloud"
  cidr_block = "172.16.0.0/16"
}

# Create a vSwitch.
resource "alicloud_vswitch" "default" {
  vpc_id     = alicloud_vpc.default.id
  cidr_block = "172.16.192.0/20"
  zone_id    = var.zone_id
}

# Create a security group.
resource "alicloud_security_group" "default" {
  vpc_id = alicloud_vpc.default.id
}

# Create an ApsaraMQ for Kafka instance.
# Disk type: ultra disk (0), disk capacity: 500 GB,
# traffic specification: alikafka.hw.2xlarge.
resource "alicloud_alikafka_instance" "default" {
  name           = var.instance_name
  partition_num  = 50
  disk_type      = 0
  disk_size      = 500
  deploy_type    = 5
  io_max_spec    = "alikafka.hw.2xlarge"
  vswitch_id     = alicloud_vswitch.default.id
  security_group = alicloud_security_group.default.id
}

Step 2: Add the topic resource

Append the following configuration to main.tf. This defines the topic with 12 partitions, cloud disk storage, and no log compaction.

variable "partition_num" {
  default = "12"
}

variable "remark" {
  default = "kafka_topic_remark"
}

resource "alicloud_alikafka_topic" "default" {
  instance_id   = alicloud_alikafka_instance.default.id
  topic         = "example-topic"
  local_topic   = "false"
  compact_topic = "false"
  partition_num = var.partition_num
  remark        = var.remark
}

Step 3: Initialize, plan, and apply

Initialize the Terraform working directory: Expected output:

   terraform init

   Initializing the backend...

   Initializing provider plugins...
   - Checking for available provider plugins...
   - Downloading plugin for provider "alicloud" (hashicorp/alicloud) 1.90.1...
   ...

   You may now begin working with Terraform. Try running "terraform plan" to see
   any changes that are required for your infrastructure. All Terraform commands
   should now work.

   If you ever set or change modules or backend configuration for Terraform,
   rerun this command to reinitialize your working directory. If you forget, other

Preview the changes:
```
   terraform plan
```

Apply the configuration. Type yes when prompted and press Enter. Expected output:

   terraform apply

   ...
   Apply complete!  Resources: 5 added, 0 changed, 0 destroyed.

Verify the topic by using either of the following methods.

Run the terraform show command

Run the terraform show command to view information about the Kafka topic.

terraform show

# alicloud_alikafka_topic.default:
resource "alicloud_alikafka_topic" "default" {
    compact_topic = false
    id            = "alikafka_post-cn-****:example-topic"
    instance_id   = "alikafka_post-cn-****"
    local_topic   = false
    partition_num = 12
    remark        = "kafka_topic_remark"
    topic         = "example-topic"
}

Log on to the ApsaraMQ for Kafka console

Log on to the ApsaraMQ for Kafka console and check the topic list for your instance.

Update a topic

To modify the partition count or description of an existing topic, update the variable values in main.tf and reapply. This example changes partition_num from 12 to 24 and updates the remark.

In main.tf, update the partition_num and remark variables:

   variable "partition_num" {
     default = "24"
   }

   variable "remark" {
     default = "updated_kafka_topic_remark"
   }

Preview the changes:
```
   terraform plan
```

Apply the changes. Type yes when prompted and press Enter. Terraform updates the topic in-place without recreating it:

   terraform apply

   alicloud_alikafka_topic.default: Modifying... [id=alikafka_post-cn-****:example-topic]
   alicloud_alikafka_topic.default: Modifications complete after 2s [id=alikafka_post-cn-****:example-topic]

   Apply complete!  Resources: 0 added, 1 changed, 0 destroyed.

Verify the result:

   terraform show

   # alicloud_alikafka_topic.default:
   resource "alicloud_alikafka_topic" "default" {
       compact_topic = false
       id            = "alikafka_post-cn-****:example-topic"
       instance_id   = "alikafka_post-cn-****"
       local_topic   = false
       partition_num = 24
       remark        = "updated_kafka_topic_remark"
       tags          = {}
       topic         = "example-topic"
   }

Query topics

Use the alicloud_alikafka_topics data source to list all topics in an instance.

Add the following data source to main.tf:

   data "alicloud_alikafka_topics" "topics_ds" {
     instance_id = alicloud_alikafka_instance.default.id
   }

Preview the changes:
```
   terraform plan
```
Apply the configuration. Type yes when prompted and press Enter.
```
   terraform apply
```

View the query results:

   terraform show

   # data.alicloud_alikafka_topics.topics_ds:
   data "alicloud_alikafka_topics" "topics_ds" {
       id          = "29****"
       ids         = [
           "alikafka_post-cn-****:example-topic",
       ]
       instance_id = "alikafka_post-cn-****"
       names       = [
           null,
       ]
       page_size   = 50
       topics      = [
           {
               compact_topic = false
               create_time   = "1732****"
               id            = "alikafka_post-cn-****:example-topic"
               instance_id   = "alikafka_post-cn-****"
               local_topic   = false
               partition_num = 24
               remark        = "updated_kafka_topic_remark"
               status        = 0
               status_name   = "Running"
               tags          = {}
               topic         = "example-topic"
           },
       ]
       total_count = 1
   }

Clean up resources

When the resources are no longer needed, delete them to stop incurring charges:

terraform destroy

Type yes when prompted and press Enter to confirm.

For more details about the terraform destroy command, see Common commands.

Complete sample code

Note

Run the code below directly in your browser. Open Terraform Explorer to get started.

Complete sample code

The following configuration provisions the infrastructure, creates a topic, and queries all topics in a single file:

variable "region" {
  default = "cn-shenzhen"
}

variable "instance_name" {
  default = "alikafkaInstanceName"
}

variable "zone_id" {
  default = "cn-shenzhen-f"
}

variable "partition_num" {
  default = "12"
}

variable "remark" {
  default = "kafka_topic_remark"
}

provider "alicloud" {
  region = var.region
}

# Create a VPC.
resource "alicloud_vpc" "default" {
  vpc_name   = "alicloud"
  cidr_block = "172.16.0.0/16"
}

# Create a vSwitch.
resource "alicloud_vswitch" "default" {
  vpc_id     = alicloud_vpc.default.id
  cidr_block = "172.16.192.0/20"
  zone_id    = var.zone_id
}

# Create a security group.
resource "alicloud_security_group" "default" {
  vpc_id = alicloud_vpc.default.id
}

# Create an ApsaraMQ for Kafka instance.
# Disk type: ultra disk (0), disk capacity: 500 GB,
# traffic specification: alikafka.hw.2xlarge.
resource "alicloud_alikafka_instance" "default" {
  name           = var.instance_name
  partition_num  = 50
  disk_type      = 0
  disk_size      = 500
  deploy_type    = 5
  io_max_spec    = "alikafka.hw.2xlarge"
  vswitch_id     = alicloud_vswitch.default.id
  security_group = alicloud_security_group.default.id
}

# Create an ApsaraMQ for Kafka topic.
resource "alicloud_alikafka_topic" "default" {
  instance_id   = alicloud_alikafka_instance.default.id
  topic         = "example-topic"
  local_topic   = "false"
  compact_topic = "false"
  partition_num = var.partition_num
  remark        = var.remark
}

# Query ApsaraMQ for Kafka topics.
data "alicloud_alikafka_topics" "topics_ds" {
  instance_id = alicloud_alikafka_instance.default.id
  depends_on = [alicloud_alikafka_topic.default]
}

References

Step 3: Create resources: Manage topics through the ApsaraMQ for Kafka console.
CreateTopic: Create topics through the API.
alicloud_alikafka_topic: Full argument and attribute reference for the Terraform resource.

、