All Products
Search

This Product

    Resource Orchestration Service:Deploy Apache Kafka on a single instance

    Document Center

    Resource Orchestration Service:Deploy Apache Kafka on a single instance

    Last Updated:Dec 20, 2024

    This topic describes how to deploy Apache Kafka on a single instance by creating a stack in the Resource Orchestration Service (ROS) console.

    Background information

    Apache Kafka is an open source stream processing platform that is written in the Scala and Java programming languages. Apache Kafka is a high-throughput distributed publish-subscribe messaging system for processing streaming data of customer behaviors on websites.

    You can use the Installs Kafka on an ECS instance (existing VPC) sample template to create an Elastic Compute Service (ECS) instance based on existing resources and associate an elastic IP address (EIP) with the instance. The existing resources include a virtual private cloud (VPC), vSwitch, and security group.

    • The following software versions are used in the sample template:

      • JDK 1.8.0: Java Development Kit (JDK)

      • Scala 2.12: the programming language

      • Apache Kafka 0.10.2.2: the compute engine

    • The default storage directory of Apache Kafka data in the data disk is /home/software/. The default storage directory of Apache Kafka binaries is /home/software/kafka/bin.

    • After you create a stack, you can obtain the value of KafkaManagerUrl and log on to the Apache Kafka management console.

    Important

    If you want to access the URL specified by KafkaManagerUrl over the Internet, you must configure an inbound rule for the security group to allow traffic on port 9000. For more information, see Add a security group rule.

    Step 1: Create a stack

    1. Log on to the ROS console.

    2. In the left-side navigation pane, choose Templates > Sample Templates.

    3. Search for the Installs Kafka on an ECS instance (existing VPC) sample template.

      image

    4. Click Create Stack.

    5. In the Configure Parameters step, configure Stack Name and the following parameters.

      Parameter

      Description

      Example

      VSwitch Zone ID

      The zone ID of the vSwitch in the VPC.

      Qingdao Zone C

      Existing VPC Instance ID

      The VPC ID of the instance.

      For more information about how to create and query a VPC, see Create and manage a VPC.

      vpc-bp1m6fww66xbntjyc****

      VSwitch ID

      The ID of the vSwitch that resides in the VPC.

      For more information about how to create and query a vSwitch, see Create and manage a vSwitch.

      vsw-bp183p93qs667muql****

      Business Security Group ID

      The security group ID of the ECS instance.

      For more information about how to query the ID of a security group, see Search for security groups.

      sg-bp15ed6xe1yxeycg7o****

      Instance Type

      The instance type of the ECS instance.

      Select a valid instance type. For more information, see Overview of instance families.

      ecs.c5.large

      Image ID

      The image ID of the ECS instance.

      For more information, see Find an image

      centos_7

      Instance Password

      The password of the ECS instance.

      Test_12****

      Public IP Bandwidth

      The bandwidth of the public IP address.

      Unit: Mbit/s.

      Valid values: 1 to 100.

      5

      Disk Type

      The disk category. Valid values:

      • cloud_efficiency: ultra disk

      • cloud_ssd: standard SSD

      • cloud_essd: Enterprise SSD (ESSD)

      • cloud: basic disk

      • ephemeral_ssd: local SSD

      For more information, see Disks.

      cloud_efficiency

      Data Disk Space

      The data disk size of the ECS instance.

      Valid values: 20 to 32786.

      Unit: GB.

      20

      Kafka Listeners Port

      The listener port. The port number must be greater than 1000.

      Default value: 9092.

      9092

      Message Retention Time

      The maximum retention period of a message.

      Default value: 24.

      Unit: hour.

      24

      Topic Specifications

      The number of topics.

      Default value: 50.

      50

    6. Click Create.

    7. On the Stack Information tab, view the stack status. Wait until the stack is created. Then, click the Outputs tab to obtain the value of KafkaManagerUrl.

    8. Access the URL specified by KafkaManagerUrl and log on to the Kafka management console.

      image

    Step 2: View resources

    1. In the left-side navigation pane, click Deployment > Stacks.

    2. On the Stacks page, click the ID of the desired stack.

    3. Click the Resources tab to view the information about resources in the stack.

      The following table describes the resource in this example.

      Resource

      Quantity

      Resource description

      Specification description

      ALIYUN::ECS::Instance

      1

      Creates an ECS instance to deploy Apache Kafka on the instance.

      • An ECS instance that has the following specifications is created:

      • Instance type: ecs.c5.large.

      • Disk category: ultra disk.

      • System disk size: 40 GB.

      • Public IP address: A public IP address is allocated.

      Note

      For more information about the pricing details of resources, go to the relevant console or refer to the pricing documentation of each resource.