Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog How to Synchronize data from an ApsaraDB RDS for MySQL instance to a Message Queue for Apache Kafka instance Using DTS

How to Synchronize data from an ApsaraDB RDS for MySQL instance to a Message Queue for Apache Kafka instance Using DTS

This topic describes how to synchronize data from an ApsaraDB RDS for MySQL instance to a Message Queue for Apache Kafka instance by using Data Transmission Service (DTS).

by Muhammad Risma, Channel Marketing Intern

This topic describes how to synchronize data from an ApsaraDB RDS for MySQL instance to a Message Queue for Apache Kafka instance by using Data Transmission Service (DTS).

Architecture

1

What is Message Queue for Apache Kafka?

Message Queue for Apache Kafka is a distributed message queue service provided by Alibaba Cloud. This service features high throughput and scalability. Message Queue for Apache Kafka is widely used in big data fields such as log collection, monitoring data aggregation, streaming data processing, and online and offline analytics. It has become an indispensable part of the big data ecosystem.

In the Apache Kafka Queuing system, messages are stored in queue mode. This allows messages in the queue to be ingested by one or more consumers, but one consumer can only consume each message at a time. As soon as the consumer reads the message, it is removed from the Apache Kafka Queue.

Message Queue for Apache Kafka Features:
-100% Compatible with Apache Kafka
-Fully Managed
-High Availability
-Data Security

Log Analysis:
2

Log Collection
AlibabaMQ for Apache Kafka capable performance makes it suitable for log collection because:

  1. It has no perceivable impact on the service.
  2. It provides real-time online analysis between systems such as Storm/Spark and offline storage warehouses like Hadoop/ODPS.

Website Tracking:
3

High Throughput and Data Integration
AlibabaMQ for Apache Kafka allows users to collect real-time website activity statistics (including browsing, searching, and other behaviors).
Publish/subscribe mode routes messages to different topics by data type.
Message streaming can be used in real-time monitoring or service analysis if subscribed to real-time message delivery or offline processing if loaded to offline data warehouses such as Hadoop and ODPS.

Data Hub:
4
Multipath Routing
Supports both real-time and batch processing: local data persistence and PageCache can be delivered to multiple computing engines at the same time without performance loss.

Precautions:
● The available storage space of the destination Message Queue for Apache Kafka instance is larger than the total size of data in the source ApsaraDB RDS for MySQL instance.

A) Create Resource Group for your ApsaraDB RDS MySQL and Alibaba Message Queue for Apache Kafka

  1. Log on to your Alibaba Cloud Console Home Account and then Log on to the Resource Management console.
    5
  2. On the left-side of the navigation pane, choose Resource Group then click Create Resource Management. You need to set the Resource Group Name and Display Name according to your liking. After that click OK.
    6Please wait until the resource group is created, this will take around 2 – 3 minutes.

7

B) Create VPC for your ApsaraDB RDS MySQL and AlibabaMQ for Apache Kafka

  1. Log on to the VPC Console
    8
  2. In the top navigation bar, select the region where you want to create the VPC and vSwitch.
    Note that the VPC and other cloud resources that you want to deploy in the VPC must belong to the same region9
  3. Set Your VPC Name and VPC CIDR Block10
  4. In the resource group section, make sure that you choose your already created Resource Group in step A. Next set The VSwitch Name, Zone, And VSwitch IPv4 CIDR Block, then click OK11

C) Create ApsaraDB RDS Instance for MySQL

  1. Log on to ApsaraDB RDS Console
    12
  2. In the left-side navigation pane Click Instance, make sure you are in the same region and also use the same resource group like the VPC created before.
    13
  3. Click Create Instance
    14
  4. Set Your Basic Configuration and then click Next: Instance Configuration
    15
  5. Make sure to use the VPC and VSwitch that we created before
    16
  6. Set to the Resource Group that we create, Set the Privilege Account Database Account and Password, then click Next: Confirm Order
    17
  7. Accept the Terms of Service by checking the checkbox shown below and click Pay Now18If you get an error like the image below, please make sure that your internet connection is stable19Or you can go back again to Basic Configuration and choose Manual Zone of Secondary Node20
  8. Now Go to the Confirm Order, Click Terms of Service and Click Pay Now
    21
  9. If you see the notification below, your ApsaraDB Instance for MySQL has been successfully created.
    22

D) Create ApsaraDB RDS Account and Database for MySQL

  1. Go to your ApsaraDB RDS Console and In the left-side navigation pane Click Instance
    23
  2. Click the Instance that you have already made in step C.
    24
  3. In the left-side navigation pane Click Account. You should see the Account Privilege that has been made before, now let’s create a Standard Account for our database that we will create later. To do that click Create Account
    25
  4. Set the Standard Account Name, Account Type, Permission(Read/write) and Password, after click OK. The account should be created.
    26
  5. Next we will make a Database for our ApsaraDB RDS. In the left-side navigation pane Click Databases, then Click Create Database
    27
  6. Set the Database Name, and use Standard Account that we made before for Authorized Account, then Click Create
    28

E) Create ApsaraDB RDS Table for MySQL
There are two ways to create an RDS table: by using DMS or using MySQL Workbench.

I. Creating Table Using DMS:
1) Click Log on Databases
29

2) Log On to your Database Instance Using your Instance ID, Database Account, and Database Account that you made before, and then click Login.
30

3) In the top section please choose the id for the ApsaraDB for MySQL that we created before in step C.
31

4) Create Table and Column by executing queries using this format:

CREATE TABLE name_of_table(
    name_of_column  type_of_cloumn,
    .
    .
    PRIMARY KEY (name_of_column)
)

Below is the example of the query32
5) Next, add data to your table by executing queries using this format:

INSERT INTO name_of_table(
    name_of_column, …
)
VALUES( values_column1, values_column2, …)

Execute the query.
33
6) You are all done. The table have been added to the database

II. Using MySQL Workbench:
1) Create Connection from your MySQL Workbench to your ApsaraDB RDS for MySQL
2) Go to ApsaraDB RDS Instance that we made before, In the left-side navigation pane Click Database Connection, then click Apply for Public Endpoint
34
3）There will be a popup message like below. Click OK
35
4） Check the public endpoint by going to the Database Connection tab. Next to the Public Endpoint section is your Public IP address. Copy and save it to your Note
36
5） In the left-side navigation pane Click Whitelist and SecGroup, and click Modify
37

6） Change your IP Addresses to 0.0.0.0/0 so your MySQL Workbench can connect to your ApsaraDB RDS Instance, then click OK
38
7） Go to your mysQL Workbench, add MySQL Connection, set Connection Name, Paste your ApsaraDB RDS for MySQL Public Endpoint to Hostname, Input your Standard Account name as Username and Password, and Click OK
39
Now you can create and insert data to your Table

F) Create AlibabaMQ for Apache Kafka

  1. Go to Message Queue for Apache Kafka Console
    40
  2. In the left-side navigation pane click Instance, make sure you are using the same Region and Resource Group that we use before for VPC and RDS, click Buy Instance
    41
  3. Select your Billing Method
    42
  4. Set your configuration, remember to use the same Region and Resource Group that we use before, then Click Buy Now
    43
  5. Accept the Terms of Service and Click Activate Now
    44
  6. Go to your Message Queue for Apache Kafka Instance, and Click Deploy
    45
  7. Use VPC and VSwitch that we created before
    IMPORTANT: Please Use Version 0.10.2 as the Synchronization ApsaraDB to Message Queue for Apache Kafka only supports kafka version up to Version 1.0. then click OK46
  8. Wait for your Message Queue for Apache Kafka to be deployed, this may take up to 10-30 minutes.47
  9. After the instance is already in the Running state, go the left-side navigation pane click Topics, and click Create Topic, then configure your Topic, Click OK48In the left-side navigation pane click Instance Details, in Basic Information49Scroll Down go to Endpoint Information, Click Domain Name, and Copy selected IP Address to your Note50

G) Synchronize data from an ApsaraDB RDS for MySQL instance to Message Queue for Apache Kafka using DTS
1) Go to Data Transmission Service Console
2) Make sure your Region and Resource Group are Correct, In the left-side navigation pane click Data Synchronization, then Click Create Task
51

3) Input Your ApsaraDB for MySQL data to Source Database, make sure your Database Type, RDS Instance ID, Instance Region, Database Account And Password are correct
52

4) For Destination Databases,
Database Type : Kafka
Access Method : Express Connect, VPN Gateway, or Smart Access Gateway
Region : The same region as VPC
IP Address : Paste Message Queue for Apache Kafka IP Address that we copy in step F
Port Number : 9092
Kafka Version : 0.10
Topic : Select Topic that have been created before
53
5) Click Test Connectivity and Proceed54
6) Configure Objects and Advanced Setting55
7) Click Advanced Settings56
8) Set your Advanced Settings, then click Next: Save Task Settings and Precheck57
9) Wait for Precheck Completed, then Click Next: Purchased Instance58
10) Purchase your Instance, then click Data Transmission Service (Pay-as-you-go) Service Terms click Buy and Start59
11) Click OK60
12) Now Go to Data Synchronization, and Click Purchased Instance to see your Running Task61

ApsaraDB Data Transmission Service Message Queue ApsaraDB RDS for MySQL ApsaraDB RDS Apache Kafka
2 2 1
Share on

Read previous post:

How to Send Emails Using DirectMail on Alibaba Cloud

Read next post:

AIRec Deployment on Alibaba Cloud: Step-by-Step Instructions and Troubleshooting

Alibaba Cloud Indonesia

100 posts | 17 followers

You may also like

Comments

Kidd Ip January 21, 2023 at 2:39 am

Thank you very much for sharing on Kafka, very useful on message queue case handling

Dikky Ryan Pratama May 15, 2023 at 8:44 am

Terimakasih

Alibaba Cloud Indonesia

100 posts | 17 followers

Related Products

More Posts by Alibaba Cloud Indonesia

See All