Create a Kafka data source - ApsaraDB for OceanBase - Alibaba Cloud Documentation Center

Before you create a data migration or synchronization project, you must create the data sources. This topic describes how to create a Kafka data source by using the data transmission service.

Limitations

The data transmission service allows you to add a Kafka instance only as the destination for data synchronization.

Background information

Data security is a major concern throughout service connectivity verification, link creation, and data transmission. Based on the security system provided by the Kafka service, data transmission can meet most security requirements in data encryption and user authentication.

Data transmission supports the following Kafka authentication methods:

GSSAPI
Generic Security Services Application Program Interface (GSSAPI) is a framework that provides generic security services. It supports the Kerberos protocol.
PLAIN
PLAIN authentication is simple and does not support dynamic changes of users. In this authentication mode, usernames and passwords are configured in plaintext, resulting in low security.
SCRAM-SHA-256
Salted Challenge Response Authentication Mechanism (SCRAM) authenticates users by username and password. SCRAM-SHA-256 can be used together with Transport Layer Security (TLS) for security authentication.
In this authentication method, users can be dynamically changed, and the user data is stored in Zookeeper. Before a broker is started, a user for communication between the broker and Zookeeper must be created. However, usernames and passwords are configured in plaintext in this authentication method.
SCRAM-SHA-512
SCRAM-SHA-512 can be used together with TLS for security authentication.

Procedure

Log on to the ApsaraDB for OceanBase console.
In the left-side navigation pane, choose Data Transmission > Data Source Management.
On the Data Sources page, click Create Data Source in the upper-right corner.
In the Create Data Source dialog box, select Kafka for Data Source Type.

Select a value for Instance Type and configure the parameters.

If you select Alibaba Cloud Kafka Instance, configure the following parameters.

Parameter	Description
Data Source Identifier	The data source identifier. We recommend that you set it to a combination of digits and letters. It must not contain any spaces and cannot exceed 32 characters in length.
Cross Alibaba Cloud Account	The data transmission service allows you to configure a data migration or synchronization project between instances under different Alibaba Cloud accounts to migrate or synchronize data across Alibaba Cloud accounts. You can choose whether to select this option based on business needs. If you select this option, enter the Alibaba Cloud account of the destination instance. If you do not have permissions on this Alibaba Cloud account, request for authorization first. For more information, see Apply for cross-account authorization.
Kafka Instance ID	The unique ID of the Kafka instance that you applied for.
Access Point	The IP address and port list of the Kafka server. The system automatically imports this list.
Username	The logon username of the Kafka instance.
Password	The logon password of the Kafka instance.
Remarks	Optional. Additional information about the data source.

If you select Create Kafka Instance in VPC or Public Network Kafka Instance, configure the following parameters.

Parameter	Description
Data Source Identifier	The data source identifier. We recommend that you set it to a combination of digits and letters. It must not contain any spaces and cannot exceed 32 characters in length.
Cross Alibaba Cloud Account	The data transmission service allows you to configure a data migration or synchronization project between instances under different Alibaba Cloud accounts to migrate or synchronize data across Alibaba Cloud accounts. You can choose whether to select this option based on business needs. If you select this option, enter the Alibaba Cloud account of the destination instance. If you do not have permissions on this Alibaba Cloud account, request for authorization first. For more information, see Apply for cross-account authorization. Important This parameter is not displayed when Kafka Instance in Public Network is selected as the instance type.
VPC	Select the unique ID of the virtual private cloud (VPC) that you applied for from the drop-down list. Important This parameter is displayed only when you set the instance type to Create Kafka Instance in VPC.
Deployment in VPC/Cross-ISP Deployment	In cross-ISP deployment, the source and destination data sources are in different networks, including different VPCs or cloud service providers. Choose Deployment in VPC or Cross-ISP Deployment based on your business needs. Select all vSwitches to which all bootstrap server and `broker server` instances of the Kafka service belong from the `vSwitches` list. Add the CIDR blocks of the vSwitch to the security group whitelist of the current VPC. A vSwitch is a basic network module of a VPC and is used to connect cloud resource instances. For more information, see Overview. Important If the instance type is set to Self-Managed Kafka Instance in VPC, you can select the deployment mode and vSwitches. In the case of cross-ISP deployment, static routing addresses (addresses or CIDR blocks in VPCs on other clouds or on-premises IDCs) are automatically associated with the first selected vSwitch.
Access Point	The IP address and port list of the Kafka server.
Enable SSL	Choose whether to enable Secure Sockets Layer (SSL) based on the business needs. To enable SSL, click Upload File and upload an SSL certificate suffixed with `.jks`.
Enable Authentication	Choose whether to enable authentication based on business needs. Kafka provides data encryption and multiple identity authentication mechanisms to ensure the security of user data and services.
Authentication Method	If authentication is enabled, you must specify the authentication method. Data transmission supports the following authentication methods: GSS-API, PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512.
KDC Server Address	The IP address or domain name of the Kerberos Key Distribution Center (KDC) server. Notice: This parameter is displayed only when you set Authentication Method to GSS-API.
User Entity	Enter the username. Notice: This parameter is displayed only when you set Authentication Method to GSS-API.
Keytab File	Click Upload File and upload a key file suffixed with `.keytab`. Notice: This parameter is displayed only when you set Authentication Method to GSS-API.
Username	The username of the account used for data migration or synchronization. Notice: This parameter is not displayed when you set Authentication Method to GSSAPI.
Password	The password of the account used for data migration or synchronization. Notice: This parameter is not displayed when you set Authentication Method to GSSAPI.
Remarks	Optional. Additional information about the data source.

Click Test Connection to verify the network connection between the data transmission system and the data source, as well as the validity of the username and password.
After the test is passed, click OK.