All Products
Search

This Product

    ApsaraMQ for Kafka:Manage a connector

    Document Center

    ApsaraMQ for Kafka:Manage a connector

    Last Updated:Dec 11, 2024

    After a connector is created, you can manage the connector in the ApsaraMQ for Kafka console. You can view the configurations and log data of a connector, suspend or resume a connector, modify the configurations of a connector, test a connector, view consumption details for a connector, and delete a connector.

    Prerequisites

    A connector is created. For information about how to create a connector, see the following topics:

    Note

    You can modify connector configurations only for Function Compute sink connectors and MySQL source connectors.

    Precautions

    • You can modify connector configurations only for Function Compute sink connectors and MySQL source connectors.

    • You can suspend a connector only if the connector is in the Running state. You can enable a connector only if the connector is in the Suspended state.

    View the configurations of a connector

    After you create a connector, you can view the information about the connector, such as the basic information, source information, and destination information.

    1. Log on to the ApsaraMQ for Kafka console.

    2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

    3. In the left-side navigation pane, click Connectors.

    4. On the Connectors page, select an instance to which your connector belongs from the Select Instance drop-down list. In the connector list, find the connector and click Details in the Actions column.

      On the page that appears, view the details of the connector in the Basic Information, Source Information, and Destination Information sections.

    View log information about a connector

    If an error occurs in a connector, you can view the log information about the connector in the ApsaraMQ for Kafka console to troubleshoot the error.

    1. Log on to the ApsaraMQ for Kafka console.

    2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

    3. In the left-side navigation pane, click Connectors.

    4. On the Connectors page, select the instance to which your connector belongs from the Select Instance drop-down list. In the connector list, find the connector and choose More > View Logs in the Actions column.

      You can also click the name of the connector or click Details in the Actions column. On the details page that appears, click View Logs in the upper-right corner.

      Note

      The position of View Logs in the Actions column varies based on the status of the connector.

      In the dialog box that appears, view the log information about the connector.

    Modify the configurations of a connector

    After a Function Compute sink connector or a MySQL source connector is created, you can modify the configurations of the connector in the ApsaraMQ for Kafka console.

    1. Log on to the ApsaraMQ for Kafka console.

    2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

    3. In the left-side navigation pane, click Connectors.

    4. On the Connectors page, select the instance to which your connector belongs from the Select Instance drop-down list. In the connector list, find the connector and choose More > Modify Configuration in the Actions column.

      You can also click the name of the connector or click Details in the Actions column. On the details page that appears, click Modify Configuration in the upper-right corner.

    5. In the Modify Configuration panel, change the values of the parameters that are described in the following table based on your business requirements. Then click OK.

      • Modify the configurations of a Function Compute sink connector.

        Parameter

        Description

        Consumer Thread Concurrency

        The number of concurrent consumer threads that are used to synchronize data from the source topic. Default value: 6. Valid values:

        • 1

        • 2

        • 3

        • 6

        • 12

        Failure Handling Policy

        Specifies whether to retain the subscription to the partition in which an error occurs after the relevant message fails to be sent. Valid values:

        • Continue Subscription: retains the subscription to the partition in which an error occurs and generates an error log entry.

        • Stop Subscription: stops the subscription to the partition where an error occurs and generates an error log entry.

        Note

        • For information about how to view log information, see Manage a connector.

        • For information about how to troubleshoot errors based on error codes, see Error codes.

        Transmission Mode

        The mode in which messages are sent. Valid values:

        • Synchronous: The producer and the consumer must be running to get ready for data exchanges.

        • Asynchronous: After the producer sends a message, the producer does not need to wait for a response from the broker and can continue processing other tasks.

        Data Size

        The maximum number of messages that can be sent at a time.

        Valid values: 1 to 5000. Default value: 20.

        Retries

        The maximum number of retries allowed after a message fails to be sent. Default value: 2. Valid values: 1 to 3. In specific cases, when a message fails to be sent, retries are not supported. For more information about error codes, see Error codes. The following content describes whether retries are supported when specific error codes are returned:

        • For 4xx, retries are not supported except in the case when 429 is returned.

        • For 5xx, retries are supported.

        Note

        The connector calls the InvokeFunction operation to send messages to Function Compute.

      • Modify the configurations of a MySQL source connector.

        Parameter

        Description

        Database Table

        The name of one or more ApsaraDB RDS for MySQL tables from which you want to synchronize data. Separate multiple table names with commas (,).

        Source tables and destination topics are related based on one-to-one mappings.

        Note

        If you add or modify a table, you must create the corresponding topic in the ApsaraMQ for Kafka console. The topic name must be in the <Topic prefix>_<Table name> format. Otherwise, an error occurs when you run the connector. For information about how to create a topic, see Step 1: Create a topic.

      After the configurations are modified, go to the Connectors page and find the connector. Click Details in the Actions column. On the Connector Details page, you can view the updated configurations.

    Suspend or enable a connector

    If you need to temporarily suspend a running connector or resume a suspended connector, you can suspend or enable the connector in the ApsaraMQ for Kafka console.

    1. Log on to the ApsaraMQ for Kafka console.

    2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

    3. In the left-side navigation pane, click Connectors.

    4. On the Connectors page, select the instance to which your connector belongs from the Select Instance drop-down list. In the connector list, find the connector and suspend or enable the connector.

      • Suspend the connector: In the Actions column, choose More > Suspend. In the Note message, click OK.

      • Enable the connector: In the Actions column, click Enable. In the Note message, click OK.

      You can also click the name of the connector or click Details in the Actions column. On the details page that appears, click Suspend or Enable in the upper-right corner.

      On the Connectors page, find the connector you suspended or resumed. The value of the Status parameter for the connector becomes Suspended or Running.

      Note

      If you attempt to resume a MySQL source connector that remains suspended for a long period of time, the operation may fail. In this case, you can click Task Configurations in the Actions column. On the DataWorks page that appears, find the corresponding connector, and click Start. In the Start dialog box, select Reset Offset, configure the Start Time Point parameter, and then click OK.

    Test a connector

    After you create a connector, you can send a message to the connector from the ApsaraMQ for Kafka console to test the connector.

    1. Log on to the ApsaraMQ for Kafka console.

    2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

    3. In the left-side navigation pane, click Connectors.

    4. On the Connectors page, select the instance to which your connector belongs from the Select Instance drop-down list. In the connector list, find the connector and click Test in the Actions column.

      You can also click the name of the connector or click Details in the Actions column. On the details page that appears, click Test in the upper-right corner.

      Note

      The position of Test in the Actions column varies based on the status of the connector.

    5. In the Send Message panel, send a test message.

      1. In the Message Key field, enter the key of the test message, such as demo.

      2. In the Message Content field, enter the content of the test message, such as {"key": "test"}.

      3. Configure the Send to Specified Partition parameter to specify whether to send the test message to a specific partition.

        • If you want to send the test message to a specific partition, click Yes and enter the partition ID, such as 0, in the Partition ID field. For information about how to query partition IDs, see View partition status.

        • If you do not want to send the test message to a specific partition, click No.

    View the consumption progress for a connector

    After a connector is created and enters the Running state, you can view the consumption details of the consumer group that corresponds to the connector.Group The name of the consumer group is connect-{Connector name}.Group

    1. Log on to the ApsaraMQ for Kafka console.

    2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

    3. In the left-side navigation pane, click Connectors.

    4. On the Connectors page, select the instance to which your connector belongs from the Select Instance drop-down list. In the connector list, find the connector and click Consumption Progress in the Actions column.

      You can also click the name of the connector or click Details in the Actions column. On the details page that appears, click Consumption Progress in the upper-right corner.

      In the panel that appears, you can view the consumption details of the consumer group that corresponds to the connector in the source topic.

    Delete a connector

    ApsaraMQ for Kafka limits the number of connectors for each instance. If you no longer need a connector, you can delete it in the ApsaraMQ for Kafka console.

    Important

    • If a MySQL source connector is in the Running state, you cannot directly delete the connector in the ApsaraMQ for Kafka console. You must log on to the DataWorks console and stop and unpublish the connector, and then submit a ticket to clear the metadata of the ApsaraMQ for Kafka connector. You can directly delete Function Compute sink connectors, MaxCompute sink connectors, Object Storage Service (OSS) sink connectors, and Elasticsearch sink connectors in the ApsaraMQ for Kafka console.

    • When you delete a connector, Message Queue for Apache Kafka deletes the five topics and two consumer groups that are required by the connector, regardless of whether the topics and consumer groups were automatically or manually created.Group

    1. Log on to the ApsaraMQ for Kafka console.

    2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

    3. In the left-side navigation pane, click Connectors.

    4. On the Connectors page, select the instance to which your connector belongs from the Select Instance drop-down list. In the connector list, find the connector and choose More > Delete in the Actions column.

      You can also click the name of the connector or click Details in the Actions column. On the details page that appears, click Delete in the upper-right corner.

    5. In the Note message, click OK to delete the connector.