DataHub:Overview

DataHub data synchronization feature

The DataHub data synchronization feature provides data source and sink capabilities. It lets you synchronize data from a topic to other Alibaba Cloud products in real-time/near real-time to facilitate data exchange between services. Currently, supported services include MaxCompute (formerly ODPS), AnalyticDB for MySQL (ADS), ApsaraDB RDS, Tablestore, Object Storage Service (OSS), Elasticsearch, and Function Compute. You can write data to DataHub once and configure data synchronization to use the data across all supported cloud services. This creates an end-to-end data loop.

Alibaba Cloud services supported by DataHub

Note:

• For MaxCompute: Syncs occur every 5 minutes or when the batch size reaches 64 MB.

• For Tablestore, OSS, Elasticsearch, Function Compute, and Hologres: Syncs occur every 20 seconds or when the batch size reaches 4 MB.

• For MySQL, RDS, and ADS 3.0: Syncs occur every 20 seconds or when the batch size reaches 512 KB.

Usage notes

1. Whenever possible, synchronize data between Alibaba Cloud services within the same region to avoid cross-region network connectivity issues and reduce network latency.

2. Because of Alibaba Cloud’s network isolation, use only the internal endpoint (classic network) of your target service when configuring the endpoint. Public endpoints are not supported.

3. DataHub data synchronization uses at-least-once semantics. In rare cases—such as network or service failures—duplicate records may appear in the destination. Deduplicate data as needed.

4. When you create a sync task, DataHub automatically creates a subscription ID to track the read offset from the DataHub topic. Do not modify this subscription ID.

5. Only the owner or creator of a DataHub topic can create connectors (sync tasks).

6. You can create up to two sync tasks per sync type.

IP whitelist

Service whitelist

MaxCompute supports project-level IP whitelists. If you enable an IP whitelist, DataHub cannot synchronize data to MaxCompute. To resolve this, MaxCompute provides a service whitelist. Add the DataHub service name to the MaxCompute service whitelist to allow successful data synchronization.

The current DataHub service name is DataHub. You can configure it using the SDK or odpscmd.

The following example shows how to configure the SDK:

public static void setServiceName() throws OdpsException {
    String endpoint = "";
    String project = "";
    String accessId = "";
    String accessKey = "";

    Account account = new AliyunAccount(accessId, accessKey);
    Odps odps = new Odps(account);
    odps.setEndpoint(endpoint);
    odps.setDefaultProject(project);

    Map<String, String> prop = new HashMap<>();
    // Add the DataHub service name to the MaxCompute service whitelist.
    // Separate multiple names with commas. Each update overwrites previous settings.
    prop.put("odps.security.ip.whitelist.services", "DataHub");
    Group group = new Groups(odps).get("groupName");
    group.updateProject(project, null, null, prop);

    // Verify the current service whitelist setting.
    prop = odps.projects().get(project).getAllProperties();
    System.out.println("current service list: " + prop.get("odps.security.ip.whitelist.services"));
}

For odpscmd settings, see the following example.

Start odpscmd, then enter the following command.

setproject odps.security.ip.whitelist.services=DataHub;

After the configuration is complete, you can verify its success using the following method.

setproject;
## If the output includes odps.security.ip.whitelist.services=DataHub, the setting succeeded.