Dataphin:Configure the GaussDB (DWS) input component

Prerequisites

• A GaussDB (DWS) data source is created. For more information, see Create a GaussDB (DWS) data source.

• The account used to configure the GaussDB (DWS) input component has the sync read permission on the data source. If the account does not have this permission, you must request it. For more information, see Request permissions on a data source.

Procedure

1. On the top menu bar of the Dataphin homepage, select Develop > Data Integration.

2. In the top menu bar of the integration page, select a Project. For projects in Dev-Prod mode, also select an Environment.

3. In the navigation pane on the left, click Batch Pipeline. In the Batch Pipeline list, click the offline pipeline that you want to develop to open its configuration page.

4. Click Component Library in the upper-right corner of the page to open the Component Library panel.

5. In the navigation pane on the left of the Component Library panel, select Input, find the GaussDB (DWS) component in the list of input components on the right, and drag it to the canvas.

6. Click the icon on the GaussDB (DWS) input component card to open the GaussDB (DWS) Input Configuration dialog box.

7. In the GaussDB (DWS) Input Configuration dialog box, configure the following parameters.

   Parameter	Description
   Step Name	The name of the GaussDB (DWS) input component. Dataphin automatically generates a step name, which you can change as needed. The naming conventions are as follows: Can contain only Chinese characters, letters, underscores (_), and digits. Cannot exceed 64 characters in length.
   Data Source	The drop-down list displays all GaussDB (DWS) data sources and project levels in the current Dataphin instance, including those for which you do or do not have sync read permission. Click the icon to copy the current data source name. For a data source that does not have read-through permission, you can click Request to request read-through permission. For more information, see Request data source permissions. If you do not have a GaussDB (DWS) data source, click Create Data Source to create a data source. For more information, see Create a GaussDB (DWS) data source.
   Schema (Optional)	Select tables across different schemas. Select the schema where the table resides. If a schema is not specified, the schema configured in the data source is used by default.
   Source Table Count	Select the number of source tables. The options include Single Table and Multiple Tables: Single Table: This option is for synchronizing business data from a single table to a single target table. Multiple Tables: Suitable for scenarios where you synchronize business data from multiple source tables to a single target table. The union algorithm is used when data from multiple tables is written to a single data table.
   Table	You can enter a table name keyword to search, or enter the exact table name and click Exact Match. After you select a table, the system automatically checks its status. Click the icon to copy the name of the currently selected table. Select the source table or tables: If you select single table for Source Tables, you can enter a table name keyword to search, or enter the exact table name and click Exact Search. After you select a table, the system will automatically check its status. Click the icon to copy the name of the currently selected table. If Multiple tables is selected for Number of source tables, add tables as follows: In the input box, enter an expression to filter for tables with the same structure. The system supports enumerations, regex-like expressions, and a mix of both. For example, table_[001-100];table_102. Click Find Matches, and in the Confirm Match Details dialog box, view the list of matching tables. Click Confirm.
   Split Key (Optional)	The system partitions data based on the configured split key field. This can be used with the concurrency setting to enable concurrent reads. You can use a column from the source table as the split key. For better transfer performance, use a primary key or an indexed column as the split key. Important If you select a date/time type, the system identifies the minimum and maximum values. It then performs a forced split based on the total time range and concurrency. The splits are not guaranteed to be even.
   Batch Read Size (Optional)	The number of records to read at one time. When reading from the source database, you can configure a specific batch size, such as 1024 records. Reading in batches instead of one record at a time reduces the number of interactions with the data source, improves I/O efficiency, and lowers network latency.
   Input Filter (Optional)	The filter conditions for data extraction. The configuration is described as follows: Configure a static value to extract corresponding data. For example, ds=20210101. Configure a variable parameter to extract a subset of data. For example, ds=${bizdate}.
   Output Fields	The output fields area displays all fields from the selected tables that match the filter conditions. If certain fields are not needed by downstream components, delete them: Note When the tenant's compute engine is GaussDB (DWS), you can view the data classification and level for the output fields of the GaussDB (DWS) input component. This feature is not supported for other compute engines. To delete a single field: To delete a small number of fields, click the icon in the Actions column to remove unwanted fields. Batch deleting fields: To delete many fields, click Field Management. In the Field Management dialog box, select multiple fields, click the left-arrow icon to move the selected input fields to the unselected input fields, and then click OK.

8. Click Confirm to complete the configuration of the GaussDB (DWS) input component.