:Configure data sources for data synchronization from Oracle

Limits

• You can configure the supplemental logging feature only in a primary Oracle database. Supplemental logging can be enabled for a primary or secondary database.
• A maximum of 500 GB incremental data can be synchronized from an Oracle instance in real time every day.
• Only the Oracle 10g, 11g, 12c non-CDB, 18c non-CDB, and 19c non-CDB databases are supported by real-time data synchronization nodes in DataWorks. The Oracle 12c CDB, 18c CDB, and 19c CDB databases are not supported. An Oracle database of Oracle 12c or a later version can serve as a CDB to host multiple PDBs.

Precautions

• For real-time data synchronization nodes of Data Integration, you can subscribe to online redo log files only in primary Oracle databases. You can subscribe to only archived log files in secondary Oracle databases. If you have a high requirement on the timeliness of your real-time data synchronization nodes, we recommend that you subscribe to real-time incremental change logs in primary Oracle databases. If you subscribe to archived log files in a secondary Oracle database, the minimum latency between the generation and the reception of archived logs is determined by the interval at which Oracle automatically switches between two archived log files. In this case, the timeliness of real-time data synchronization nodes cannot be ensured.
• We recommend that you retain the archived log files of your Oracle database for three days. When a large amount of data is written to an Oracle database, the speed of real-time data synchronization may be slower than that of log generation. This provides sufficient time for you to trace data when issues occur during data synchronization. You can analyze archive log files to troubleshoot issues and recover data.
• Real-time data synchronization nodes of Data Integration cannot perform truncate operations on tables without a primary key in an Oracle database. A real-time data synchronization node analyzes logs of a table without a primary key by performing a logminer operation. In this process, the real-time data synchronization node uses rowid to find rows. If truncate operations are performed, the rowid values in the table are changed. This causes an error in the running of real-time data synchronization nodes.
• When you run real-time data synchronization nodes on a DataWorks server with the configurations of 24 vCPUs and 192 GiB memory, the utilization of a single CPU core on an Oracle database server varies based on the status of logs in the database. If most logs are not update logs and the synchronization nodes can process about 30 to 50 thousand data records every second, the utilization of a single CPU core can be 25% to 35%. If most logs are update logs, the DataWorks server may encounter a performance bottleneck during real-time data synchronization, and the utilization of a single CPU core on the Oracle database server is only 1% to 5%.

Procedure

1. Configure a whitelist for an Oracle database.
   Add the CIDR block of the VPC where the exclusive resource group resides to the whitelist of the Oracle database.
   1. View and record the elastic IP address (EIP) and CIDR block of the exclusive resource group for Data Integration.
      1. Log on to the DataWorks console.
      2. In the left-side navigation pane, click Resource Groups.
      3. On the Exclusive Resource Groups tab, find the exclusive resource group for Data Integration and click View Information in the Actions column.
      4. In the Exclusive Resource Groups dialog box, view and record the values of the EIPAddress and CIDR Blocks parameters.
      5. On the Exclusive Resource Groups tab, find the exclusive resource group for Data Integration and click Network Settings in the Actions column.
      6. On the VPC Binding tab of the page that appears, view and record the CIDR block of the vSwitch with which the exclusive resource group for Data Integration is associated.
   2. Add the elastic IP address (EIP) of the exclusive resource group and the CIDR block to the whitelist of the Oracle database.
2. Create an account and grant the required permissions to the account.
   You must create an account to log on to the database. The account must have the required permissions on the Oracle database.
   1. Create an account.
      For more information, see Create an Oracle database account.
   2. Grant permissions to the account.
      You can execute the following statements to grant permissions to the account. Replace Account for data synchronization with the created account when you execute a specific statement.

      grant create session to 'Account for data synchronization';  // Authorize the synchronization account to access the database. 
      grant connect to 'Account for data synchronization';  // Authorize the synchronization account to connect to the database. 
      grant select on nls_database_parameters to 'Account for data synchronization';  // Authorize the synchronization account to query the settings of nls_database_parameters. 
      grant select on all_users to 'Account for data synchronization';  // Authorize the synchronization account to query all users in the database. 
      grant select on all_objects to 'Account for data synchronization';  // Authorize the synchronization account to query all objects in the database. 
      grant select on DBA_MVIEWS to 'Account for data synchronization';  // Authorize the synchronization account to check the materialized view of the database. 
      grant select on DBA_MVIEW_LOGS to 'Account for data synchronization';  // Authorize the synchronization account to view the materialized view logs of the database. 
      grant select on DBA_CONSTRAINTS to 'Account for data synchronization';  // Authorize the synchronization account to view the constraints on all tables of the database. 
      grant select on DBA_CONS_COLUMNS to 'Account for data synchronization';  // Authorize the synchronization account to view information about all columns under specified constraints on all tables of the database. 
      grant select on all_tab_cols to 'Account for data synchronization';  // Authorize the synchronization account to view information about columns in tables, views, and clusters of the database. 
      grant select on sys.obj$ to 'Account for data synchronization';  // Authorize the synchronization account to view objects in the database. sys.obj$ indicates an object table that is contained in the data dictionary table. The object table contains all objects. 
      grant select on SYS.COL$ to 'Account for data synchronization';  // Authorize the synchronization account to view definitions of columns in tables of the database. SYS.COL$ stores definitions of columns in tables. 
      grant select on sys.USER$ to 'Account for data synchronization';  // Authorize the synchronization account to view the system table of the database. sys.USER$ indicates a default user session service. 
      grant select on sys.cdef$ to 'Account for data synchronization';  // Authorize the synchronization account to view the system table of the database. 
      grant select on sys.con$ to 'Account for data synchronization';  // Authorize the synchronization account to view the constraints of the database. sys.con$ records the constraints. 
      grant select on all_indexes to 'Account for data synchronization';  // Authorize the synchronization account to view all indexes of the database. 
      grant select on v_$database to 'Account for data synchronization';  // Authorize the synchronization account to check the v_$database view of the database. 
      grant select on V_$ARCHIVE_DEST to 'Account for data synchronization';  // Authorize the synchronization account to check the V_$ARCHIVE_DEST view of the database. 
      grant select on v_$log to 'Account for data synchronization';  // Authorize the synchronization account to check the v_$log view of the database. v_$log displays log information about control files. 
      grant select on v_$logfile to 'Account for data synchronization';  // Authorize the synchronization account to check the v_$logfile view of the database. v_$logfile contains information about redo log files. 
      grant select on v_$archived_log to 'Account for data synchronization';  // Authorize the synchronization account to check the v$archived_log view of the database. v$archived_log contains information about archived logs. 
      grant select on V_$LOGMNR_CONTENTS to 'Account for data synchronization';  // Authorize the synchronization account to check the V_$LOGMNR_CONTENTS view of the database. 
      grant select on DUAL to 'Account for data synchronization';   // Authorize the synchronization account to view the DUAL table of the database. DUAL is a virtual table that contains SELECT syntax rules. In Oracle, only one record is retained in the DUAL table. 
      grant select on v_$parameter to 'Account for data synchronization';  // Authorize the synchronization account to check the v_$parameter view of the database. v$parameter is a dynamic dictionary table that stores the values of parameters in the database. 
      grant select any transaction to 'Account for data synchronization';  // Authorize the synchronization account to view transactions of the database. 
      grant execute on SYS.DBMS_LOGMNR to 'Account for data synchronization';  // Authorize the synchronization account to use the LOGMNR tool. The LOGMNR tool helps you analyze transactions and retrieve lost data. 
      grant alter session to 'Account for data synchronization';  // Authorize the synchronization account to modify connection configurations of the database. 
      grant select on dba_objects to 'Account for data synchronization';  // Authorize the synchronization account to view all objects of the database. 
      grant select on v_$standby_log to 'Account for data synchronization';  // Authorize the synchronization account to check the v_$standby_log view of the database. v_$standby_log contains archived logs of the secondary database. 
      grant select on v_$ARCHIVE_GAP to 'Account for data synchronization';  // Authorize the synchronization account to query missing archived logs. 

      To synchronize full offline data, you must also execute the following statement to grant the query permission on all tables to the synchronization account:

      grant select any table to 'Account for data synchronization'; 

      In Oracle 12c and later, you must execute the following statement to grant the log mining permission to the synchronization account. The log mining feature is built in Oracle versions earlier than 12c. You do not need to execute the statement in these versions.

      grant LOGMINING TO 'Account for data synchronization';

3. Enable the generation of archived log files and supplemental logging, and switch a redo log file.
   Log on to the primary database and perform the following steps:

   1. Execute the following SQL statements to enable the generation of archived log files:

      shutdown immediate;
      startup mount;
      alter database archivelog;
      alter database open;

   2. Enable supplemental logging.

      SQL statements:

      alter database add supplemental log data(primary key) columns; // Enable supplemental logging for primary key columns. 
      alter database add supplemental log data(unique) columns; // Enable supplemental logging for unique index columns. 

   3. Switch a redo log file.
      After you enable supplemental logging, you must execute the following statement multiple times to switch a redo log file. We recommend that you execute the following statement for five times:

      alter system switch logfile;

      Note This ensures that data can be written to the next log file after the current log file is full. Data about historical operations will not be lost. This facilitates data recovery.
4. Check character encoding formats of the database.
   Execute the following statement to check character encoding formats of the database:

   select * from v$nls_parameters where PARAMETER IN ('NLS_CHARACTERSET', 'NLS_NCHAR_CHARACTERSET');

   • v$nls_parameters stores values of parameters in the database.
   • NLS_CHARACTERSET indicates a database character set. NLS_NCHAR_CHARACTERSET indicates a national character set. These two sets are used to store data of the character type.
   If the database contains the character encoding formats that are not supported for data synchronization, change the formats before you synchronize data.
5. Check the data types of tables in the database.
   You can execute the SELECT statement to query the data types of tables in the database. Sample statement that is executed to query the data types of the 'tablename' table:

   select COLUMN_NAME,DATA_TYPE from all_tab_columns where TABLE_NAME='tablename'; 

   • COLUMN_NAME: the name of the column.
   • DATA_TYPE: the data type of the column.
   • all_tab_columns: the view that stores information about all columns in tables of the database.
   • TABLE_NAME: the name of the table for a query. When you execute the preceding statement, replace 'tablename' with the name of the table for a query.

   You can also execute the select * from 'tablename'; statement to query the information about the desired table and obtain data types.
   If a table contains one of these data types, remove the table from the real-time synchronization solution list or change the data type before you synchronize data.