Notes and limits for data synchronization when the source database is Redis - Data Transmission Service

If your source database is a Redis database, such as a self-managed Redis or Tair (Redis OSS-compatible), review the notes and limits in this topic before you configure a sync task. This ensures that the data synchronization task runs as expected.

Overview of sync solutions for a Redis source

Review the notes and limits for your sync task based on the following sync solutions:

Two-way synchronization between Redis instances
One-way synchronization between Redis instances

Two-way synchronization between Redis instances

Type	Description
Source database limits	To ensure sync quality, Data Transmission Service (DTS) inserts a key with the prefix DTS_REDIS_TIMESTAMP_HEARTBEAT into the source database to record update timestamps. If the source database uses a cluster architecture, DTS inserts this key into each shard. This key is filtered out during synchronization and expires after the sync task ends. If the source database is a read-only instance or the DTS account does not have write (SETEX) permissions, the reported latency may be inaccurate. Based on the Redis replication backlog mechanism, increase the value of the `repl-backlog-size` parameter in the `redis.conf` configuration file to ensure sync link stability. Do not run the `FLUSHDB` and `FLUSHALL` commands on the source database. Otherwise, data inconsistency between the source and destination databases will occur. If some keys in the source database use an expiration (expire) policy, the number of keys in the destination database (for example, viewed using the info command) might be less than in the source database. This can happen because some keys may have expired but have not been promptly deleted. Operational limits for synchronizing data from a Basic Edition Redis instance to a cluster architecture Redis instance: Because a cluster only allows a single command to operate on a single slot, if you run a multi-key operation on the source database where the keys are not in the same slot or involve multiple slots, the following error occurs: `CROSSSLOT Keys in request don't hash to the same slot` Run only single-key operations during DTS synchronization to avoid link interruptions. Two-way synchronization is supported only between Tair (Redis OSS-compatible) (Enterprise Edition) instances. Supported instances: Redis Open-Source Edition: Instances where the Deployment Mode is Classic, and the Version Type is Tair (Enterprise Edition). Tair (Enterprise Edition): All instances. If the source is a Tair Enterprise Edition instance whose Storage Medium is Persistent Memory, you must ensure that the appendonly parameter value is `yes`. For more information, see Procedure. If the source is a child instance of a Tair (Enterprise Edition) instance with the Global Distributed Cache feature, DTS can only sync data from the selected child instance. Data from other child instances cannot be synced due to the limitations of this feature.
Other limits	During synchronization, operations such as scaling (for example, adding or removing shards), instance upgrades or downgrades (for example, increasing memory, changing specifications or configurations, or migrating zones), or changing the endpoint and port number on the source or destination Redis instance will interrupt the sync task. This is because DTS cannot obtain continuous log data and correct connection information. If this occurs, delete the data that has been synced to the destination Redis instance before you reconfigure the task to ensure data consistency. When the source or destination instance is a self-managed Redis (where the Access Method is not a Alibaba Cloud Instance), if the Redis endpoint changes during synchronization (for example, due to instance migration, including scenarios such as master-standby switchover), the sync task may experience retries, delays, failures, or even data inconsistency. Please promptly check the status of the sync task. If the DTS task shows abnormalities such as retries, delays, or failures, please reconfigure the sync task. After an instance migration (including a failover) is triggered on the destination Redis instance, data might be written only to memory and not to the secondary database. This can cause data loss. Keep the source and destination database versions the same, or sync from an earlier version to a later version to ensure compatibility. If you sync from a later version to an earlier version, compatibility issues may occur. During initial full data synchronization, DTS consumes some read and write resources from the source and destination databases. This may increase the database load. Before you start data synchronization, evaluate the performance of your source and destination databases. We recommend that you perform data synchronization during off-peak hours. During DTS synchronization, do not write data to the destination database from any source other than DTS. Otherwise, data inconsistency between the source and destination databases will occur. If both the forward and remote sync tasks sync a specific table, and the forward sync task syncs both the full and incremental data of that table, the remote sync task syncs only the incremental data of that table. If the destination instance uses a cluster architecture and a shard reaches its memory limit, or if the destination instance runs out of storage space, the DTS task will fail due to an out of memory (OOM) error. If the destination database runs out of memory and triggers data eviction, data inconsistency may occur. This is because the default eviction policy (maxmemory-policy) for Tair (Redis OSS-compatible) is volatile-lru. The sync task continues to run, but the data becomes inconsistent. To prevent this, you can set the eviction policy of the destination database to noeviction. If the destination database runs out of memory, data writes fail and the task fails. This prevents data loss due to eviction. Note For more information about data eviction policies, see Introduction to Redis data eviction policies. If the Transparent Data Encryption (TDE) feature is enabled for the source or destination instance, you cannot use DTS to sync data. If any of the following situations occur during data synchronization, a full data resynchronization to the destination may be triggered, which can cause data inconsistency. A transient connection disconnection occurs on the source or destination Redis instance, causing resumable transmission to fail. A primary/secondary switchover or a failover occurs on the source or destination Redis instance. The endpoint of the source or destination Redis instance changes. The sync objects of the DTS instance are modified. If a Tair (Redis OSS-compatible) instance has the TLS (Transport Layer Security) encryption feature enabled, you must connect to DTS using an SSL-encrypted (TLSv1.3 is not supported). Connecting to a Tair (Redis OSS-compatible) instance with SSL enabled as a Alibaba Cloud Instance is not currently supported. If a sync instance includes both full and incremental tasks, restarting the instance may cause DTS to re-run both the full and incremental tasks. Restarting a sync instance may cause some commands to be executed repeatedly. This can lead to data inconsistency between the source and destination databases, especially if non-idempotent commands (such as `INCRBY` and `LPUSH`) exist or if a full task is re-run. If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters. Note Only DTS task parameters are modified—not database parameters. Parameters that may be adjusted include those listed in Modify instance parameters.

One-way synchronization between Redis instances

Type	Description
Source database limits	To ensure sync quality, Data Transmission Service (DTS) inserts a key with the prefix DTS_REDIS_TIMESTAMP_HEARTBEAT into the source database to record update timestamps. If the source database uses a cluster architecture, DTS inserts this key into each shard. This key is filtered out during synchronization and expires after the sync task ends. If the source database is a read-only instance or the DTS account does not have write (SETEX) permissions, the reported latency may be inaccurate. Based on the Redis replication backlog mechanism, increase the value of the `repl-backlog-size` parameter in the `redis.conf` configuration file to ensure sync link stability. If some keys in the source database use an expiration (expire) policy, the number of keys in the destination database (for example, viewed using the info command) might be less than in the source database. This can happen because some keys may have expired but have not been promptly deleted. Operational limits for synchronizing data from a Basic Edition Redis instance to a cluster architecture Redis instance: Because a cluster only allows a single command to operate on a single slot, if you run a multi-key operation on the source database where the keys are not in the same slot or involve multiple slots, the following error occurs: `CROSSSLOT Keys in request don't hash to the same slot` Run only single-key operations during DTS synchronization to avoid link interruptions. If the source is a Tair Enterprise Edition instance whose Storage Medium is Persistent Memory, you must ensure that the appendonly parameter value is `yes`. For more information, see Procedure. If the source is a child instance of a Tair (Enterprise Edition) instance with the Global Distributed Cache feature, DTS can only sync data from the selected child instance. Data from other child instances cannot be synced due to the limitations of this feature.
Other limits	If the source instance is a self-managed Redis, we recommend that you run the `config set repl-timeout 600` command to set the repl-timeout parameter, which specifies the replication timeout period between the slave and master nodes, to 600 seconds. If the source database contains a large volume of data, you can increase the value of the repl-timeout parameter as needed. During synchronization, operations such as scaling (for example, adding or removing shards), instance upgrades or downgrades (for example, increasing memory, changing specifications or configurations, or migrating zones), or changing the endpoint and port number on the source or destination Redis instance will interrupt the sync task. This is because DTS cannot obtain continuous log data and correct connection information. If this occurs, delete the data that has been synced to the destination Redis instance before you reconfigure the task to ensure data consistency. When the source or destination instance is a self-managed Redis (where the Access Method is not a Alibaba Cloud Instance), if the Redis endpoint changes during synchronization (for example, due to instance migration, including scenarios such as master-standby switchover), the sync task may experience retries, delays, failures, or even data inconsistency. Please promptly check the status of the sync task. If the DTS task shows abnormalities such as retries, delays, or failures, please reconfigure the sync task. After an instance migration (including a failover) is triggered on the destination Redis instance, data might be written only to memory and not to the secondary database. This can cause data loss. Keep the source and destination database versions the same, or sync from an earlier version to a later version to ensure compatibility. If you sync from a later version to an earlier version, compatibility issues may occur. During initial full data synchronization, DTS consumes some read and write resources from the source and destination databases. This may increase the database load. Before you start data synchronization, evaluate the performance of your source and destination databases. We recommend that you perform data synchronization during off-peak hours. During DTS synchronization, do not write data to the destination database from any source other than DTS. Otherwise, data inconsistency between the source and destination databases will occur. If the destination instance uses a cluster architecture and a shard reaches its memory limit, or if the destination instance runs out of storage space, the DTS task will fail due to an out of memory (OOM) error. If the destination database runs out of memory and triggers data eviction, data inconsistency may occur. This is because the default eviction policy (maxmemory-policy) for Tair (Redis OSS-compatible) is volatile-lru. The sync task continues to run, but the data becomes inconsistent. To prevent this, you can set the eviction policy of the destination database to noeviction. If the destination database runs out of memory, data writes fail and the task fails. This prevents data loss due to eviction. Note For more information about data eviction policies, see Introduction to Redis data eviction policies. If the Transparent Data Encryption (TDE) feature is enabled for the source or destination instance, you cannot use DTS to sync data. If any of the following situations occur during data synchronization, a full data resynchronization to the destination may be triggered, which can cause data inconsistency. A transient connection disconnection occurs on the source or destination Redis instance, causing resumable transmission to fail. A primary/secondary switchover or a failover occurs on the source or destination Redis instance. The endpoint of the source or destination Redis instance changes. The sync objects of the DTS instance are modified. If a Tair (Redis OSS-compatible) instance has the TLS (Transport Layer Security) encryption feature enabled, you must connect to DTS using an SSL-encrypted (TLSv1.3 is not supported). Connecting to a Tair (Redis OSS-compatible) instance with SSL enabled as a Alibaba Cloud Instance is not currently supported. If a sync instance includes both full and incremental tasks, restarting the instance may cause DTS to re-run both the full and incremental tasks. Restarting a sync instance may cause some commands to be executed repeatedly. This can lead to data inconsistency between the source and destination databases, especially if non-idempotent commands (such as `INCRBY` and `LPUSH`) exist or if a full task is re-run. If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters. Note Only DTS task parameters are modified—not database parameters. Parameters that may be adjusted include those listed in Modify instance parameters.