Use DataWorks to write data to Redis in offline mode - DataWorks

DataWorks provides Redis Writer for you to write data to Redis data sources. This topic describes the capabilities of synchronizing data to Redis data sources.

Limits

The synchronization tasks that use this type of data source must run on exclusive resource groups for Data Integration.
If you rerun a synchronization task that uses Redis Writer to write LIST-type values to Redis, the data synchronization result is not idempotent. In this case, you must manually delete the data written by Redis Writer in the previous run from Redis before you rerun the synchronization task.

Supported data types

Redis supports values of the following data types: STRING, LIST, SET, ZSET (sorted set), and HASH. For more information about Redis, visit redis.io.

Develop a data synchronization task

For more information about the configuration procedure, see Configure a batch synchronization task by using the codeless UI and Configure a batch synchronization task by using the code editor.
For information about all parameters that are configured and the code that is run when you use the code editor to configure a batch synchronization task, see Appendix: Code and parameters.

Appendix: Code and parameters

Appendix: Configure a batch synchronization task by using the code editor

If you use the code editor to configure a batch synchronization task, you must configure parameters for the reader and writer of the related data source based on the format requirements in the code editor. For more information about the format requirements, see Configure a batch synchronization task by using the code editor. The following information describes the configuration details of parameters for the reader and writer in the code editor.

Code for Redis Writer

In the following code, a batch synchronization task is configured to synchronize data from a MySQL database to Redis.

{
    "type":"job",
    "version":"2.0", // The version number.
    "steps":[
        { // The following sample code is used to configure the Reader. For more information about the parameters of the Reader, see the topic of the related Reader. 
            "stepType":"mysql",   
            "parameter": {
                "envType": 0,
                "datasource": "xc_mysql_demo2", 
                "column": [
                    "id",
                    "value",
                    "table"
                ],
                "connection": [
                    {
                        "datasource": "xc_mysql_demo2",
                        "table": []
                    }
                ],
                "where": "",
                "splitPk": "",
                "encoding": "UTF-8"
            },,
            "name":"Reader",
            "category":"reader"
        },
        {// The following sample code is used to configure the Writer: 
            "stepType":"redis",                    // The plug-in name. Set the value to redis. 
            "parameter":{                          // Configure the following parameters of Redis Writer. 
                "expireTime":{                     // The expiration time of the values that are cached in Redis. Set the value to a period of time in seconds or a UNIX timestamp. For example, you can set this parameter to "seconds":"1000".
                            }, 
                "keyFieldDelimiter":"u0001",       // The delimiter that you want to use to separate keys when Redis Writer writes data to Redis. 
                "dateFormat":"yyyy-MM-dd HH:mm:ss",// The format in which Redis Writer writes the data of the DATE type to Redis. 
                "datasource":"xc_mysql_demo2",     // The name of the data source. The name must be the same as the name of the data source that you added. 
                "envType": 0,                      // The type of the environment. Set the value to 1 for the development environment. Set the value to 0 for the production environment. 
                "writeMode":{                      // The mode in which data is written to Redis. 
                    "type": "string",                 // The data type of the value that you want to write to Redis. 
                    "mode":"set",                  // The mode in which the data of a type specified by the type parameter is written to Redis. 
                    "valueFieldDelimiter":"u0001", // The delimiter that you want to use to separate column values. 
                             },
                "keyIndexes":[0,1],                // The indexes of source columns that are used as the key when Redis Writer writes data to Redis. In this example, the value is set to [0,1], which indicates that the first and second columns of the source data are used as the key. 
                "batchSize":"1000"                 // The number of data records to write at a time. 
        "column": [                        // If you set the type parameter to string and the mode parameter to set and do not configure the column parameter, the values to be written to Redis are strings that are connected by delimiters. For example, for a CSV source file, if the value of id is 1, the value of name is Bob, the value of age is 18, and the value of sex is male, the values to be written to Redis are "18::male". If you set the type parameter to string and the mode parameter to set and configure the column parameter in the following format, the values are written to Redis in the JSON format, such as {"id":1,"name":"Bob","age":18,"sex":"male"}.
                {
                "name": "id",
                "index": "0"

                },
                {
                "name": "name",
                "index": "1"
                },
                {
                "name": "age",
                "index": "2"
                },
                {
                "name": "sex",
                "index": "3"
                }
            ]
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"                           // The maximum number of dirty data records allowed. 
        },
        "speed":{
            "throttle":true,// Specifies whether to enable throttling. The value false indicates that throttling is disabled, and the value true indicates that throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":1, // The maximum number of parallel threads. 
            "mbps":"12"// The maximum transmission rate. Unit: MB/s. 
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

Parameters in code for Redis Writer

Parameter	Description	Required	Default value
expireTime	The expiration time of the values that are cached in Redis. Unit: seconds. If you do not configure this parameter, the default value `0` is used, which indicates that the values are permanently valid. You can specify the value of the expireTime parameter in one of the following modes: seconds: the relative period of time in seconds starting from the current point in time. It specifies the period during which the values are valid. unixtime: the number of seconds that have elapsed since 00:00:00 on January 1, 1970. It specifies the point in time when the values expire.	No	0
keyFieldDelimiter	The delimiter that you want to use to separate keys when Redis Writer writes data to Redis. Example: key=key1\u0001id. If multiple keys need to be concatenated, this parameter is required. If the data contains only one key, this parameter is not required.	No	\u0001
dateFormat	The format in which Redis Writer writes the data of the DATE type to Redis. Set the value to yyyy-MM-dd HH:mm:ss.	No	No default value
datasource	The name of the data source. The name must be the same as the name of the data source that you added.	Yes	No default value
selectDatabase	The database to which you want to write data. The first database in a Redis cluster is database 0. Data Integration does not allow you to select the database to which you want to write data in a Redis cluster.	No	database 0
writeMode	The mode in which Redis Writer writes data to Redis. Redis Writer can write values of the following types: STRING LIST SET ZSET (sorted set) HASH The value of the writeMode parameter varies based on the data types of the values that you want to write to Redis. For more information, see Description of the writeMode parameter. Note When you configure Redis Writer, you can specify only one data type for the writeMode parameter. If you do not configure this parameter, the default value string is used.	No	string
keyIndexes	The indexes of source columns that are used as the key. The indexes start from 0. This indicates that the index of the first column is 0, and the index of the second column is 1, and so on. If you want to specify a specific column of the source as the key, set this parameter to the index of the column. For example, if you want to specify the first column as the key, set this parameter to `0`. If you want to specify a range of contiguous columns of the source as the key, set this parameter to a closed interval that specifies the indexes of these columns. For example, if you want to specify the second, third, and fourth columns as the key, set this parameter to `[1,3]`. Note After you configure the keyIndexes parameter, Redis Writer uses the remaining columns as the value. If you want to synchronize only some of the columns in the source, specify the names of the columns when you configure the related Reader.	Yes	No default value
batchSize	The number of data records to write at a time. Set this parameter to an appropriate value based on your business requirements. This greatly reduces the interactions between Data Integration and Redis and increases throughput. If you set this parameter to an excessively large value, an out of memory (OOM) error may occur during data synchronization.	No	1,000
timeout	The timeout period of the connection to Redis when data is written to Redis. Unit: milliseconds.	No	30,000
redisMode	The deployment mode of Redis. Cluster mode: If you set the redisMode parameter to Cluster mode, the cluster mode takes effect. In this mode, when you synchronize data from another data source to a Redis cluster, Data Integration directly connects to and communicates with the Redis cluster. If you want to write data to a self-managed Redis cluster or an ApsaraDB for Redis cluster instance that uses the direct connection mode, you must set the redisMode parameter to this value. This mode does not support batch data write. Non-cluster mode: If you leave the redisMode parameter empty, the non-cluster mode takes effect. In most cases, if you want to write data to an ApsaraDB for Redis cluster instance that uses the proxy connection mode, an ApsaraDB for Redis read/write splitting instance, or an ApsaraDB for Redis standard master-replica instance, you must use the non-cluster mode. This mode supports batch data write. Note Only exclusive resource groups for Data Integration are supported.	No	No default value
column	The names of the columns to which you want to write data. If you set the type parameter to string and the mode parameter to set, take note of the following points: If the column parameter is not configured, the values stored in Redis are strings that are connected by delimiters. For example, in a CSV source file, the value of id is 1, the value of name is Bob, the value of age is 18, and the value of sex is male. In this case, the values to be written to Redis are "18::male". If the column parameter is configured in the "column": [{"index":"0", "name":"id"}, {"index":"1", "name":"name"}] format, the values are stored in Redis in the JSON format of {"id":"Value of a source column","name":"Value of a source column"}. For example, the value of id is 1, and the value of name is Bob. In this case, the values stored in Redis are {"id":1,"name":"Bob"}.	No	No default value

Description of the writeMode parameter

When you configure Redis Writer, you can specify only one data type for the writeMode parameter. If you do not configure this parameter, the default value string is used.

Data type of values	type parameter (required)	mode parameter (required)	valueFieldDelimiter parameter (optional)	Configuration example of the writeMode parameter
STRING	Set the type parameter to `string`.	The mode in which Redis Writer writes the data of the STRING type to Redis. Take note of the following rules when you set the type parameter to string: Set the mode parameter to `set`. If the data that you want to write to Redis already exists in Redis, existing data is overwritten.	The delimiter that you want to use to separate column values. Default value: `\u0001`. This parameter is required if you specify two or more columns as the value. For example, you specify three columns of the source data as the value and use the default delimiter. In this case, the column values are concatenated in the value1\u0001value2\u0001value3 format. This parameter is not required if you specify only one column as the value.	`"writeMode":{ "type": "string", "mode": "set", "valueFieldDelimiter": "\u0001" }`
LIST	Set the type parameter to `list`.	The mode in which Redis Writer writes the data of the LIST type to Redis. Valid values: `lpush`: indicates that data is stored at the leftmost of the list. `rpush`: indicates that data is stored at the rightmost of the list.		`"writeMode":{ "type": "list", "mode": "lpush\|rpush", "valueFieldDelimiter": "\u0001" }`
SET	Set the type parameter to `set`.	The mode in which Redis Writer writes the data of the SET type to Redis. Take note of the following rules when you set the type parameter to set: Set the mode parameter to `sadd`, which indicates that data is stored to a set. If the data that you want to write to Redis already exists in Redis, existing data is overwritten.		`"writeMode":{ "type": "set", "mode": "sadd", "valueFieldDelimiter": "\u0001" }`
ZSET (sorted set)	Set the type parameter to `zset`.	The mode in which Redis Writer writes the data of the ZSET type to Redis. Take note of the following rules when you set the type parameter to zset: Set the mode parameter to `zadd`, which indicates that data is stored to a sorted set. If the data that you want to write to Redis already exists in Redis, existing data is overwritten.	You do not need to configure this parameter.	`"writeMode":{ "type": "zset", "mode": "zadd" }` Note If the data type is set to ZSET, each row of the source data must meet the following requirements: A row can contain only one score and one value except for the key. The score must be placed before the value. This way, Redis Writer can distinguish between the score and value.
HASH	Set the type parameter to `hash`.	The mode in which Redis Writer writes the data of the HASH type to Redis. Take note of the following rules when you set the type parameter to hash: Set the mode parameter to `hset`, which indicates that data is stored to a hash sorted set. If the data that you want to write to Redis already exists in Redis, existing data is overwritten.	You do not need to configure this parameter.	`"writeMode":{ "type": "hash", "mode": "hset" }` Note If the data type is set to HASH, each row of the source data must meet the following requirements: A row can contain only one attribute and one value except for the key. The attribute must be placed before the value. This way, Redis Writer can distinguish between the attribute and value.