DataWorks provides Memcache Writer for you to write data to Memcache data sources. You can configure a batch synchronization task for a Memcache data source only by using the code editor. This topic describes the capabilities of synchronizing data to Memcache data sources.
Limits
You can configure a batch synchronization task to synchronize data to a Memcache data source only by using the code editor.
Data type mappings
Memcache Writer writes data only in the text format. The method of converting data types varies based on the format in which Memcache Writer writes data.
text: Memcache Writer uses the specified column delimiter to serialize source data to a string.
binary: This format is not supported.
Develop a data synchronization task
For information about the entry point for and the procedure of configuring a data synchronization task, see the following sections. For information about the parameter settings, view the infotip of each parameter on the configuration tab of the task.
Add a data source
Before you configure a data synchronization task to synchronize data from or to a specific data source, you must add the data source to DataWorks. For more information, see Add and manage data sources.
Configure a batch synchronization task to synchronize data of a single table
For more information about the configuration procedure, see Configure a batch synchronization task by using the code editor.
For information about all parameters that are configured and the code that is run when you use the code editor to configure a batch synchronization task, see Appendix: Code and parameters.
Appendix: Code and parameters
Appendix: Configure a batch synchronization task by using the code editor
If you use the code editor to configure a batch synchronization task, you must configure parameters for the reader and writer of the related data source based on the format requirements in the code editor. For more information about the format requirements, see Configure a batch synchronization task by using the code editor. The following information describes the configuration details of parameters for the reader and writer in the code editor.
Code for Memcache Writer
{
"type":"job",
"version":"2.0",// The version number.
"steps":[
{
"stepType":"stream",
"parameter":{},
"name":"Reader",
"category":"reader"
},
{
"stepType":"ocs",// The plug-in name.
"parameter":{
"writeFormat":"text",// The format in which Memcache Writer writes the source data.
"expireTime":1000,// The expiration time of the source data that is cached in Memcache databases.
"indexes":0,
"datasource":"",// The name of the data source.
"writeMode":"set",// The write mode.
"batchSize":"256"// The number of data records to write at a time.
},
"name":"Writer",
"category":"writer"
}
],
"setting":{
"errorLimit":{
"record":"0"// The maximum number of dirty data records allowed.
},
"speed":{
"throttle":true,// Specifies whether to enable throttling. The value false indicates that throttling is disabled, and the value true indicates that throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true.
"concurrent":1, // The maximum number of parallel threads.
"mbps":"12"// The maximum transmission rate. Unit: MB/s.
}
},
"order":{
"hops":[
{
"from":"Reader",
"to":"Writer"
}
]
}
}Parameters in code for Memcache Writer
Parameter | Description | Required | Default value |
datasource | The name of the data source. It must be the same as the name of the added data source. You can add data sources by using the code editor. | Yes | No default value |
writeMode | The write mode. Valid values:
| Yes | No default value |
writeFormat | The format in which Memcache Writer writes the source data. Only the text format is supported. text: serializes the source data to the text format. Memcache Writer uses the first column of the source data as the key and serializes the subsequent columns to strings by using the specified delimiter. Then, Memcache Writer writes key-value pairs to Memcache. For example, the following source data exists: If you set the column delimiter to \^, data is written to Memcache in the following format: | No | No default value |
expireTime | The expiration time of the source data that is cached in Memcache. Memcache supports the expiration time in the following formats:
Note If the specified expiration time is greater than 30 days, the server identifies the time as a UNIX timestamp. | No | 0, which indicates that the data never expires. |
batchSize | The number of data records to write at a time. Set this parameter to an appropriate value based on your business requirements. This greatly reduces the interactions between Data Integration and Memcache and increases throughput. If you set this parameter to an excessively large value, an out of memory (OOM) error may occur during data synchronization. | No | 1,024 |