Workers - ApsaraMQ for Kafka - Alibaba Cloud Documentation Center

This topic describes the definition and configurations of workers in Custom Connector.

Background information

A worker is a Java Virtual Machine (JVM) process that runs connector logic. Each worker creates a set of tasks that run in parallel threads and is responsible for copying the data.

Tasks do not store status and can be started, stopped, or restarted at any time. To meet the requirements on the elasticity of workers, Serverless App Engine (SAE) provides a data pipeline that is elastic and scalable. SAE automatically scales the pipeline within a specified range based on the threshold of CPU utilization or memory usage.

Worker configuration overview

The configuration parameters of workers are compatible with the configuration parameters of Kafka Connect. For information about all configuration parameters of Kafka Connect, see 3.5 Kafka Connect Configs.

Default worker configurations

ApsaraMQ for Kafka provides a one-stop platform to partially host Kafka Connect. The following code describes the default configurations:

key.converter=org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable=true
value.converter=org.apache.kafka.connect.storage.StringConverter
value.converter.schemas.enable=true

offset.flush.interval.ms=60000
request.timeout.ms=40000
task.shutdown.graceful.timeout.ms=10000

plugin.path=/opt/kafka/connect/plugins
rest.advertised.port=8083
topic.creation.enable=false
listeners=http://:8083

Parameters that support custom values

When you create a connector, you can specify custom values for the following parameters. The custom values overwrite the default values that are provided by ApsaraMQ for Kafka.

Required parameters (The values of these parameters are preset in the console)


Parameter	Description	Example
bootstrap.servers	The endpoint that is used to access the Message Queue for Apache Kafka instance.	alikafka-post-cn-7mz301t5****.alikafka.aliyuncs.com:9092
offset.storage.topic	The name of the Message Queue for Apache Kafka topic that is used to store the information about offsets.	topic_offset
config.storage.topic	The name of the Message Queue for Apache Kafka topic that is used to store the information about configurations.	topic_config
status.storage.topic	The name of the Message Queue for Apache Kafka topic that is used to store the information about status.	topic_status
group.id	The ID of the Connect cluster group to which the worker belongs.	test

The following code describes the preset values of the preceding required parameters in the console:

group.id=connect-eb-cluster-35345
offset.storage.topic=connect-eb-offset-35345
config.storage.topic=connect-eb-config-35345
status.storage.topic=connect-eb-status-35345
consumer.group.id=connector-eb-cluster-mongo-sink
bootstrap.servers=alikafka-pre-cn-zpr3156gn006-1-vpc.alikafka.aliyuncs.com:9092,alikafka-pre-cn-zpr3156gn006-2-vpc.alikafka.aliyuncs.com:9092,alikafka-pre-cn-zpr3156gn006-3-vpc.alikafka.aliyuncs.com:9092

Optional parameters

key.converter
key.converter.schemas.enable
value.converter
value.converter.schemas.enable
exactly.once.source.support
heartbeat.interval.ms
rebalance.timeout.ms
session.timeout.ms
client.dns.lookup
connections.max.idle.ms
connector.client.config.override.policy
receive.buffer.bytes
request.timeout.ms
send.buffer.bytes
worker.sync.timeout.ms
worker.unsync.backoff.ms
access.control.allow.methods
access.control.allow.origin
admin.listeners
client.id
config.providers
connect.protocol
header.converter
metadata.max.age.ms
offset.flush.interval.ms
offset.flush.timeout.ms
reconnect.backoff.max.ms
reconnect.backoff.ms
retry.backoff.ms
scheduled.rebalance.max.delay.ms
task.shutdown.graceful.timeout.ms
topic.tracking.allow.reset
topic.tracking.enable

Parameters that do not support custom values

The following items describe the parameters that do not support custom values:

Parameters that use the default values that are provided by ApsaraMQ for Kafka
```
plugin.path
rest.advertised.port
topic.creation.enable
listeners
```

Parameters that are not passed to Kafka Connect

sasl.*
ssl.*
security.*
rest.advertised.host.name
rest.advertised.listener
rest.extension.classes
client.*
inter.worker.*
metrics.*
metrics.context.*
response.http.headers.config
socket.*