Common errors and troubleshooting methods for pulling data from a Logstore - Simple Log Service

If a transformation rule requires data to be pulled from another Logstore, errors may occur when the data is pulled or updated. This topic describes the errors and provides solutions.

After the data transformation engine reads data from a Logstore, the engine transforms the data. If the transformation rule requires data to be pulled from external resources such as Object Storage Service (OSS), ApsaraDB RDS, and other Logstores, errors may occur when the data is pulled or updated.

Data pull or update errors may have the following impacts:

Log events that conflict with the transformation rule will cause errors and be dropped. These log events will not be included in the transformation result.
Dropped log events will not be further transformed. The transformation job will continue to transform other log events.
If a log event is split into multiple log events and a single split is dropped, all associated splits will be dropped.
Note
These log event splits are associated in a tree structure before they are exported.

Error 1: A required parameter is missing

Transformation rule

e_table_map(res_log_logstore_pull(endpoint="cn-shenzhen.log.aliyuncs.com",ak_id="xxx",
        ak_secret="xxx",project="etl-test-shenzhen",
        fields=["__source__"]),field="processid",output_fields=["__source__"])

Error log

error when calling : res_log_logstore_pull\nDetail: res_log_logstore_pull() missing 1 required positional argument: 'logstore'", "requestId": "

Troubleshooting method
The error message missing 1 required positional argument is displayed because the required parameter logstore is missing.
Solution
Check the orchestration syntax and configure the missing parameter.

Error 2: The delete_data parameter is not configured

Transformation rule

e_table_map(res_log_logstore_pull(endpoint="xx",ak_id="xxx",
        ak_secret="xxx",project="etl-test-shenzhen",logstore="rds-mysql-test",
        fields=["__source__"],primary_keys="cid"),field="processid",output_fields=["__source__"])

Error log

when setting parameter primart_keys,need set delete_data\nDetail: None

Troubleshooting method
This error occurs because the delete_data parameter is not configured when the primary_keys parameter is configured. The primary_keys and delete_data parameters must be configured at the same time.
Solution
Configure the primary_keys and delete_data parameters, and then restart the data transformation job.

Error 3: The target Logstore does not exist

Transformation rule

e_table_map(res_log_logstore_pull(endpoint="xx",ak_id="xxx",
        ak_secret="xxx",project="etl-test-shenzhen",logstore="pull_logstore_test9900881",
        fields=["__source__"],primary_keys="cid"),field="processid",output_fields=["__source__"])

Error log

message:  fetch data get errors:{"errorCode": "LogStoreNotExist", "errorMessage": "logstore pull_logstore_test9900881 does not exist", "requestId": "5D7227AA269948500404B777"},retrytimes=2

Troubleshooting method
This error occurs because the specified Logstore does not exist.
Solution
Check the orchestration syntax and enter the valid Logstore name.

Error 4: The specified AccessKey pair is invalid

Transformation rule

e_table_map(res_log_logstore_pull(
        endpoint="cn-hangzhou.log.aliyuncs.com",
        ak_id="xx",
        ak_secret="xx",
        project="sls-test",
        logstore="pull_logstore_test",
        fields=[("id", "new_id"), "name", "status"],
        from_time="begin"), "name", "new_id")

Error log

message:  fetch data get errors:{"errorCode": "SignatureNotMatch", "errorMessage": "signature gdaL/nWSRtve5FOB+QqHO/sBdnA= not match", "requestId": "5D760261ED35D40AA4AB1953"},retrytimes=1

message:  fetch data get errors:{"errorCode": "Unauthorized", "errorMessage": "AccessKeyId not found: xx", "requestId": "5D7602A01808F9EAA6EB0E2B"},retrytimes=3

Troubleshooting method
This error occurs because the ak_secret and ak_id parameters in the orchestration syntax do not match or the ak_id does not exist.
Solution
Specify valid values for the ak_secret and ak_id parameters, and then restart the data transformation job.

Error 5: The current RAM user does not have sufficient permissions

Transformation rule

e_table_map(res_log_logstore_pull(
        endpoint="cn-hangzhou.log.aliyuncs.com",
        ak_id="xx",
        ak_secret="xx",
        project="sls-test",
        logstore="pull_logstore_test",
        fields=[("id", "new_id"), "name", "status"],
        from_time="begin"), "name", "new_id")

Error log

message:  fetch data get errors:{"errorCode": "Unauthorized", "errorMessage": "denied by sts or ram, action: log:ListShards, resource: acs:log:cn-hangzhou:1654218965343050:...}

Troubleshooting method
The Unauthorized error message is displayed because the current RAM user does not have sufficient read and write permissions on the specified Logstore.
Solution
Grant sufficient permissions to the RAM user.

Error 6: The specified endpoint is invalid

Transformation rule

e_table_map(res_log_logstore_pull(
        endpoint="xxx",
        ak_id="xx",
        ak_secret="xx",
        project="sls-test",
        logstore="pull_logstore_test",
        fields=[("id", "new_id"), "name", "status"],
        from_time="begin"), "name", "new_id")

Error log

message:  fetch data get errors:{"errorCode": "ProjectNotExist", "errorMessage": "The Project does not exist : ali-sls-etl-regression-test", "requestId": "5D760AB12ECD0722AA1DD681"}

message:  fetch data get errors:{"errorCode": "LogRequestError", "errorMessage": "HTTPConnectionPool(host='ali-sls-etl-regression-test.xx', port=80): Max retries exceeded with url: /logstores/pull_logstore_test/shards (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f968a298ef0>: Failed to establish a new connection: [Errno -2] Name or service not known))

Troubleshooting method
The ProjectNotExist or LogRequestError error message is displayed because the specified endpoint is invalid or the required project does not exist in the specified endpoint.
Solution
Correct the endpoint settings. For more information, see Endpoints.