Elasticsearch:Use the logstash-input-maxcompute plug-in

Prerequisites

• The logstash-input-maxcompute plug-in is installed.

  For more information, see Install and remove a plug-in.

• Alibaba Cloud MaxCompute is activated, a project is created, a table is created for the project, and data is imported to the table.

  For more information, see Prepare and Getting Started.

Use logstash-input-maxcompute

After the prerequisites are met, you can create a pipeline by following the instructions provided in Use configuration files to manage pipelines. When you create the pipeline, configure the pipeline parameters based on the descriptions in the table of the Parameters section. After you configure the parameters, save the settings and deploy the pipeline. This way, Logstash can be triggered to read data from MaxCompute and transfer the data to the destination data source.

The following code provides a pipeline configuration example. For more information about the parameters, see Parameters.

input {
    maxcompute {
        access_id => "Your accessId"
        access_key => "Your accessKey"
        endpoint => "maxcompute service endpoint"
        project_name => "Your project"
        table_name => "Your table name"
        partition => "pt='p1',dt='d1'"
        thread_num => 1
        dirty_data_file => "/ssd/1/<Logstash cluster ID>/logstash/data/XXXXX.txt"
    }
}

output {
    stdout {
        codec => rubydebug
    }
}

Parameters