Use the logstash-output-oss plug-in - Elasticsearch - Alibaba Cloud Documentation Center

The logstash-output-oss plug-in allows you to transfer data to Object Storage Service (OSS).

Note

logstash-output-oss is an open source plug-in maintained by Alibaba Cloud. For more information, see logstash-output-oss.

Prerequisites

The logstash-output-oss plug-in is installed.
For more information, see Install a Logstash plug-in.
OSS is activated.
For more information, see Activate OSS.
A read/write OSS bucket is created. The AccessKey ID and AccessKey secret of your account that has write permissions on the OSS bucket are obtained.
For more information, see Create buckets.
Data sources are prepared.
The data sources can be selected from all the input plug-ins that are supported. For more information, see Input plugins.

logstash-output-oss application

After the prerequisites are met, you can create a pipeline based on the instructions provided in Use configuration files to manage pipelines. When you create a pipeline, configure the pipeline parameters based on the descriptions that are provided in the table of the Parameters section. After you configure the parameters, you must save the settings and deploy the pipeline. Then, Alibaba Cloud Logstash is triggered to transfer data from data sources to OSS.

The following example shows how to transfer data from Beats to OSS:

input {
  beats {
    port => "8044"
    codec => json {
      charset => "UTF-8"

  }
}
output {
   oss {
    endpoint => "http://oss-cn-hangzhou-internal.aliyuncs.com"              
    bucket => "zl-log-output-test"                          
    access_key_id => "LTAIaxxxxx******"                 
    access_key_secret => "zuxxxx8hBpXs3e6i******"         
    prefix => "oss/database"                         
    recover => true                                      
    rotation_strategy => "size_and_time"                  
    time_rotate => 1                                     
    size_rotate => 1000
    temporary_directory => "/ssd/1/<Logstash cluster ID>/logstash/data/22"                            
    encoding => "gzip"                                 
    additional_oss_settings => {
      max_connections_to_oss => 1024                      
      secure_connection_enabled => false                  
    }
    codec => json {
      charset => "UTF-8"
    }
  }
}

Note

Alibaba Cloud Logstash supports data transmission only within a virtual private cloud (VPC). If source data is on the Internet, configure a NAT gateway to access Alibaba Cloud Logstash over the Internet. For more information, see Configure a NAT gateway for data transmission over the Internet.

Parameters

The following table describes the parameters supported by logstash-output-oss.

Parameter	Type	Required	Description
`endpoint`	String	Yes	The endpoint that is used to access OSS. For more information, see Regions and endpoints.
`bucket`	String	Yes	The name of the OSS bucket.
`access_key_id`	String	Yes	The AccessKey ID of your account that has write permissions on the OSS bucket.
`access_key_secret`	String	Yes	The AccessKey secret of your account that has write permissions on the OSS bucket.
`prefix`	String	No	The prefix of file names. If this parameter is not specified, it is empty by default. Important You can set this parameter to a string. In this case, many temporary files may be created on your on-premises machine.
`recover`	Boolean	No	Specifies whether to continue to upload the data on your on-premises machine if the program exits abnormally. Default value: true.
`additional_oss_settings`	Hash	No	Additional OSS client configurations. Valid values: `server_side_encryption_algorithm`: the server-side encryption method. Only AES-256 is supported. `secure_connection_enabled`: specifies whether to enable HTTPS. Default value: false. `max_connections_to_oss`: the maximum number of connections. Default value: 1024.
`temporary_directory`	String	Yes	The temporary directory that you use to cache data before the data is uploaded to OSS. Set the value to `/ssd/1/<Logstash cluster ID>/logstash/data/<customized>`. After the data transfer task is completed, the directory is deleted within seconds.
`rotation_strategy`	String	No	The file rotation strategy. Valid values: `size`, `time`, and `size_and_time`. Default value: size_and_time.
`size_rotate`	Number	No	If the size of a file is greater than or equal to the value of `size_rotate`, OSS rotates the file. This parameter is valid only when rotation_strategy is set to size. Default value: 31457280. Unit: bytes.
`time_rotate`	Number	No	If the lifecycle of a file is greater than or equal to the value of `time_rotate`, OSS rotates the file. This parameter is valid only when rotation_strategy is set to time. Default value: 15. Unit: minutes.
`upload_workers_count`	Number	No	The number of threads that are used to upload data at the same time.
`upload_queue_size`	Number	No	The size of the upload queue.
`encoding`	String	No	The method that is used to encode messages before you upload files to OSS. Both standard compression and GZIP compression are supported. Valid values: `gzip` and `none`. Default value: none.

Temporary files

When the logstash-output-oss plug-in transfers data to OSS, the plug-in creates a temporary file on your on-premises machine. Data is temporarily stored in this file. The logstash-output-oss plug-in transfers data to OSS on a regular basis. You can use the temporary_directory parameter to specify the path of the temporary file if you have specific requirements on the path.

The following example shows a temporary file path:

/ssd/1/<Logstash cluster ID>/logstash/data/eaced620-e972-0136-2a14-02b7449b****/logstash/1/ls.oss.eaced620-e972-0136-2a14-02b7449b****.2018-12-24T14.27.part-0.data

Path element	Description
/ssd/1/<Logstash cluster ID>/logstash/data/	The temporary file directory specified by `temporary_directory`.
eaced620-e972-0136-2a14-02b7449b****	The random Universally Unique Identifier (UUID).
logstash/1	The OSS object prefix.
ls.oss	Indicates that the temporary file is generated by the logstash-output-oss plug-in.
2018-12-24T14.27	The time when the temporary file was created.
part-0	The prefix of the temporary file.
.data	The file name extension. If `encoding` is set to `gzip`, the file name extension is `.gz`. Otherwise, the file name extension is `.data`.