Simple Log Service:Use Logstash to consume log data

Features

• Distributed collaborative consumption: Multiple servers can be configured to consume log data from a Logstore at the same time.

• High performance: If you use a Java consumer group, the consumption speed of a single-core CPU can reach 20 MB/s.

• High reliability: Simple Log Service saves consumption checkpoints. This mechanism resumes log consumption from the last checkpoint after a consumption exception is resolved.

• Automatic load balancing: Shards are automatically allocated based on the number of consumers in a consumer group. If a consumer joins or leaves the consumer group, shards are automatically reallocated.

Procedure

1. Install Logstash.

   1. Download the installation package.

   2. Decompress the downloaded package to the specified directory.

2. Install the Logstash input plug-in.

   1. Download the Logstash input plug-in. To download the plug-in, visit logstash-input-sls.

   2. Install the Logstash input plug-in.

      logstash-plugin install logstash-input-sls

      Note

      For information about the causes of installation failures and solutions, see Plug-in installation and configuration.

3. Start Logstash.

   logstash -f logstash.conf

   The following table describes the parameters.

   Parameter	Type	Required	Description
   endpoint	string	Yes	The endpoint for the region where the Simple Log Service project resides. For more information, see Endpoints.
   access_id	string	Yes	The AccessKey ID of the Alibaba Cloud account or RAM user that is used to access the consumer group. For more information, see Use custom policies to grant permissions to a RAM user.
   access_key	string	Yes	The AccessKey secret of the Alibaba Cloud account or RAM user that is used to access the consumer group. For more information, see Use custom policies to grant permissions to a RAM user.
   project	string	Yes	The name of the Simple Log Service project.
   logstore	string	Yes	The name of the Simple Log Service Logstore.
   consumer_group	string	Yes	The name of the consumer group.
   consumer_name	string	Yes	The name of the consumer. The name must be unique in a consumer group.
   position	string	Yes	The position where data consumption starts. Valid values: begin: consumes data from the first log that is written to the Logstore. end: consumes data from the current time. yyyy-MM-dd HH:mm:ss: consumes data from the specified point in time.
   checkpoint_second	number	No	The interval at which checkpoints are recorded. We recommend that you set the interval to a value between 10 seconds and 60 seconds. Minimum value: 10. Default value: 30. Unit: seconds.
   include_meta	boolean	No	Specifies whether the logs contain metadata, such as the log source, time, tags, and topic fields. Default value: true.
   consumer_name_with_ip	boolean	No	Specifies whether to include an IP address in a consumer name. Default value: true. You must set this parameter to true if distributed collaborative consumption is applied.

Examples

input {
  logservice{
  endpoint => "your project endpoint"
  // Configure environment variables. In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
  access_id => process.env.ALIBABA_CLOUD_ACCESS_KEY_ID
  access_key => process.env.ALIBABA_CLOUD_ACCESS_KEY_SECRET
  project => "your project name"
  logstore => "your logstore name"
  consumer_group => "consumer group name"
  consumer_name => "consumer name"
  position => "end"
  checkpoint_second => 30
  include_meta => true
  consumer_name_with_ip => true
  }
}

output {
  stdout {}
}