Two Methods That Can Greatly Reduce the Wait Time When Logstash Starts

By Wei Bin, CTO of Puxiang Communication Technology

Logstash is the most powerful extract, transform, and load (ETL) tool in the Elastic Stack. Logstash is slightly larger than Beats, but it provides rich features and powerful processing capabilities. You may have experienced Logstash's slow startup speed. Is there any way to reduce the wait time when Logstash starts and improve the efficiency of profile editing and processing? This article proposes a technique to help you resolve these two problems so that you can make better use of this powerful tool.

Solutions

Reduce the restart frequency of Logstash to save precious time.
Quickly input the content to be processed to Logstash.

1. Enable Reload Configuration

When you start Logstash, you can add the -r parameter to hot-reload the profile. The following content describes the operation results.

After you modify the profile, the new profile automatically takes effect, and you do not need to restart Logstash.

The following figure shows the definition of the -r parameter.

After you edit a profile, such as the test.conf file, run the following startup command:

bin/logstash -f test.conf -r

After the startup is complete, modify the content of the test.conf file and save it. After 1 second, Logstash generates a log output similar to the following code. In particular, the part marked in the red box indicates that the file has been reloaded.

If an error occurs in the new profile, you will see an error log in Logstash and can fix the problem based on the error information provided.

This resolves the first problem.

2. Use the http input Plug-in

Another difficulty in editing a profile is the detailed testing required for the input content in different formats. This operation prevents parsing errors. Previously, we usually resolved this problem by using standard input methods, such as the stdin input method. However, standard input methods did not support advanced text editing, and the profile hot-reloading feature does not support standard input methods.

To address this problem, we recommend that you use the http input plug-in. The following code shows how to configure this plug-in.

input{
    http{
        port => 7474
        codec => "json"
    }
}

Then, you can use your favorite http request tool, such as POSTMan or Insomnia, to send the to-be-tested content to http://localhost:7474. The following figure is a screenshot of Insomnia.

This solves the second problem.

Summary

I am sure you are eager to try out what you have learned. You can open your computer, open Logstash, and edit the test.conf file by entering the following code:

input{
    http{
        port => 7474
        codec => "json"
    }
}
filter{
}
output{
        stdout{
        codec => rubydebug{
            metadata => true
        }
    }
}

Then, run the following startup command:

bin/logstash -f test.conf -r

Open Insomnia, enter the content to be tested, and click Send to start editing profiles.

About the Author

Wei Bin, CTO of Puxiang Communication Technology, is an open-source software advocate. He was the first Elastic certified engineer in China and the initiator of the community projects Elastic Daily and Elastic Talk. Wei Bin was awarded the 2019 Annual Partner Architect - Special Contribution Award by Elastic China. He has rich practical experiences in open-source software such as Elasticsearch, Kibana, Beats, Logstash, and Grafana. He provides consulting and training services for customers in many industries, including retail, finance, insurance, securities, and technology. He helps customers identify the roles of open-source software in their business, implement open-source software from scratch, and scale it out to produce business value.

Declaration: This article is adapted from "Logstash Techniques You Must Master" with the authorization from the author Wei Bin. We reserve the right to investigate unauthorized use.

Community

Two Methods That Can Greatly Reduce the Wait Time When Logstash Starts

Solutions

1. Enable Reload Configuration

2. Use the http input Plug-in

Summary

About the Author

Read next post:

Data Geek

You may also like

Comments

Dikky Ryan Pratama May 9, 2023 at 5:43 am

Data Geek

Related Products

Big Data Consulting for Data Technology Solution

Big Data Consulting Services for Retail Solution

Alibaba Cloud Elasticsearch

ApsaraDB for HBase