Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Evaluating Next-Generation Log Processing: iLogtail SPL vs. Logstash

Evaluating Next-Generation Log Processing: iLogtail SPL vs. Logstash

This article introduces the iLogtail 2.0 log processing solution and its new SPL processing mode, comparing its features and advantages over the popular Logstash tool.

By Abing

Introduction

In today's data-driven era, log collection and processing tools are essential to ensure system stability and optimize O&M efficiency. With the increasing volume of enterprise data and the growing complexity of system architectures, traditional log processing tools are facing multiple challenges in terms of performance, flexibility, and usability. As a popular open-source log collection and processing tool, Logstash has been widely used in various IT environments. However, with the diversification of requirements and the development of technologies, a new-generation log processing solution iLogtail came into being.

Recently, iLogtail 2.0 has introduced the SPL processing mode to further enhance log processing capabilities. In this article, we will focus on the in-depth discussion of the reason why iLogtail was chosen and its unique advantages over Logstash in SPL data processing. By comparing the architecture, performance, and functionality of these two tools, it is expected to reveal how iLogtail stands out from the increasingly complex log processing requirements, and help you make an informed technical choice.

Introduction to iLogtail and SPL

iLogtail is an open-source log collection tool provided by Alibaba Group. Since it was designed, it has been optimized for large-scale data and cloud-native scenarios to provide a low-latency and efficient log collection solution. iLogtail has many production-level features such as lightweight, high performance, and automated configuration, which are widely used internally by Alibaba Group and tens of thousands of Alibaba Cloud customers. You can deploy it in physical machines, virtual machines, and Kubernetes environments to collect telemetry data, such as logs, traces, and metrics.

iLogtail 2.0 newly supports the SPL processing mode which is a stream processing language designed for unstructured data such as logs. The design of SPL is inspired by the Unix pipeline. In iLogtail, after a piece of log data is collected from a data source, it will go through a series of processing operations, just like a pipeline, and finally flow to the downstream storage. Therefore, it can be said that the design of SPL naturally adapts to the scenario of log processing.

1

In addition to the flexible processing functions provided by the language, the iLogtail SPL itself demonstrates a large number of high-performance technologies. By using SIMD and other parallel computing, it maintains high performance while ensuring flexibility.

iLogtail SPL vs. Logstash Filter

2

The filter plug-ins of Logstash allow users to filter, parse, and modify log data. Common plug-ins include grok (to parse regular expressions), date (to parse time), and mutate (to modify fields). By combining these plug-ins into a pipeline, Logstash provides powerful log data processing capabilities. So which one performs better between the filter plug-ins of Logstash and the iLogtail SPL? Next, we will compare them in terms of features and scenario performance.

Features

According to the latest official documentation, Logstash currently provides 49 filter plug-ins, basically covering common log processing scenarios. Users can write configurations in the config file to combine these plug-ins.

As a stream processing language, SPL also provides rich operators and programmability. According to the latest official documentation, SPL provides 8 data processing instructions, including regular expression parsing and JSON parsing. In addition, SPL is compatible with most SQL instructions, providing 110 common data parsing and processing instructions in SQL, and is still under development to provide more processing features. Based on these instructions, users can write SPL statements to meet various data processing requirements.

Since they support multiple features, the next issue involved is usability. Next, we will list some common log processing scenarios and compare the configuration methods of Logstash and iLogtail SPL.

Regular Expression Parsing + Time Parsing

Taking the Nginx access log as an example, enter the following statements:

142.207.88.67 - - [18/Jun/2024:12:14:26 +0800] "DELETE http://www.districtdot-com.biz/syndicate HTTP/1.1" 289 3715 "http://www.chiefscalable.biz/webservices" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_2_1 like Mac OS X; en-US) AppleWebKit/534.46.7 (KHTML, like Gecko) Version/5.0.5 Mobile/8B119 Safari/6534.46.7"

3

Logstash does not support native regular expressions, but only supports the regular extension of Grok. For basic features, there is little difference in the user experience between Logstash and iLogtail SPL.

Add, Delete, and Filter Fields under Different Conditions

Taking a simple JSON log as an example, add a new field based on the user-agent field:

{"url": "POST /PutData HTTP/1.1", "user-agent": "aliyun-sdk-java"}

4

Since the config file has its own syntax, Logstash provides a certain degree of programmability. However, as a config file language, it is not particularly good at data processing. Compared with the syntax of SPL which is compatible with SQL statements, the syntax of Logstash is significantly more complex, which makes config files lengthy and difficult to manage. In addition, the processing plug-ins of Logstash are implemented in the Ruby environment, which presents certain limitations in leveraging data parallelism. This will be discussed in detail in the next chapter.

Extract and Parse Nested JSON

In many cases, raw logs are in a complex nested JSON structure. The fields need to be further processed before they can be stored in the backend. Assume that we need to process the following Nginx logs:

2024-06-24 12:26:04.063 INFO 24 --- [traceId=edda5daxxxxxxxxxcfa3387d48][ xnio-1 task-1] c.g.c.gateway.filter.AutoTestFilter : {"traceId":"edda5da8xxxxxxxxxxxxxxxxxxx387d48","headers":[{"x-forwarded-proto":"http,http","x-tenant-id":"123","x-ca-key":"a62d5xxxxxxxxxxxxxxxxxxxxxxxxb1cff8637","x-forwarded-port":"80,80","x-forwarded-for":"10.244.2.0","x-ca-client-ip":"10.244.2.0","x-product-code":"xxxxx","authorization":"bearer 0ed29xxxxxxxxxxxxxxxxxxxxxxxxx71899","x-forwarded-host":"gatxxxxxxxxx.gm","x-forwarded-prefix":"/xxxxxx","trace-id":"edda5da8278xxxxxxxxxxxxxxxxxxx49cfa3387d48","x-ca-api-id":"1418470181321347075","x-ca-env-code":"TEST"}],"appName":"Super administrator","responseTime":15,"serverName":"test-server","appkey":"a62d54b6bxxxxxxxxxxxxxxxxxxx37","time":"2021-08-01 12:26:04.062","responseStatus":200,"url":"/test/v4/orgs/123/list-children","token":"bearer 0ed29c72-0d68-4e13-a3f3-c77e2d971899"}

The processing steps are as follows:

  1. Extract the message field to obtain the request body in the JSON format.
  2. Parse the JSON data of the body.
  3. There is only one object in the header, and its fields are tiled to the outermost layer.
  4. Separate the x-forwarded-port to get the first port.
  5. Splice the x-forwarded-host and the first port into a new field by a colon (:), and then splice the x-forwarded-prefix.
  6. Delete the remaining unnecessary fields.

5

Logstash does not support parsing nested JSON logs, so it requires a series of operations, which is troublesome. iLogtail SPL supports JsonPath parsing, which allows users to extract the values of nested fields from JSON strings more concisely. In addition, similar to Logstash, iLogtail SPL also supports various operations on strings, including a series of complex parsing operations.

Scenario Performance

As a log collector, its resource occupation is also critical. Too much resource occupation will affect the user's business. This is also unacceptable. Then, this article will compare the resource usage of iLogtail and Logstash when they are collecting logs.

Logstash does not support log collection in the Kubernetes scenario. Therefore, a Linux host is used to test the resource usage of the two collectors when they are collecting the Nginx access log mentioned before.

  • Test environment: 8 core 16GB, operating system Ubuntu 22.04
  • Test version: iLogtail 2.0.4 and Logstash 8.14.1 (Both of the two use default parameter configuration.)

  • Test scenario:

    • Scenario 1: regular expression parsing + time parsing
    • Scenario 2: add, delete, and filter fields under different conditions
    • Scenario 3: extract and parse nested JSON
  • Test data: generated logs at 1MB/s, 5MB/s, and 10MB/s are written to files with a total duration of 3 minutes

To avoid differences in data collection performance caused by downstream data, both iLogtail and Logstash use filter plug-ins to discard all logs after passing through the preceding config file. With this method, we will only test and compare the resource usage of the two collectors during log collection and processing.

The following table describes the detailed resource usage of iLogtail and Logstash processes in three scenarios:

Scenario 1: regular expression parsing+ time parsing

6
7

Observation: Logstash has a significantly higher startup latency than iLogtail at the beginning of the collection. In addition, the CPU usage and memory usage of Logstash are much higher than those of iLogtail. The CPU resources used by Logstash to process logs at 1MB/s are similar to those used by iLogtail to process logs at 10MB/s. In addition, the CPU usage of Logstash also fluctuates significantly.

Scenario 2: add, delete, and filter fields under different conditions

8
9

Observation: Logstash still processes logs after log generation is completed (at the 180th second), indicating that its processing capability has reached the bottleneck and a relatively large collection delay has occurred.

To further explore the collection bottlenecks of Logstash and iLogtail, we further refine and test different log generation rates. When the CPU usage is lower than 0.1, it is regarded as the end of collection processing, and the collection delay within 3 seconds is regarded as normal. Finally, according to our test, the processing bottleneck of Logstash in Scenario 2 is the log generation at 8MB/s. However, iLogtail can collect logs at 50MB/s almost without delay, and its resource usage remains relatively low (130% of CPU usage and 61MB of memory usage). The processing performance of iLogtail is six times as high as that of Logstash!

Scenario 3: extract and parse nested JSON

10
11

Observation: Similar to Scenario 1, you can observe similar fluctuations in the CPU usage of Logstash. However, the parsing of Scenario 3 is more complicated, so the glitches that exist in the CPU usage are now relatively sparse.

Analysis

12

From the above three scenarios from simple to complex, we can analyze and draw some conclusions:

1. Resource usage:

  1. iLogtail SPL provides better CPU usage and memory usage than Logstash. iLogtail has an average memory size of about 40MB, and its maximum memory size is about 50MB. However, once Logstash is started, it occupies about 900MB of memory. After the log processing starts, the memory usage will be significantly increased with a maximum of 1250MB. That is to say, the resources used by iLogtail only account for about 5% of those occupied by Logstash.
  2. From the perspective of CPU usage, it can be seen that iLogtail SPL maintains a relatively stable usage when processing logs. However, the resource usage of Logstash is not very stable. On the one hand, there is a very noticeable surge in CPU usage when it starts log collection. There are two possible reasons: one is the accumulation of a large number of logs due to collection delay, and the other is the initialization required to start the collection. On the other hand, the CPU usage of Logstash fluctuates frequently during continuous collection, resulting in many glitches.

2. Real-time:

  1. The preceding metrics are monitored at the same time when log generation starts. By observing the changes in CPU usage in each scenario, it can be found that iLogtail immediately detects log generation events and then begins to collect logs. However, Logstash has a significant delay. The CPU usage does not change until 10 seconds later after the first log is generated.
  2. In addition, let's observe scenarios where a single log is short but has many entries, like Scenario 2. Log generation ends at the 180th second. In the next 20 seconds, there is no new log generated. iLogtail completes processing at about 180th second. Its CPU usage drops to nearly 0 in the following 20 seconds. In contrast, Logstash can process all logs within 200 seconds under low traffic conditions (1MB/s). However, once the traffic increases (10MB/s), Logstash cannot process all the logs within 200 seconds. This causes a relatively large delay in log collection.

3. Performance:

In the extreme test of Scenario 2, the maximum traffic that the iLogtail SPL can handle is six times as heavy as that for Logstash Filter. iLogtail SPL is written in C++ at the bottom layer and uses SIMD and other high-performance computing technologies to process log data in parallel. Therefore, it has reached a high level in terms of both performance and resource usage. However, many filter plug-ins of Logstash are written in Ruby. As a scripting language, its inherent limitations constrain its processing performance. At the same time, the design of Logstash filter Ruby plug-ins does not effectively leverage the streaming nature of log data. It is designed to process only one log at a time, and cannot process a group of logs in parallel, which is one of the reasons for its poor performance. This is especially pronounced in scenarios like Scenario 2 where the log entries are short but numerous. Of course, this can be solved by increasing the number of threads, but it will lead to higher resource usage.

4. Scalability:

Logstash allows users to write custom filter plug-ins in Ruby or Java. iLogtail SPL currently does not support user extensions, but in the future, it will provide UDF and other methods for users to easily implement custom processing needs. At the same time, the development team is also developing new features. In addition, iLogtail provides a plug-in processing mode similar to Logstash and supports C++ and Go programming. C++ processing plug-ins provide stronger performance while Go processing plug-ins provide greater flexibility.

Summary

In summary, iLogtail shows significant advantages over Logstash in log processing, thanks to its SPL processing mode. From the perspective of features, although Logstash provides extensive features through a variety of filter plug-ins, it is cumbersome and not intuitive to configure for advanced requirements such as complex logic processing, dynamic field operations, and nested JSON parsing. In contrast, iLogtail SPL not only integrates common data processing instructions but also incorporates SQL compatibility, which greatly improves the simplicity and readability of configurations and reduces the difficulty for users to get started.

In terms of performance, the advantages of iLogtail are more pronounced. Thanks to its core implementation in C++ and full utilization of modern hardware features such as SIMD, iLogtail SPL comprehensively surpasses the Ruby-based Logstash Filter in terms of resource usage, stability, and processing speed. Especially in high throughput scenarios, iLogtail can maintain stable high performance with low resource usage, which is particularly important for cost-sensitive enterprises and large-scale distributed systems.

In conclusion, iLogtail 2.0 with SPL processing mode is undoubtedly a more attractive choice for enterprises seeking efficient, flexible, and resource-friendly log processing solutions. It meets diverse log processing requirements while setting new industry standards in performance and usability, providing powerful tools for O&M teams to cope with increasing data processing challenges. With continuous technological iteration and ecosystem improvement, iLogtail aims to help enterprises better manage data streams, enhance operational efficiency, and accelerate digital transformation in the future.

performance O&M Log Service Storage Service Open Source Logstash cloud native Log Processing iLogtail SPL processing Logstash Filter
0 1 0
Share on

Read previous post:

Build RAG Applications with Spring Cloud Alibaba AI

Read next post:

Koordinator v1.5 Continuous Optimization, Join CNCF Sandbox

Alibaba Cloud Native Community

506 posts | 48 followers

You may also like

Comments

Alibaba Cloud Native Community

506 posts | 48 followers

Related Products

More Posts by Alibaba Cloud Native Community

See All