By Abing
In today's data-driven era, log collection and processing tools are essential to ensure system stability and optimize O&M efficiency. With the increasing volume of enterprise data and the growing complexity of system architectures, traditional log processing tools are facing multiple challenges in terms of performance, flexibility, and usability. As a popular open-source log collection and processing tool, Logstash has been widely used in various IT environments. However, with the diversification of requirements and the development of technologies, a new-generation log processing solution iLogtail came into being.
Recently, iLogtail 2.0 has introduced the SPL processing mode to further enhance log processing capabilities. In this article, we will focus on the in-depth discussion of the reason why iLogtail was chosen and its unique advantages over Logstash in SPL data processing. By comparing the architecture, performance, and functionality of these two tools, it is expected to reveal how iLogtail stands out from the increasingly complex log processing requirements, and help you make an informed technical choice.
iLogtail is an open-source log collection tool provided by Alibaba Group. Since it was designed, it has been optimized for large-scale data and cloud-native scenarios to provide a low-latency and efficient log collection solution. iLogtail has many production-level features such as lightweight, high performance, and automated configuration, which are widely used internally by Alibaba Group and tens of thousands of Alibaba Cloud customers. You can deploy it in physical machines, virtual machines, and Kubernetes environments to collect telemetry data, such as logs, traces, and metrics.
iLogtail 2.0 newly supports the SPL processing mode which is a stream processing language designed for unstructured data such as logs. The design of SPL is inspired by the Unix pipeline. In iLogtail, after a piece of log data is collected from a data source, it will go through a series of processing operations, just like a pipeline, and finally flow to the downstream storage. Therefore, it can be said that the design of SPL naturally adapts to the scenario of log processing.
In addition to the flexible processing functions provided by the language, the iLogtail SPL itself demonstrates a large number of high-performance technologies. By using SIMD and other parallel computing, it maintains high performance while ensuring flexibility.
The filter plug-ins of Logstash allow users to filter, parse, and modify log data. Common plug-ins include grok (to parse regular expressions), date (to parse time), and mutate (to modify fields). By combining these plug-ins into a pipeline, Logstash provides powerful log data processing capabilities. So which one performs better between the filter plug-ins of Logstash and the iLogtail SPL? Next, we will compare them in terms of features and scenario performance.
According to the latest official documentation, Logstash currently provides 49 filter plug-ins, basically covering common log processing scenarios. Users can write configurations in the config file to combine these plug-ins.
As a stream processing language, SPL also provides rich operators and programmability. According to the latest official documentation, SPL provides 8 data processing instructions, including regular expression parsing and JSON parsing. In addition, SPL is compatible with most SQL instructions, providing 110 common data parsing and processing instructions in SQL, and is still under development to provide more processing features. Based on these instructions, users can write SPL statements to meet various data processing requirements.
Since they support multiple features, the next issue involved is usability. Next, we will list some common log processing scenarios and compare the configuration methods of Logstash and iLogtail SPL.
Taking the Nginx access log as an example, enter the following statements:
142.207.88.67 - - [18/Jun/2024:12:14:26 +0800] "DELETE http://www.districtdot-com.biz/syndicate HTTP/1.1" 289 3715 "http://www.chiefscalable.biz/webservices" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_2_1 like Mac OS X; en-US) AppleWebKit/534.46.7 (KHTML, like Gecko) Version/5.0.5 Mobile/8B119 Safari/6534.46.7"
Logstash does not support native regular expressions, but only supports the regular extension of Grok. For basic features, there is little difference in the user experience between Logstash and iLogtail SPL.
Taking a simple JSON log as an example, add a new field based on the user-agent field:
{"url": "POST /PutData HTTP/1.1", "user-agent": "aliyun-sdk-java"}
Since the config file has its own syntax, Logstash provides a certain degree of programmability. However, as a config file language, it is not particularly good at data processing. Compared with the syntax of SPL which is compatible with SQL statements, the syntax of Logstash is significantly more complex, which makes config files lengthy and difficult to manage. In addition, the processing plug-ins of Logstash are implemented in the Ruby environment, which presents certain limitations in leveraging data parallelism. This will be discussed in detail in the next chapter.
In many cases, raw logs are in a complex nested JSON structure. The fields need to be further processed before they can be stored in the backend. Assume that we need to process the following Nginx logs:
2024-06-24 12:26:04.063 INFO 24 --- [traceId=edda5daxxxxxxxxxcfa3387d48][ xnio-1 task-1] c.g.c.gateway.filter.AutoTestFilter : {"traceId":"edda5da8xxxxxxxxxxxxxxxxxxx387d48","headers":[{"x-forwarded-proto":"http,http","x-tenant-id":"123","x-ca-key":"a62d5xxxxxxxxxxxxxxxxxxxxxxxxb1cff8637","x-forwarded-port":"80,80","x-forwarded-for":"10.244.2.0","x-ca-client-ip":"10.244.2.0","x-product-code":"xxxxx","authorization":"bearer 0ed29xxxxxxxxxxxxxxxxxxxxxxxxx71899","x-forwarded-host":"gatxxxxxxxxx.gm","x-forwarded-prefix":"/xxxxxx","trace-id":"edda5da8278xxxxxxxxxxxxxxxxxxx49cfa3387d48","x-ca-api-id":"1418470181321347075","x-ca-env-code":"TEST"}],"appName":"Super administrator","responseTime":15,"serverName":"test-server","appkey":"a62d54b6bxxxxxxxxxxxxxxxxxxx37","time":"2021-08-01 12:26:04.062","responseStatus":200,"url":"/test/v4/orgs/123/list-children","token":"bearer 0ed29c72-0d68-4e13-a3f3-c77e2d971899"}
The processing steps are as follows:
Logstash does not support parsing nested JSON logs, so it requires a series of operations, which is troublesome. iLogtail SPL supports JsonPath parsing, which allows users to extract the values of nested fields from JSON strings more concisely. In addition, similar to Logstash, iLogtail SPL also supports various operations on strings, including a series of complex parsing operations.
As a log collector, its resource occupation is also critical. Too much resource occupation will affect the user's business. This is also unacceptable. Then, this article will compare the resource usage of iLogtail and Logstash when they are collecting logs.
Logstash does not support log collection in the Kubernetes scenario. Therefore, a Linux host is used to test the resource usage of the two collectors when they are collecting the Nginx access log mentioned before.
Test scenario:
To avoid differences in data collection performance caused by downstream data, both iLogtail and Logstash use filter plug-ins to discard all logs after passing through the preceding config file. With this method, we will only test and compare the resource usage of the two collectors during log collection and processing.
The following table describes the detailed resource usage of iLogtail and Logstash processes in three scenarios:
Observation: Logstash has a significantly higher startup latency than iLogtail at the beginning of the collection. In addition, the CPU usage and memory usage of Logstash are much higher than those of iLogtail. The CPU resources used by Logstash to process logs at 1MB/s are similar to those used by iLogtail to process logs at 10MB/s. In addition, the CPU usage of Logstash also fluctuates significantly.
Observation: Logstash still processes logs after log generation is completed (at the 180th second), indicating that its processing capability has reached the bottleneck and a relatively large collection delay has occurred.
To further explore the collection bottlenecks of Logstash and iLogtail, we further refine and test different log generation rates. When the CPU usage is lower than 0.1, it is regarded as the end of collection processing, and the collection delay within 3 seconds is regarded as normal. Finally, according to our test, the processing bottleneck of Logstash in Scenario 2 is the log generation at 8MB/s. However, iLogtail can collect logs at 50MB/s almost without delay, and its resource usage remains relatively low (130% of CPU usage and 61MB of memory usage). The processing performance of iLogtail is six times as high as that of Logstash!
Observation: Similar to Scenario 1, you can observe similar fluctuations in the CPU usage of Logstash. However, the parsing of Scenario 3 is more complicated, so the glitches that exist in the CPU usage are now relatively sparse.
From the above three scenarios from simple to complex, we can analyze and draw some conclusions:
In the extreme test of Scenario 2, the maximum traffic that the iLogtail SPL can handle is six times as heavy as that for Logstash Filter. iLogtail SPL is written in C++ at the bottom layer and uses SIMD and other high-performance computing technologies to process log data in parallel. Therefore, it has reached a high level in terms of both performance and resource usage. However, many filter plug-ins of Logstash are written in Ruby. As a scripting language, its inherent limitations constrain its processing performance. At the same time, the design of Logstash filter Ruby plug-ins does not effectively leverage the streaming nature of log data. It is designed to process only one log at a time, and cannot process a group of logs in parallel, which is one of the reasons for its poor performance. This is especially pronounced in scenarios like Scenario 2 where the log entries are short but numerous. Of course, this can be solved by increasing the number of threads, but it will lead to higher resource usage.
Logstash allows users to write custom filter plug-ins in Ruby or Java. iLogtail SPL currently does not support user extensions, but in the future, it will provide UDF and other methods for users to easily implement custom processing needs. At the same time, the development team is also developing new features. In addition, iLogtail provides a plug-in processing mode similar to Logstash and supports C++ and Go programming. C++ processing plug-ins provide stronger performance while Go processing plug-ins provide greater flexibility.
In summary, iLogtail shows significant advantages over Logstash in log processing, thanks to its SPL processing mode. From the perspective of features, although Logstash provides extensive features through a variety of filter plug-ins, it is cumbersome and not intuitive to configure for advanced requirements such as complex logic processing, dynamic field operations, and nested JSON parsing. In contrast, iLogtail SPL not only integrates common data processing instructions but also incorporates SQL compatibility, which greatly improves the simplicity and readability of configurations and reduces the difficulty for users to get started.
In terms of performance, the advantages of iLogtail are more pronounced. Thanks to its core implementation in C++ and full utilization of modern hardware features such as SIMD, iLogtail SPL comprehensively surpasses the Ruby-based Logstash Filter in terms of resource usage, stability, and processing speed. Especially in high throughput scenarios, iLogtail can maintain stable high performance with low resource usage, which is particularly important for cost-sensitive enterprises and large-scale distributed systems.
In conclusion, iLogtail 2.0 with SPL processing mode is undoubtedly a more attractive choice for enterprises seeking efficient, flexible, and resource-friendly log processing solutions. It meets diverse log processing requirements while setting new industry standards in performance and usability, providing powerful tools for O&M teams to cope with increasing data processing challenges. With continuous technological iteration and ecosystem improvement, iLogtail aims to help enterprises better manage data streams, enhance operational efficiency, and accelerate digital transformation in the future.
494 posts | 48 followers
FollowAlibaba Cloud Native Community - May 13, 2024
Alibaba Cloud Native Community - March 29, 2024
Alibaba Cloud Community - October 19, 2021
Alibaba Cloud Community - August 2, 2022
Alibaba Cloud Community - June 27, 2023
Alibaba Cloud Native Community - May 31, 2024
494 posts | 48 followers
FollowManaged Service for Grafana displays a large amount of data in real time to provide an overview of business and O&M monitoring.
Learn MoreA unified, efficient, and secure platform that provides cloud-based O&M, access control, and operation audit.
Learn MoreAn all-in-one service for log-type data
Learn MorePlan and optimize your storage budget with flexible storage services
Learn MoreMore Posts by Alibaba Cloud Native Community