Performance comparison between iLogtail and Filebeat

One. Introduction

Some time ago, iLogtail [1] Ali Tens of Thousands of Instances Observable Collector was open sourced, which introduced that the collection performance of iLogtail can reach 100MB/s per core, which has a performance advantage of 5-10 times compared with the open source collection Agent. Many friends are curious about the specific performance data and resource consumption of iLogtail. This article will compare the Agent FileBeat, which is currently widely used in the industry and has relatively better performance, to test how the two Agents perform under different stress scenarios.

Two: test test description

With the popularization of Kubernetes, the demand for log collection under Kubernetes is becoming more and more normalized. Therefore, the following sections will conduct a comparative test of container standard output stream collection and static file collection in the container (for those who use static file collection, please refer to static file collection in the container. In the comparison test, the pure static file collection of iLogtail is slightly better than the static file collection in the container of test 2). The details of the test items are as follows:

Experiment 1: Constant collection configuration 4, comparison of standard output stream collection performance of Filebeat & iLogtail at raw log generation rates of 1M/s, 2M/s, and 3M/s.
Experiment 2: Constant collection configuration 4, Filebeat & iLogtail performance comparison of in-container file collection at raw log generation rates of 1M/s, 2M/s, and 3M/s.
In a real production environment, the operability of log collection components is also very important, which is convenient for operation and maintenance and later upgrades. Compared with Sidecar mode, it is more common to deploy collection components in Daemonset mode under K8s. However, due to the fact that Daemonset distributes the collection configuration of the entire cluster to each collection node at the same time, the working configuration of a single collection node must be smaller than the number of full collection configurations. Therefore, we will conduct the following two parts of the experiment to verify whether the expansion of the collection configuration will Affect the working efficiency of the collector:

Experiment 3: A constant input rate of 3M/s, a comparison of the standard output stream acquisition performance of Filebeat & iLogtail under the acquisition configuration of 50, 100, 500, and 1000 copies.
Experiment 4: A constant input rate of 3M/s, Filebeat & iLogtail in the container file acquisition performance comparison under the acquisition configuration of 50, 100, 500, and 1000 copies.

Finally, the iLogtail high-flow pressure test will be performed, as follows:

Experiment 5: Standard output stream acquisition performance of iLogtail at 5M/s, 10M/s, 10M/s, 40M/s.
Experiment 6: iLogtail's in-container file collection performance at 5M/s, 10M/s, 10M/s, and 40M/s.

Three: Test environment

All the collected environment data are stored in [2]. Interested students can do the whole comparison test experiment by themselves. The following sections describe the specific configurations of different collection modes. If you only care about the collection and comparison results, you can skip this part and continue reading .

1. Environment

Operating environment: Alibaba Cloud ACK Pro version
Node configuration: ecs.g6.xlarge (4 vCPU 16GB) disk ESSD
Underlying container: Containerd
iLogtail version: 1.0.28
FileBeat version: v7.16.2

2. Data source

For the data source, we first remove the difference caused by regular parsing or multi-line splicing capabilities, and only compare the most basic single-line collection. The data generation source simulates the nginx access log, and the size of a single log is 283B. The following configuration describes 1000 input source at bar/s rate.

3. Filebeat standard output stream acquisition configuration

Filebeat natively supports container file collection, adding kubernetes meta information through the add_kubernetes_metadata component, in order to avoid performance differences caused by the output component, discarding data through the drop_event plug-in to avoid output, the filebeat test configuration is as follows (harvester_buffer_size is adjusted to 512K, filebeat.registry.flush: 30s, the queue.mem parameter is appropriately expanded to increase throughput).

4. Filebeat container file collection configuration

Filebeat does not natively support file collection in the container, so it is necessary to manually mount the log printing path on the host HostPath. Here we use the subPath and DirectoryOrCreate functions to separate the service printing paths. The following is to simulate the independence of different service log printing paths.

filebeat uses the basic log reading function to read logs under the /testlog path. In order to avoid performance differences caused by output components, data is discarded through the drop_event plug-in to avoid output. The test configuration is as follows (harvester_buffer_size is adjusted to 512K, filebeat.registry.flush: 30s, the queue.mem parameter is appropriately expanded to increase throughput).

5. iLogtail standard output stream collection configuration

iLogtail also natively supports standard output stream collection. The service_docker_stdout component already extracts kubernetes meta information. In order to avoid performance differences caused by output components, all logs are filtered through processor_filter_regex. The test configuration is as follows.

6. iLogtail container file collection configuration

iLogtail natively supports file collection in the container, but because the metadata collected in the file exists in the tag, there is no filter plug-in for now. In order to avoid performance differences caused by the output component, we use an empty output plug-in for output. The test configuration is as follows.

Four: Filebeat and iLogtail comparison test

The comparison items between Filebeat and iLogtail mainly include the following: standard output stream collection performance, file collection performance in the container, standard output stream multi-user configuration performance, file multi-user configuration performance in the container, and large-traffic collection performance.

1. Standard output stream acquisition performance comparison

Input data source: 283B/s, the underlying container contianerd, the standard output stream is expanded to 328B, a total of 4 input sources:

1M/s input log 3700 records/s,
2M/s input log 7400 records/s,
3M/s Input log entries 11100 entries/s.

The following shows the performance comparison of different acquisitions of the standard output stream. It can be seen that iLogtail has a ten-fold performance advantage compared to Filebeat (the percentage of CPU is the percentage of single core):

The following shows the memory comparison of different collections of the standard output stream. It can be seen that the overall memory of logtail and filebeat is not much different, and there is no memory surge with the increase of collection traffic:

2. Comparison of file collection performance in the container

Input data source: 283B/s, 4 input sources in total:

1M/s input log 3700 records/s,
2M/s input log 7400 records/s,
3M/s Input log entries 11100 entries/s.

The following shows the performance comparison of different collection files in the container. Since the files in the Filebeat container share the collection components with the container collection and omit the Kubernetes meta-related components, there is a big performance improvement compared to the standard output stream collection. The file in the container of iLogtail The collection adopts the Polling + inotify mechanism, which also has a performance improvement compared to the standard output stream collection of the container, but it can be seen that iLogtail has a 5-fold performance advantage compared to Filebeat (the percentage of CPU is the percentage of single core):

The following shows the memory comparison of different collections of the standard output stream. It can be seen that the overall memory of logtail and filebeat is not much different, and there is no memory surge with the increase of collection traffic:

3. Comparison of expansion performance of collection configuration

The expansion performance comparison of the collection configuration, the input source is set to 4, the total input rate is 3M/s, and the 50 collection configuration, the 100 collection configuration, the 500 collection configuration, and the 1000 collection configuration are respectively compared.

Standard output stream acquisition configuration expansion comparison

The following shows the performance comparison of different collections of standard output streams. It can be seen that Filebeat has a lot of regular matching work under the standard output stream collection path var/log/containers because the bottom layer of container collection and static file collection share the same static file collection logic. , it can be seen that although the amount of collected data has not increased due to the increase in the collection configuration, the CPU consumption has increased by 10%+, and iLogtail has globally shared the container path discovery mechanism for the container collection model, thus avoiding the performance loss caused by regular logic (the percentage of CPU as a percentage of a single core).

In terms of memory expansion, it can be seen that both Filebeat and iLogtail have memory expansion due to the increase in collection configuration, but the expansion size of both is within an acceptable range.

In-container file collection configuration expansion comparison

The following shows the performance comparison of different collectors for file collection in the container. It can be seen that Filebeat static file collection avoids the regularization of the standard output stream common path. Slightly better than standard output stream capture performance (the percentage of CPU is the percentage of single core).

In terms of memory expansion, it can also be seen that both Filebeat and iLogtail have memory expansion due to the increase in collection configuration, but the expansion size of both is within an acceptable range.

4. iLogtail collection performance test

Since FileBeat has a collection delay problem in scenarios with a large amount of logs, the following scenarios are only tested for iLogtail, and the standard output stream of the container and the file in the container are collected for iLogtail at 5M/s, 10M/s, and 20M/s respectively performance pressure test.

Number of input sources: 10
Single log size 283B
5M/s corresponds to a log rate of 18526 records/s, and a single input source produces a rate of 1852 records/s
10M/s corresponds to a log rate of 37052 records/s, and a single input source produces a rate of 3705 records/s
20M/s corresponds to a log rate of 74104 records/s, and a single input source produces a rate of 7410 records/s
40M/s corresponds to a log rate of 148208 records/s, and a single input source generates a rate of 14820 records/s
Similar to the above experiment, it can be seen that in terms of CPU consumption, container file collection is slightly better than container standard output stream collection performance (the percentage of CPU is the percentage of single core), mainly due to the underlying Polling + inotify mechanism of container file collection.

In terms of memory, since the collection of standard output streams mainly depends on GO, and the collection of container files mainly depends on C, due to the existence of the GC mechanism, as the speed increases, the memory consumed by standard output stream collection will gradually exceed that of file collection in the container. The memory consumed.


5. Comparative summary

Five: Why is there a huge difference between the standard output of the Filebeat container and the file collection?

Through the above experiments, we can see that FIlebeat has a large CPU difference in different working modes. The following flame graph can be obtained by pprof collected from the standard output stream of the dump container. It can be seen that the add_kubernets_meta plug-in collected by the Filebeat container is a performance bottleneck. The add_kubernets_meta adopts the mode of listening to the api-server with each node, and there is also the problem of pressure on the api-server.

The kubernetes meta obtained by iLogtail is completely compatible with the kubernetes CRI protocol, and the meta data is read directly through the kubernets sandbox, which ensures the high-performance collection efficiency of iLogtail.

Six: iLogtail DaemonSet scene optimization

Through the above comparison, we can see that iLogtail has excellent memory and CPU consumption compared with Filebeat. Some friends may wonder why iLogtail has such extreme performance. The following mainly explains the optimization of iLogtail Daemonset scenarios, and how standard output streams compare It has 10 times the performance of Filebeat.

First of all, for the standard output stream scenario, compared to other open source collectors, such as Filebeat or Fluentd. Generally, the collection of container standard output stream files is realized by monitoring var/log/containers or /var/log/pods/. For example, the path structure of /var/log/pods/ is: /var/log/pods/__//, through this path to reuse the static file collection mode of the physical machine for collection.

As for iLogtail, it fully supports containerization. Through the discovery mechanism, iLogtail maintains a list of Node node containers globally, and monitors and maintains this container list in real time. When we have a list of containers, we have the following advantages:

The collection path no longer depends on the static configuration path, and the collection source can be dynamically selected based on the container label, thereby simplifying the user access cost.

The dynamic path of the node automatically mounted by the container can be detected according to the container meta-information, so iLogtail can collect files in the container without mounting, while collectors such as Filebeat need to mount the path in the container to the path of the host machine, and then collect static files .

For new access collection configurations, the history container list is reused to quickly access collections. For empty collection configurations, due to the existence of the container discovery global sharing mechanism, the existence of an empty round-robin monitoring path mechanism is avoided, thereby ensuring that In such a highly dynamic environment, the cost of iLogtail operability can be controlled.

Seven: Conclusion

To sum up, in the highly dynamic Kubernetes environment, iLogtail will not cause large memory expansion due to multiple configuration problems caused by the deployment model of Daemonset, and in terms of static file collection, iLogtail has about 5 times Performance advantage, and for standard output stream collection, due to iLogtail's collection mechanism, iLogtail has a performance advantage of about 10 times. However, compared with old open source products such as Filebeat or Fluentd, there is still a lot of lack of documentation and community building. Friends who are interested in iLogtail are welcome to join in and jointly create an easy-to-use and high-performance iLogtail product.