The Application Real-Time Monitoring Service (ARMS) agent enhances bytecodes during application runtime to manage application performance. Similar to other performance management solutions implemented through bytecode enhancement, the ARMS agent for Java brings a certain amount of performance overhead. However, the ARMS team has adopted a number of techniques to reduce the performance overhead of the agent without compromising application stability. In this test report, scenarios with different traffic are simulated to test the performance overhead of the ARMS agent for Java V4.x. Before you connect your application to Application Monitoring, we recommend that you refer to this report to perform a comprehensive evaluation.
Test scenarios
Flowchart
Developed in Spring MVC, the Java application will access MySQL and Redis based on the requests sent from Alibaba Cloud Performance Testing (PTS). For the ${mall-gateway}/case/api/v1/mysql/execute request, the application accesses MySQL. For the ${mall-gateway}/case/api/v1/redis/execute request, the application accesses Redis. Each type of requests account for 50% of the queries per second (QPS).
Test environment
The stress testing source is provided by PTS.
The Java application, MySQL, and Redis are all deployed in the same Alibaba Cloud Container Service for Kubernetes (ACK) cluster. The instance type of nodes is ecs.u1-c1m2.2xlarge, and the operating system version of the nodes is CentOS Linux release 7.9.2009 (Core).
Each pod of the application has 2 cores, 4 GB of memory, and two replicas.
The version of the ARMS agent for Java is 4.1.11.
Test procedure
Perform three stress tests at 500, 1,000, and 2,000 QPS. Each stress test lasts for 1 hour. Before each stress test, warm up the Java application at 100 QPS for 3 minutes. The test result will be used as baseline performance metrics.
Install the ARMS agent for Java V4.1.11. Configure a sampling rate of 10% in the sampling policy, and repeat the stress tests in Step 1 to compare the differences in the CPU overhead, memory overhead, and response time of the Java application.
Install the ARMS agent for Java V4.1.11. Configure a sampling rate of 100% in the sampling policy, and repeat the stress tests in Step 1 to compare the differences in the CPU overhead, memory overhead, and response time of the Java application.
For more information about how to configure the sampling policy, see Select a trace sampling mode for the ARMS agent V3.2.8 and later.
The basic features of Application Monitoring are enabled, including metrics, traces, and quantiles. All plug-ins are also enabled.
Baseline performance metrics
Item | CPU | Memory | Response time (Unit: milliseconds) |
500 QPS | 5.9% | 12.1% | 57.4 |
1,000 QPS | 11.1% | 12.3% | 63.2 |
2,000 QPS | 20.9% | 12.7% | 70.3 |
The CPU metric represents the percentage of CPU used by pods to the total CPU (2 cores).
The memory metric represents the percentage of memory used by pods to the total memory. Because the memory used by pods grows naturally until it reaches the requests value, the report uses the actual memory usage at the end of the stress tests.
The response time metric represents the average response time of requests. Unit: milliseconds.
Performance metrics of the ARMS agent for Java V4.x
Item | 10% sampling rate | 100% sampling rate | ||||
CPU | Memory | Response time (Unit: milliseconds) | CPU | Memory | Response time (Unit: milliseconds) | |
500 QPS | 7.5% | 14.8% | 58.1 | 8.9% | 15.2% | 58.2 |
1,000 QPS | 14.6% | 15.3% | 64.6 | 15.5% | 15.9% | 64.8 |
2,000 QPS | 27.3% | 16.7% | 74.2 | 27.8% | 17.4% | 75.8 |
The relationship between the sampling rate and both the CPU and QPS is nonlinear. By default, there is a upper limit for trace collection per second.
Performance overhead of the ARMS agent for Java V4.x
Item | 10% sampling rate | 100% sampling rate | ||||
CPU | Memory | Response time (Unit: milliseconds) | CPU | Memory | Response time (Unit: milliseconds) | |
500 QPS | +1.6% | +2.7% | +0.7 | +3.0% | +3.1% | +0.8 |
1,000 QPS | +3.5% | +3.0% | +1.4 | +4.4% | +3.6% | +1.6 |
2,000 QPS | +6.4% | +4.0% | +3.9 | +6.9% | +4.7% | +5.5 |
Conclusion
The additional CPU and memory overhead caused by the ARMS agent for Java V4.x are within 10%.
The ARMS agent for Java V4.x has small impact on the response time (10%), which increases only by 1 millisecond at 1,000 QPS.
The performance overhead at a 100% fixed sampling rate are slightly greater than those at a 10% fixed sampling rate.