Performance testing tools

Alibaba Cloud PTS

Alibaba Cloud Performance Testing Service (PTS) is a commercial performance testing tool that allows you to conduct performance testing tasks on demand. PTS can support up to millions of concurrent virtual users and tens of millions of transactions per second (TPS) and is fully compatible with JMeter. PTS provides various features, including scenario orchestration, API debugging, traffic customization, and traffic recording. PTS allows you to quickly create performance testing scripts to precisely simulate different levels of user access to business systems. This helps efficiently improve the performance and stability of the system.

Alibaba Cloud PTS has the following features:

• O&M-free and ready to use out of the box

  PTS is a software as a service (SaaS)-based service that can generate millions of concurrent virtual users and tens of millions of TPS.

• Multi-protocol compatibility

  Supports various mainstream protocols including HTTP1.1, HTTP2, JDBC, MQTT, Kafka, RocketMQ, Redis, WebSocket, RMTP, HLS, TCP, UDP, Spring Cloud, Dubbo, and gRPC.

• Traffic customization

  Allows you to customize traffic based on geographic regions, specific telecom operators, and IPv6.

• Stability and security

  The engine developed by Alibaba operates based on a semaphore model to effectively manage high concurrency. It has been tested through extensive use during major events such as the Double 11 shopping festival of Alibaba, which proves its ability to maintain stability and security even under billion-level traffic loads.

• One-stop performance testing solution

  PTS allows you to build complex performance testing scenarios without the need for coding. The PTS solution covers the lifecycle of performance testing, including creating testing scenarios, setting up testing models, starting testing, analyzing and identifying issues, and reviewing reports.

• 100% compatibility with JMeter

Apache JMeter

JMeter is a Java-based performance testing tool developed by Apache. Apache JMeter has the following features:

• Distributed load testing.

• Provides a user-friendly graphical interface from which you can create and orchestrate complex test plans and use advanced features such as assertions and logical controllers to handle complex testing scenarios.

• High extensibility: Developers and testers can create custom plugins to meet specific load testing requirements.

• Strong community support: JMeter is supported by a strong open-source community with active developer participation.

• Allows you to view resource metrics and performance reports. However, the range of available monitoring and reporting metrics is limited.

• Due to the dependence of JMeter on the Java Virtual Machine (JVM), a single JMeter instance cannot handle extremely high concurrency levels. JMeter supports only the concurrency testing model and does not support the throughput testing model.

• Challenges in large-scale distributed testing: Scaling JMeter for large production environments can be costly and complex even if JMeter provides distributed testing capabilities.

• Lack of test management features: JMeter does not provide features for managing test cases and load testing scripts.

ApacheBench

ApacheBench (ab) is a command-line tool designed for performance testing of HTTP-based applications and systems. ApacheBench has the following features:

• Excellent scalability.

• Limited protocol support: primarily supports the HTTP protocol and does not support other mainstream protocols such as HTTPS and WebSocket.

• Request control: allows you to manage the total number of requests, concurrency level, and testing duration.

• User-friendly: Easy to use as a command-line tool.

• Single-node testing: operates as a standalone tool without distributed load testing capabilities or a graphical user interface.

• Lacks advanced features such as scenario management, session management, and complex business-related testing.

• Single-domain testing: Each testing session is limited to a single domain or IP address.

• Basic metrics: provides limited statistical metrics during the testing process and does not capture system load or other detailed performance indicators.

wrk

wrk is a benchmark testing tool specifically designed for HTTP-based systems and applications. wrk has the following features:

• As a lightweight performance testing tool, wrk is easy to install and set up.

• Easy to use.

• Uses an asynchronous event-driven framework and supports high concurrency on a single machine.

• Operates as a standalone tool without distributed load testing capabilities.

• Supports only the HTTP protocol.

• Does not provide a graphical user interface or features like workflow orchestration or assertions, which makes wrk less suitable for complex testing scenarios.

Summary