This topic describes how to perform stress testing for the maximum number of connections to multiple ApsaraDB for MongoDB replica set instances of different specifications. The testing is performed by access to the ApsaraDB for MongoDB instances from an Elastic Compute Service (ECS) instance.
Test environment
Create an ECS instance and ApsaraDB for MongoDB replica set instances of different specifications. For more information, see Create a replica set instance and Create an ECS instance.
The following table describes the configurations of the ECS instance and the ApsaraDB for MongoDB replica set instances used in the testing.
Configuration item | ECS instance | ApsaraDB for MongoDB instances that use cloud disks | ApsaraDB for MongoDB instances that use local disks |
Region and zone | Beijing Zone H | Beijing Zone H | Beijing Zone H |
Network type | Virtual Private Cloud (VPC) | VPC | VPC |
Instance family | c6e, compute-optimized instance family with enhanced performance | General-purpose and dedicated | General-purpose and dedicated |
Instance type | ecs.c6e.2xlarge | Three available instance types. For more information, see Test results. | Two available instance types. For more information, see Test results. |
Storage type | Enterprise SSD (ESSD) AutoPL disks | ESSDs | Local disks |
Image or engine version | Alibaba Cloud Linux 3.2104 LTS 64-bit | 4.19.91-26.al7.x86_64 | 3.10.0-327.ali2017.alios7.x86_64 |
Kernel version | N/A |
|
|
The ApsaraDB for MongoDB instances used in the testing adopt the three-node architecture that consists of a primary node, a secondary node, and a hidden node.
The ECS instance and the ApsaraDB for MongoDB instances used in the testing are deployed in the same zone of the same region, and the average Round-Trip Time (RTT) is 0.103 ms.
Test tool
The open-source Yahoo Cloud Serving Benchmark (YCSB) 0.17.0 tool is used in the testing.
NoteYCSB is a Java tool that can be used to benchmark the performance of multiple types of databases. For more information about how to install and use YCSB, see YCSB.
A custom connection stress testing program is used in the testing. For more information, see Additional information about stress testing for 96,000 connections.
Test method
- Add the primary private IP address of the ECS instance to a whitelist of the ApsaraDB for MongoDB instance. For more information, see Modify an IP address whitelist. Note You can log on to the ECS console and view the Primary Private IP Address of the ECS instance in the Network Information section of the Instance Details page.
- Connect to the ECS instance. For more information, see Create and manage an ECS instance by using the ECS console (express version).
Use the YCSB tool to load test data.
// Run the ycsb run command. ./bin/ycsb.sh load mongodb -s -p workload=site.ycsb.workloads.CoreWorkload -p recordcount=10000000 -p mongodb.url="mongodb://test:****@dds-bp13e84d11****.mongodb.rds.aliyuncs.com:3717/admin" -p table=test -threads 8
Modify the following settings:
recordcount=1000000
: the total amount of data loaded to an ApsaraDB for MongoDB instance used in the testing.mongodb.url="mongodb://test:****@dds-bp13e84d11****.mongodb.rds.aliyuncs.com:3717/admin"
: the connection string of an ApsaraDB for MongoDB instance used in the testing. In the testing, the database account is test and the database is admin.NoteYou can obtain the endpoint in the Internal Connections - VPC section of the Database Connections page in the ApsaraDB for MongoDB console.
threads 8
: the number of concurrent threads on the client.
Run the following command to perform performance stress testing:
./bin/ycsb.sh run mongodb -s -p workload=site.ycsb.workloads.CoreWorkload -p recordcount=10000000 -p operationcount=5000000 -p readproportion=50 -p updateproportion=50 -p requestdistribution=zipfian -p mongodb.url="mongodb://test:****@dds-bp13e84d11****.mongodb.rds.aliyuncs.com:3717/admin&maxPoolSize=8000" -p table=test -threads 8000
Modify the following settings:
recordcount=1000000
: the total amount of data loaded to an ApsaraDB for MongoDB instance used in the testing.operationcount=5000000
: the total number of read and write operations.insertproportion=0
: the ratio of load operations.readproportion=50
: the ratio of read operations.updateproportion=50
: the ratio of update operations.mongodb.url="mongodb://test:****@dds-bp13e84d11****.mongodb.rds.aliyuncs.com:3717/admin"
: the connection string of an ApsaraDB for MongoDB instance used in the testing. In the testing, the database account is test and the database is admin.NoteYou can obtain the endpoint in the Internal Connections - VPC section of the Database Connections page in the ApsaraDB for MongoDB console.
When you execute the preceding statement, you must configure the
maxPoolSize
parameter. Otherwise, the default number of connections is 100. In this case, theMongoWaitQueueFullException
error is returned during the testing, and the expected maximum number of connections cannot be reached.
For more information about how to view the monitoring data of the ECS instance and an ApsaraDB for MongoDB instance used in the testing, see Basic monitoring.
On the Basic Monitoring tab, select a period of time for the stress testing to view the CPU Utilization, Memory Usage, QPS, Connections, and Connection Utilization metrics.
Additional information about stress testing for 96,000 connections
YCSB testing depend on the Java environment. The maximum heap memory size of a Java virtual machine (JVM) is limited. If the testing uses more than 20,000 concurrent threads, the Cannot allocate memory
error is returned, which interrupts the test.
After you increase the value of the JAVA_OPTS
parameter, the error still occurs. To resolve the issue, use a custom connection stress testing program. The stress testing program cyclically generates several threads. Each thread generates a MongoClient. After the MongoClient performs a query, the MongoClient maintains connections for a period of time without being released.
A single machine on which a stress testing client runs has a limited number of ports. Therefore, the machine cannot meet the requirements of stress testing for connections (up to 96,000 connections) with the specifications of 32 cores and 128 GB of memory. In this case, you must run the same stress testing program on multiple machines.
Execute the following Bash statement to query the current port range of a machine:
sysctl net.ipv4.ip_local_port_range
Sample response:
net.ipv4.ip_local_port_range = 40000 65535
Execute the following statement to expand the port range of a machine and then perform connection stress testing:
sudo sysctl -w net.ipv4.ip_local_port_range="10240 65535"
Test results
Instances that use local disks
General-purpose instance that has 16 cores and 64 GB of memory
Maximum number of connections: 32,000
QPS | Connections | Connection utilization | CPU utilization | Memory usage |
If you specify a collection interval at the minute level on the Basic Monitoring tab, the number of connections at several points in time is not monitored for a general-purpose instance that has 16 cores and 64 GB of memory due to the timeouts of collection commands caused by 100% CPU utilization. In this case, minute-level valleys occur, and the actual number of connections at the points in time remains 32,000.
Dedicated instance that has 2 cores and 16 GB of memory
Maximum number of connections: 8,000
QPS | Connections | Connection utilization | CPU utilization | Memory usage |
Instances that uses ESSD
Dedicated instance that has 4 cores and 8 GB of memory
Maximum number of connections: 8,000
QPS | Connections | Connection utilization | CPU utilization | Memory usage |
If you select a collection interval at the minute level on the Basic Monitoring tab, the maximum number of connections reached within the interval cannot be displayed on the tab. The actual number of connections reaches 8,000. This can be confirmed by configuring granular monitoring or viewing the connections
sub-document in the result of the serverStatus
statement.
Dedicated instance that has 32 cores and 128 GB of memory
Maximum number of connections: 96,000
QPS | Connections | Connection utilization | CPU utilization | Memory usage |
The test tool used when the maximum number of connections is 96,000 is different from the tools used for the test tool used when the maximum numbers of connections is 8,000 and 16,000. Therefore, the preceding monitoring screenshots related to QPS, CPU utilization, and memory usage have discrepancy.
General-purpose instance that has 8 cores and 32 GB of memory
Maximum number of connections: 16,000
QPS | Connections | Connection utilization | CPU utilization | Memory usage |
Summary
ApsaraDB for MongoDB replica set instances of different specifications and storage types can reach their maximum number of connections corresponding to their specifications.
After the maximum number of connections is reached, ApsaraDB for MongoDB rejects subsequent connections. The requests have a high latency or are stuck on your application due to the failed connection establishment.
A larger number of concurrent connections consume more resources such as CPU and memory. We recommend that you adjust the number of connections to your instance based on your business requirements.