Stress testing of their maximum connections to replica set instances - ApsaraDB for MongoDB

This topic describes how to perform stress testing for the maximum number of connections to multiple ApsaraDB for MongoDB replica set instances of different specifications. The testing is performed by access to the ApsaraDB for MongoDB instances from an Elastic Compute Service (ECS) instance.

Test environment

Create an ECS instance and ApsaraDB for MongoDB replica set instances of different specifications. For more information, see Create a replica set instance and Create an ECS instance.

The following table describes the configurations of the ECS instance and the ApsaraDB for MongoDB replica set instances used in the testing.

Configuration item	ECS instance	ApsaraDB for MongoDB instances that use cloud disks	ApsaraDB for MongoDB instances that use local disks
Region and zone	Beijing Zone H	Beijing Zone H	Beijing Zone H
Network type	Virtual Private Cloud (VPC)	VPC	VPC
Instance family	c6e, compute-optimized instance family with enhanced performance	General-purpose and dedicated	General-purpose and dedicated
Instance type	ecs.c6e.2xlarge	Three available instance types. For more information, see Test results.	Two available instance types. For more information, see Test results.
Storage type	Enterprise SSD (ESSD) AutoPL disks	ESSDs	Local disks
Image or engine version	Alibaba Cloud Linux 3.2104 LTS 64-bit	4.19.91-26.al7.x86_64	3.10.0-327.ali2017.alios7.x86_64
Kernel version	N/A	Major version: MongoDB 4.4 Baseline of minor version: 4.4.28	Major version: MongoDB 4.2 Baseline of minor version: 4.2.23

The ApsaraDB for MongoDB instances used in the testing adopt the three-node architecture that consists of a primary node, a secondary node, and a hidden node.
The ECS instance and the ApsaraDB for MongoDB instances used in the testing are deployed in the same zone of the same region, and the average Round-Trip Time (RTT) is 0.103 ms.

Test tool

The open-source Yahoo Cloud Serving Benchmark (YCSB) 0.17.0 tool is used in the testing.
Note
YCSB is a Java tool that can be used to benchmark the performance of multiple types of databases. For more information about how to install and use YCSB, see YCSB.
A custom connection stress testing program is used in the testing. For more information, see Additional information about stress testing for 96,000 connections.

Test method

Add the primary private IP address of the ECS instance to a whitelist of the ApsaraDB for MongoDB instance. For more information, see Modify an IP address whitelist.
Note You can log on to the ECS console and view the Primary Private IP Address of the ECS instance in the Network Information section of the Instance Details page.
Connect to the ECS instance. For more information, see Create and manage an ECS instance by using the ECS console (express version).
Use the YCSB tool to load test data.
```
// Run the ycsb run command.
./bin/ycsb.sh load mongodb -s -p workload=site.ycsb.workloads.CoreWorkload -p recordcount=10000000 -p mongodb.url="mongodb://test:****@dds-bp13e84d11****.mongodb.rds.aliyuncs.com:3717/admin" -p table=test -threads 8
```
Modify the following settings:
- recordcount=1000000: the total amount of data loaded to an ApsaraDB for MongoDB instance used in the testing.
- mongodb.url="mongodb://test:****@dds-bp13e84d11****.mongodb.rds.aliyuncs.com:3717/admin": the connection string of an ApsaraDB for MongoDB instance used in the testing. In the testing, the database account is test and the database is admin.
  Note
  You can obtain the endpoint in the Internal Connections - VPC section of the Database Connections page in the ApsaraDB for MongoDB console.
- threads 8: the number of concurrent threads on the client.
Run the following command to perform performance stress testing:
```
./bin/ycsb.sh run mongodb -s -p workload=site.ycsb.workloads.CoreWorkload -p recordcount=10000000 -p operationcount=5000000 -p readproportion=50 -p updateproportion=50 -p requestdistribution=zipfian -p mongodb.url="mongodb://test:****@dds-bp13e84d11****.mongodb.rds.aliyuncs.com:3717/admin&maxPoolSize=8000" -p table=test -threads 8000
```
Modify the following settings:
- recordcount=1000000: the total amount of data loaded to an ApsaraDB for MongoDB instance used in the testing.
- operationcount=5000000: the total number of read and write operations.
- insertproportion=0: the ratio of load operations.
- readproportion=50: the ratio of read operations.
- updateproportion=50: the ratio of update operations.
- mongodb.url="mongodb://test:****@dds-bp13e84d11****.mongodb.rds.aliyuncs.com:3717/admin": the connection string of an ApsaraDB for MongoDB instance used in the testing. In the testing, the database account is test and the database is admin.
  Note
  You can obtain the endpoint in the Internal Connections - VPC section of the Database Connections page in the ApsaraDB for MongoDB console.
  When you execute the preceding statement, you must configure the maxPoolSize parameter. Otherwise, the default number of connections is 100. In this case, the MongoWaitQueueFullException error is returned during the testing, and the expected maximum number of connections cannot be reached.
For more information about how to view the monitoring data of the ECS instance and an ApsaraDB for MongoDB instance used in the testing, see Basic monitoring.
On the Basic Monitoring tab, select a period of time for the stress testing to view the CPU Utilization, Memory Usage, QPS, Connections, and Connection Utilization metrics.

Additional information about stress testing for 96,000 connections

YCSB testing depend on the Java environment. The maximum heap memory size of a Java virtual machine (JVM) is limited. If the testing uses more than 20,000 concurrent threads, the Cannot allocate memory error is returned, which interrupts the test.

After you increase the value of the JAVA_OPTS parameter, the error still occurs. To resolve the issue, use a custom connection stress testing program. The stress testing program cyclically generates several threads. Each thread generates a MongoClient. After the MongoClient performs a query, the MongoClient maintains connections for a period of time without being released.

A single machine on which a stress testing client runs has a limited number of ports. Therefore, the machine cannot meet the requirements of stress testing for connections (up to 96,000 connections) with the specifications of 32 cores and 128 GB of memory. In this case, you must run the same stress testing program on multiple machines.

Execute the following Bash statement to query the current port range of a machine:

sysctl net.ipv4.ip_local_port_range

Sample response:

net.ipv4.ip_local_port_range = 40000    65535

Execute the following statement to expand the port range of a machine and then perform connection stress testing:

sudo sysctl -w net.ipv4.ip_local_port_range="10240 65535"

Test results

Instances that use local disks

General-purpose instance that has 16 cores and 64 GB of memory

Maximum number of connections: 32,000

QPS	Connections	Connection utilization	CPU utilization	Memory usage

Note

If you specify a collection interval at the minute level on the Basic Monitoring tab, the number of connections at several points in time is not monitored for a general-purpose instance that has 16 cores and 64 GB of memory due to the timeouts of collection commands caused by 100% CPU utilization. In this case, minute-level valleys occur, and the actual number of connections at the points in time remains 32,000.

Dedicated instance that has 2 cores and 16 GB of memory

Maximum number of connections: 8,000

QPS	Connections	Connection utilization	CPU utilization	Memory usage

Instances that uses ESSD

Dedicated instance that has 4 cores and 8 GB of memory

Maximum number of connections: 8,000

QPS	Connections	Connection utilization	CPU utilization	Memory usage

Note

If you select a collection interval at the minute level on the Basic Monitoring tab, the maximum number of connections reached within the interval cannot be displayed on the tab. The actual number of connections reaches 8,000. This can be confirmed by configuring granular monitoring or viewing the connections sub-document in the result of the serverStatus statement.

Dedicated instance that has 32 cores and 128 GB of memory

Maximum number of connections: 96,000

QPS	Connections	Connection utilization	CPU utilization	Memory usage

Note

The test tool used when the maximum number of connections is 96,000 is different from the tools used for the test tool used when the maximum numbers of connections is 8,000 and 16,000. Therefore, the preceding monitoring screenshots related to QPS, CPU utilization, and memory usage have discrepancy.

General-purpose instance that has 8 cores and 32 GB of memory

Maximum number of connections: 16,000

QPS	Connections	Connection utilization	CPU utilization	Memory usage

Summary

ApsaraDB for MongoDB replica set instances of different specifications and storage types can reach their maximum number of connections corresponding to their specifications.
After the maximum number of connections is reached, ApsaraDB for MongoDB rejects subsequent connections. The requests have a high latency or are stuck on your application due to the failed connection establishment.
A larger number of concurrent connections consume more resources such as CPU and memory. We recommend that you adjust the number of connections to your instance based on your business requirements.