eas-benmark is a versatile tool for distributed stress testing that is embedded in Elastic Algorithm Service (EAS). With eas-benchmark, you can run dynamic stress tests, view real-time monitoring data that is accurate to the second, and generate a visualized report after the tests are completed. This helps you assess the performance of the service that you deploy in EAS. This topic describes how to create and manage stress testing tasks.
Prerequisites
A model service is deployed. For more information, see Model service deployment by using the PAI console.
Background information
You can log on to the EASCMD client and run the bench
commands to create and manage stress testing tasks. For information about how to create and manage stress testing tasks, see the "Create a stress testing task" and "Manage stress testing tasks" sections of this topic. For information about how to log on to the EASCMD client, see Download the EASCMD client and complete identity authentication.
Create a stress testing task
Use the EASCMD client
Run the bench create
command to create a stress testing task. After the task is created, you can use the returned URL to view the real-time monitoring data. In this example, the 64-bit Windows version of the EASCMD client is used.
eascmdwin64.exe bench create [bench_desc_json]
bench_desc_json specifies the name of the JSON configuration file for the stress testing task. Sample configuration:
{
"service": {
"serviceName": "xgb_test"
},
"data": {
"content": "W1sxLDAsMCwwLDEsMSwwLDEsMCwxLDEsMCwwLDEsMCwxLDAsMSwwLDAsMSwxLDEsMCwxLDEsMCwwLDAsMSwxLDEsMCwxLDEsMSwxLDAsMSwxLDEsMCwxLDAsMCwwLDEsMSwwLDAsMCwxLDAsMSwwLDEsMCwwLDEsMCwwLDEsMCwxLDAsMCwxLDAsMCwwLDAsMSwwLDEsMCwxLDAsMCwxLDEsMSwwLDAsMSwwLDAsMCwwLDEsMSwxLDAsMSwxLDAsMCwxLDAsMSwwLDEsMSwxLDEsMCwxLDAsMCwxLDEsMSwxLDAsMCwwLDEsMSwwXV0K"
}
}
You can use the path parameter to specify the path of multiple Object Storage Service (OSS) objects that store stress testing data, as shown in the following configuration. You can also set the path parameter to the path of the ZIP file that contains the stress testing data in the OSS://XX.zip format.
{
"service": {
"serviceName": "xgb_test"
},
"data": {
"path": "oss://examplebucket/test1.bin,oss://examplebucket/test2.bin"
}
}
The following table describes the parameters in the JSON configuration file.
Parameter | Nested parameter | Required | Description |
service | serviceName | Yes | The name of the service for which you want to create the stress testing task. |
data | content | No | A single entry of stress testing data in the Base64-encoded string format. If you want to configure multiple data entries, specify the path parameter. For information about the format and file type of stress testing data, see the "Stress testing data" section of this topic. |
path | No | The path of the stress testing data. You can specify an HTTP or OSS path. You can specify multiple paths. Separate the paths with commas (,). You can also package multiple stress testing files into a ZIP file and then specify the path of the ZIP file. Note Base64 encoding is not required for the stress testing data that is stored in files. | |
multiLine | No | Specifies whether to separate the stress testing data by row. Valid values: true and false. Default value: false. | |
http | headers | No | The HTTP request headers. The value of this parameter is of the LIST type. Example: |
timeout | No | The HTTP request latency. Unit: milliseconds. Default value: 20000. | |
optional | mode | No | The stress testing mode. Valid values:
For more information about the stress testing modes, see the "Stress testing modes" section of this topic. |
duration | No | The stress testing duration. Unit: seconds. Default value: 600. Maximum value: 1200. | |
agentCount | No | If you set the mode parameter to manual, this parameter is required. The number of stress testing agents. Default value: 1. A higher value specifies a higher workload. | |
concurrency | No | If you set the mode parameter to manual, this parameter is required. The concurrency of each stress testing agent. Default value: 2. A higher value specifies a higher workload. To scale the workload during stress testing, increase the concurrency of each stress testing agent before you increase the number of stress testing agents. | |
adjustInterval | No | If you set the mode parameter to scan, this parameter is required. The interval at which the workload are automatically increased. Unit: seconds. Default value: 60. | |
minQPS | No | If you set the mode parameter to scan, this parameter is required. The initial queries per second (QPS) for stress testing. Default value: 100. | |
maxQPS | No | If you set the mode parameter to scan or auto, this parameter is required. The maximum QPS for stress testing. | |
maxRT | No | If you set the mode parameter to scan or auto, this parameter is required. The maximum response time (TP99). If the actual response time exceeds this threshold, the QPS is automatically adjusted until the response time drops below the threshold. | |
qpsGrowthDelta | No | If you set the mode parameter to scan, this parameter is required. The QPS increment. Default value: 50. | |
faultTolerate | No | If you set the mode parameter to scan or auto, this parameter is required. The maximum tolerance rate for request errors. A request error occurs if the returned status code is not 200. A value of 0.01 specifies that the error handling mechanism is triggered when over 1% of requests encounter an error. Default value: 0.001. | |
faultAction | No | If you set the mode parameter to scan or auto, this parameter is required. The operation of eas-benchmark when the request error rate exceeds the threshold that is specified by the faultTolerate parameter. Valid values:
|
For example, if the JSON configuration file is named bench_xgb.json, you can run the following command to create a stress testing task:
eascmdwin64.exe bench create bench_xgb.json
Sample command output:
[RequestId]: DE240637-4976-59AF-A28C-BAA55C0A****
[OK] Task [benchmark-xgb-test-b514] is creating
[OK] [Agnet: 0/1]: Succeed to start benchmark master
[OK] [Agnet: 0/1]: Succeed to start benchmark master
[OK] [Agnet: 1/1]: Benchmark task is Running
[OK] Benchmark task is Running
[OK] Click the link http://127.0.0.1:18222/eas-benchmark/statsview to observe realtime visualization details, you can turn it off with CTRL+C.
Turning off will not interrupt the benchmark test task, and you can reopen it by the visualize command:
eascmd -c [config_file] bench visualize benchmark-xgb-test-b514
Use the PAI console
You can create a stress testing task that runs in automatic mode in the Platform for AI (PAI) console. To create a stress testing task in the PAI console, perform the following steps:
Go to the Elastic Algorithm Service (EAS) page.
Log on to the PAI console.
In the left-side navigation pane, click Workspaces. On the Workspaces page, click the name of the workspace that you want to manage.
In the left-side navigation pane, choose .
Click the Stress Testing Task tab and then click Create Stress Testing Task.
In the Create Stress Testing Task panel, configure the following parameters and then click Create.
Parameter
Description
Service
The name of the service for which you want to create the stress testing task.
Testing Data
The stress testing data. Valid values: Single Data Entry, Data Address, OSS Object, and Upload Local File. For information about the format and file type of stress testing data, see the "Stress testing data" section of this topic.
NoteIf you select Data Address or OSS Object, you can specify multiple file paths. Separate the paths with commas (,).
Specifies whether the data in the file needs to be separated by line breaks
This parameter is valid only if you set the Testing Data parameter to Data Address, OSS Object, or Upload Local File.
If you turn on the switch, the data in the uploaded file is separated by rows and each row is used as a data entry for stress testing. If you turn off the switch, the entire content in the uploaded file is used as a data entry for stress testing.
Duration (s)
The stress testing duration. Default value: 300. Unit: seconds.
Maximum QPS
The maximum QPS for stress testing. Default value: 10000.
Maximum Response Time (ms)
The maximum response time. Unit: milliseconds. If the actual response time exceeds the value of this parameter, the QPS is automatically adjusted until the response time falls in the normal range.
Manage stress testing tasks
Use the EASCMD client
View the list of stress testing tasks
Run the
bench list
command to view the list of stress testing tasks that you created. In this example, the 64-bit Windows version of the EASCMD client is used.eascmdwin64.exe bench ls
Sample command:
eascmdwin64.exe bench ls
Sample command output:
[RequestId]: 7F953F8E-8897-5785-808A-CA64830215F6 +-------------------------+--------------------------+-------------+----------------+---------+---------------------+ | TASKNAME | TASKID | REGION | AVAILABLEAGENT | STATUS | CREATETIME | +-------------------------+--------------------------+-------------+----------------+---------+---------------------+ | benchmark-xgb-test-7846 | eas-b-ql470xog6qeh25**** | cn-shanghai | 0 | Stopped | 2022-06-17 17:58:01 | | benchmark-xgb-test-b514 | eas-b-bdnzvwq0z0h3xq**** | cn-shanghai | 2 | Running | 2022-06-20 12:18:54 | +-------------------------+--------------------------+-------------+----------------+---------+---------------------+
View the details of a stress testing task
Run the
bench desc
command to view the details of a stress testing task. In this example, the 64-bit Windows version of the EASCMD client is used.eascmdwin64.exe bench desc [benchmark_task_name]
benchmark_task_name specifies the name of the stress testing task whose details you want to view. Sample command:
eascmdwin64.exe bench desc benchmark-demo-test-c7eb
Sample command output:
+----------------+------------------------------------------------------------------------------+ | TaskName | benchmark-xgb-test-b514 | | TaskId | eas-b-bdnzvwq0z0h3xq**** | | ServiceName | xgb_test | | Region | cn-shanghai | | DesiredAgent | 2 | | AvailableAgent | 2 | | Status | Running | | Message | Benchmark task is running | | CreateTime | 2021-10-20 12:38:35 | | UpdateTime | 2021-10-20 12:38:45 | | Config | { | | | "base": { | | | "agentCount": 2, | | | "concurrency": 40, | | | "duration": 1200, | | | "requestCount": | | | 922337203685477****, | | | }, | | | ... | | | } | +----------------+------------------------------------------------------------------------------+
Enable real-time visualization for a stress testing task
Run the
bench visualize
command to enable real-time visualization for a stress testing task. This command launches a web server on your local machine at 127.0.0.1, which provides a web page that displays the real-time monitoring data of the stress testing task. In this example, the 64-bit Windows version of the EASCMD client is used.eascmdwin64.exe bench visualize [benchmark_task_name]
Sample command:
eascmdwin64.exe bench visualize benchmark-xgb-test-b514
Sample command output:
[OK] Click the link http://127.0.0.1:18734/eas-benchmark/statsview to observe realtime visualization details, you can turn it off with CTRL+C. Turning off will not interrupt the benchmark test task, and you can reopen it by the visualize command: eascmd -c [config_file] bench visualize benchmark-xgb-test-b514
To view the real-time monitoring data, visit
http://127.0.0.1:18734/eas-benchmark/statsview
in a browser.Obtain the stress testing report
When the status of a stress testing task changes to Stopped, the task is completed and the stress testing report is saved to OSS. You can run the
bench report
command to obtain the report. In this example, the 64-bit Windows version of the EASCMD client is used.eascmdwin64.exe bench report [benchmark_task_name]
Sample command:
eascmdwin64.exe bench report benchmark-demo-test-c7eb
Sample command output:
[OK] Benchmark task benchmark-demo-test-c7eb report url: http://eas-benchmark.oss-cn-chengdu.aliyuncs.com/summary/benchmark-demo-test-c7eb-10004.html
Copy the link after url: in the command output and enter the link in a browser to view the stress testing report, as shown in the following figure.
Adjust the stress testing workload
If you use the manual mode for stress testing, you can run the
bench update
command to modify the number of agents and the concurrency of each agent. In this example, the 64-bit Windows version of the EASCMD client is used.eascmdwin64.exe bench update [benchmark_task_name] -Doptional.concurrency=<attr_value> -Doptional.agentCount=<attr_value>
<attr_value> specifies the value that you want to set. Sample command:
eascmdwin64.exe bench update benchmark-demo-b99c -Doptional.concurrency=2 -Doptional.agentCount=1
Sample command output:
[RequestId]: 9920C672-4D41-5CC4-8EC0-C690F76EB2BA [OK] Running [TaskName: benchmark-demo-b99c, DesiredAgent:1, AvailableAgent: 1, Message: Benchmark task is Updating] [OK] Benchmark task benchmark-demo-b99c was updated successfully
Stop a stress testing task
Run the
bench stop
command to stop a stress testing task. In this example, the 64-bit Windows version of the EASCMD client is used.eascmdwin64.exe bench stop [benchmark_task_name]
Sample command:
eascmdwin64.exe bench stop benchmark-xgb-test-b514
Sample command output:
Are you sure to stop the benchmark task [benchmark-xgb-test-b514] in [cn-shanghai]? [Y/n] [OK] Task [benchmark-xgb-test-b514] is stopping [OK] [Agnet: 0/1]: Benchmark task is Running [OK] [Agnet: 0/1]: Benchmark task is Stopped [OK] Benchmark task is stopped
If you ran the command that enables real-time visualization in a terminal window before you stop the task, the system outputs the stress testing report in the terminal window. You can also run the
bench report
command to obtain the report.Start a stress testing task
Run the
bench start
command to start a stopped stress testing task. In this example, the 64-bit Windows version of the EASCMD client is used.eascmdwin64.exe bench start [benchmark_task_name]
NoteCompared with the
bench create
command, this command restarts a stress testing task based on the most recently updated configuration of the task.Sample command:
eascmdwin64.exe bench start benchmark-xgb-test-b514
Sample command output:
Are you sure to start the benchmark task [benchmark-xgb-test-b514] in [cn-shanghai]? [Y/n] [OK] Task [benchmark-xgb-test-b514] is starting [OK] [Agnet: 0/1]: Succeed to start benchmark master [OK] [Agnet: 1/1]: Benchmark task is Running [OK] Benchmark task is Running [OK] Click the link http://127.0.0.1:18947/eas-benchmark/statsview to observe realtime visualization details, you can turn it off with CTRL+C. Turning off will not interrupt the benchmark test task, and you can reopen it by the visualize command: eascmd -c [config_file] bench visualize benchmark-xgb-test-b514
Delete a stress testing task
After a stress testing task is completed, eas-benchmark retains records of the task based on the task status. The following table describes the retention periods.
Task status
Retention period
Stopped
48 hours
CreateFailed, UpdateFailed, Terminated, or Error
10 minutes
When the retention period ends, the system automatically deletes the task.
You can also run the
bench delete
command to manually delete a stress testing task. Format:eascmdwin64.exe bench delete [benchmark_task_name]
Sample command:
eascmdwin64.exe bench delete benchmark-xgb-test-b514
Sample command output:
Are you sure to delete the benchmark task [benchmark-xgb-test-b514] in [cn-shanghai]? [Y/n] [OK] Benchmark task benchmark-xgb-test-b514 is Deleting [OK] Benchmark task was deleted successfully
Use the PAI console
On the Stress Testing Task tab, you can view and manage stress testing tasks.
You can start, stop, or delete a stress testing task with one click.
You can click the refresh button in the upper-right corner to refresh the status of stress testing tasks.
When the Status column of a task displays Running, you can click View in the Actions column to view the real-time monitoring data of the task.
When the Status column of a task displays Stopped, you can click View in the Actions column to view the stress testing report.
Stress testing data
Format
The format of stress testing data varies based on the model and the processor of your service.
If your service processes unstructured request data, such as audio, image, or text data, upload the files that contain the request data.
If your service processes structured request data, such as data in the TFRequest format, use EAS SDK to create the request data. You can also upload the generated binary file as the request data. For more information, see Warm up model services (advanced).
File type
Choose a file type for stress testing data based on your business requirements. Common file types include TEXT, JPG, and BIN.
Stress testing modes
When you use the EASCMD client to create a stress testing task, you can configure the stress testing mode by specifying the mode parameter that is nested in the optional parameter in the JSON configuration file. Valid values:
auto
Automatic mode. In this mode, eas-benchmark automatically creates the stress testing agents, configures the concurrency of each agent, and selects the optimal algorithm to determine the maximum capacity of the service. You need to only specify the service and data parameters in the JSON configuration file. Retain the default settings of other parameters. Sample configuration:
{ "service": { "serviceName": "demo" }, "data": { "path": "https://examplebucket.oss-cn-chengdu.aliyuncs.com/data/warmup.tf.bin" }, "optional" { "maxQPS": 1000, "duration": 300 } }
scan
Periodic mode. In this mode, eas-benchmark dynamically increases the workload of the service based on parameters such as minQPS, maxQPS, adjustInterval, and qpsGrowthDelta, until the actual value of the maxRT, maxQPS, or faultTolerate parameter exceeds the specified one. Sample configuration:
{ "service": { "serviceName": "demo" }, "data": { "content": "aGVsbG8K" }, "optional": { "mode": "scan", "maxQPS": 1000, "minQPS": 500, "qpsGrowthDelta": 100, "adjustInterval": 30 } }
manual
Manual mode. In this mode, you can manually adjust the number of agents and the concurrency of each agent during stress testing. Sample configuration:
{ "service": { "serviceName": "demo" }, "data": { "content": "aGVsbG8K" }, "optional": { "mode": "manual", "agentCount": 1, "concurrency": 5 } }
API reference
You can call APIs to create and manage stress testing tasks. For more information, see Benchmark Task.
References
After the performance of the service meets your business requirements, you can call the service to run inference tasks. For more information, see Overview.