Throttling is one of the most commonly used methods for traffic control. You can use throttling to prevent backend services from being overwhelmed by excessive external requests. Throttling prevents cascaded avalanches. The throttling feature helps you block some requests if the number of concurrent requests is large. This ensures the availability of backend services. Cloud-native gateways of Microservices Engine (MSE) allow you to configure route-level throttling policies. These fine-grained policies ensure that the number of requests on a route does not exceed a specified threshold during a specified period of time. This topic describes how to configure a throttling policy for a cloud-native gateway.
The throttling feature in the MSE console has been optimized to provide more throttling methods. You can use the new throttling methods only if your gateway version is 1.1.0 or later. The throttling feature that is provided for the gateway version 1.0.0 is also supported. Therefore, you must configure a throttling policy based on the version of your gateway.
If your gateway version is 1.1.0 or later, follow the steps in Configure a throttling policy for a gateway of 1.1.0 or later in this topic to configure a throttling policy.
If your gateway version is 1.0.0, follow the steps in Configure a throttling policy for a gateway of 1.0.0 in this topic to configure a throttling policy.
Configure a throttling policy for a gateway of 1.1.0 or later
The throttling threshold that you configure is a gateway-level throttling threshold. You can calculate the throttling threshold for a gateway node by using the following formula: Gateway-level throttling threshold/Number of nodes. If the calculated throttling threshold is a decimal value, the decimal value is rounded up to the nearest integer. For example, the queries per second (QPS) of a demo route is 1001. If your gateway has two nodes, the QPS limit for the demo route on each gateway node is 501.
Log on to the MSE console. In the top navigation bar, select a region.
In the left-side navigation pane, choose Cloud-native Gateway > Gateways. On the Gateways page, click the name of the gateway.
In the left-side navigation pane, click Routes, and click the Routes tab.
Find the routing rule that you want to modify and click Policies in the Actions column.
In the Policies section of the page that appears, click the Throttling tab.
Gateways of 1.2.25 or later
Configure a throttling rule
Throttling rules are used to block traffic immediately if the system detects that the QPS of a route reaches a specified threshold. This prevents a breakdown of backend services due to traffic surges and ensures high availability of backend services.
On the Throttling tab, click the Throttling Rules subtab.
On the Throttling Rules subtab, configure the parameters.
Parameter
Description
Gateway QPS Threshold
Enter a value for Gateway QPS Threshold.
Web Fallback Behavior
Select Return Specified Content or Go to Specified Page for Web Fallback Behavior.
If you set Web Fallback Behavior to Go to Specified Page:
Redirect URL
Enter the address to be redirected to.
If you set Web Fallback Behavior to Return Specified Content:
HTTP Status Code
Enter the HTTP status code. The default value is 429.
Returned Content Type
Select Plain Text or JSON for Returned Content Type.
Returned HTTP Text
Enter the returned text.
Enable
If you turn on this switch, the configured throttling rule takes effect.
Click New or Save to create or update the rule. In the message that appears, click OK.
Configure a concurrency rule
Concurrency rules are used to block traffic immediately if the system detects that the total number of requests that are being processed by a gateway reaches a specified threshold. When you configure concurrency rules, you can specify the maximum number of concurrent requests that can be processed by backend services. This ensures availability of backend services in scenarios in which a large number of concurrent requests are initiated.
On the Throttling tab, click the Concurrency Rules subtab.
On the Concurrency Rules subtab, configure the parameters.
Parameter
Description
Gateway Parallelism Threshold
Enter a value for Gateway Parallelism Threshold.
Web Fallback Behavior
Select Return Specified Content or Go to Specified Page for Web Fallback Behavior.
If you set Web Fallback Behavior to Go to Specified Page:
Redirect URL
Enter the address to be redirected to.
If you set Web Fallback Behavior to Return Specified Content:
HTTP Status Code
Enter the HTTP status code. The default value is 429.
Returned Content Type
Select Plain Text or JSON for Returned Content Type.
Returned HTTP Text
Enter the returned text.
Enable
If you turn on this switch, the configured concurrency rule takes effect.
Click New or Save to create or update the rule. In the message that appears, click OK.
Configure a circuit breaking rule
Circuit breaking rules are used to lower the dependency priority of a route immediately if the system detects that the response time (RT) of requests or the percentage of abnormal requests on the route reaches a specified threshold. If circuit breaking is triggered, the system does not call the requests on the route in the specified period of time. This ensures high availability of backend services. After the specified period of time elapses, the system resumes the calls to the requests on the route.
On the Throttling tab, click the Fuse rule subtab.
On the Fuse rule subtab, configure the parameters.
Parameter
Description
Statistical window duration
The length of the time window. The valid range is from 1 second to 120 minutes.
Minimum number of requests
The minimum number of requests to trigger circuit breaking. If the number of requests in the current time window is less than the value of this parameter, circuit breaking is not triggered even if the circuit breaking rule is met.
Threshold Type
Valid values: Slow call ratio (%) and Abnormal proportion (%).
If you set this parameter to Slow call ratio (%), you must configure the Slow call RT parameter. The Slow call RT parameter specifies a threshold for the response time of requests. If the response time of a request exceeds the value of the Slow call RT parameter, the request is counted as a slow call. Set the downgrade threshold to a threshold for the percentage of slow calls. After the rule is enabled, if the number of requests that are initiated in a specified period of time is greater than the specified minimum number of requests and the percentage of slow calls is greater than the specified threshold, circuit breaking is automatically implemented on requests that are processed in the next circuit breaking period. After the circuit breaking period elapses, the circuit breaker starts to detect the RT of the next request. If the RT of the next request is less than the value of the Slow call RT parameter, circuit breaking ends. If the RT of the next request is greater than the value of the Slow call RT parameter, circuit breaking is retriggered.
If you select Abnormal proportion (%), you must set the downgrade threshold to a threshold for the percentage of abnormal requests. After the circuit breaking rule is enabled, if the number of abnormal requests in a specified period of time is greater than the specified minimum number of requests and the percentage of abnormal requests is greater than the specified threshold, circuit breaking is automatically implemented on requests that are processed in the next circuit breaking period.
Fusing time (s)
The period in which circuit breaking is implemented. If circuit breaking is implemented on the requests for the route, the calls to all the requests for the route fail in the configured circuit breaking period.
Web Fallback Behavior
Select Return Specified Content or Go to Specified Page for Web Fallback Behavior.
If you set Web Fallback Behavior to Go to Specified Page:
Redirect URL
Enter the address to be redirected to.
If you set Web Fallback Behavior to Return Specified Content:
HTTP Status Code
Enter the HTTP status code. The default value is 429.
Returned Content Type
Select Plain Text or JSON for Returned Content Type.
Returned HTTP Text
Enter the returned text.
Enable
If you turn on this switch, the configured throttling rule takes effect.
Click Create or Save to create or update the rule. In the message that appears, click OK.
Gateways of 1.1.0 to 1.2.24
Configure a throttling rule
Throttling rules are used to block traffic immediately if the system detects that the QPS of a route reaches a specified threshold. This prevents a breakdown of backend services due to traffic surges and ensures high availability of backend services.
On the Throttling tab, click the Throttling Rules subtab, and click Add Throttling Rule.
In the Add Throttling Protection Rule dialog box, configure the parameters.
In the Configure a Throttling Rule step, enter a value in the Overall QPS Threshold field and turn on the Whether to open switch. Then, click Next to configure the throttling behavior.
In the Configure Throttling Behavior step, click New behavior to configure the HTTP response behavior after throttling is enabled.
After you configure the throttling behavior, click New.
Configure a concurrency rule
Concurrency rules are used to block traffic immediately if the system detects that the total number of requests that are being processed by a gateway reaches a specified threshold. When you configure concurrency rules, you can specify the maximum number of concurrent requests that can be processed by backend services. This ensures availability of backend services in scenarios in which a large number of concurrent requests are initiated.
On the Throttling tab, click the Concurrency Rules subtab, and click Add Concurrency Rules.
In the Add Concurrency Protection Rule dialog box, configure the parameters.
In the Configure a Protection Rule step, enter a value in the Overall Request Concurrency Threshold field and turn on the Whether to open switch. Then, click Next to configure the throttling behavior.
In the Configure Throttling Behavior step, click Add Behavior to configure the HTTP response behavior after throttling is triggered.
After you configure the throttling behavior, click New.
Configure a circuit breaking rule
Circuit breaking rules are used to lower the dependency priority of a route immediately if the system detects that the response time (RT) of requests or the percentage of abnormal requests on the route reaches a specified threshold. If circuit breaking is triggered, the system does not call the requests on the route in the specified period of time. This ensures high availability of backend services. After the specified period of time elapses, the system resumes the calls to the requests on the route.
On the Throttling tab, click the Fuse rule subtab, and click Added fusing rule.
In the Add Circuit Breaking Protection Rule dialog box, configure the parameters.
In the Configure a Protection Rule step, enter a value in the Overall Request Concurrency Threshold field and turn on the Whether to open switch. Then, click Next to configure the throttling behavior.
Parameter
Description
Statistical window duration
The length of the time window. The valid range is from 1 second to 120 minutes.
Minimum number of requests
The minimum number of requests to trigger circuit breaking. If the number of requests in the current time window is less than the value of this parameter, circuit breaking is not triggered even if the circuit breaking rule is met.
Threshold Type
Select Slow call ratio (%) or Abnormal proportion (%).
If you select Slow call ratio (%), you must specify the Slow call RT parameter. The Slow call RT parameter specifies the maximum RT. If the RT of a request is greater than the value of the Slow call RT parameter, a slow call is counted. Set the downgrade threshold to a threshold for the percentage of slow calls. After the rule is enabled, if the number of requests that are initiated in a specified period of time is greater than the specified minimum number of requests and the percentage of slow calls is greater than the specified threshold, circuit breaking is automatically implemented on requests that are processed in the next circuit breaking period. After the circuit breaking period elapses, the circuit breaker starts to detect the RT of the next request. If the RT of the next request is less than the value of the Slow call RT parameter, circuit breaking ends. If the RT of the next request is greater than the value of the Slow call RT parameter, circuit breaking is retriggered.
If you select Abnormal proportion (%), you must set the downgrade threshold to a threshold for the percentage of abnormal requests. After the rule is enabled, if the number of abnormal requests in a specified period of time is greater than the specified minimum number of requests and the percentage of abnormal requests is greater than the specified threshold, circuit breaking is automatically implemented on requests that are processed in the next circuit breaking period.
Fusing time (s)
The period in which circuit breaking is implemented. If circuit breaking is implemented on the requests for the route, the calls to all the requests for the route fail in the configured circuit breaking period.
Fuse recovery strategy
Specifies whether a circuit breaker retriggers circuit breaking after the circuit breaking period elapses.
If you select Single probe recovery, the circuit breaker detects the status of the next request after the circuit breaking period elapses. If no slow request calls occur or the request is normal, circuit breaking ends. Otherwise, circuit breaking is retriggered.
In the Configure Throttling Behavior step, click Add Behavior to configure the HTTP response behavior after throttling is triggered.
After you configure the throttling behavior, click New.
Configure a throttling policy for a gateway of 1.0.0
Log on to the MSE console.
In the left-side navigation pane, choose Cloud-native Gateway > Gateways. In the top navigation bar, select a region.
On the Gateways page, click the name of the gateway.
In the left-side navigation pane, choose Routes > Route Settings. Find the routing rule that you want to modify and click Policies in the Actions column.
In the Policies section of the page that appears, click the Throttling tab.
If no throttling policy exists, click Configure policy.
If a throttling policy already exists, click the
icon next to Throttling Threshold.
In the Throttling Threshold section, configure the Time Window and Maximum Number of Requests parameters and click OK.
Parameter description:
Time Window: the period of time in which throttling is implemented. The default value is 1 second. To configure this parameter, you must specify a time window with a specific unit.
Single-machine Request Threshold: the maximum number of single-server requests allowed by the gateway.
After the throttling policy is configured, turn on Enabled.
If you turn on the Enabled switch, the number of requests for the route is subject to the settings of the Time Window and Single-machine Request Threshold parameters.
If you turn off the Enabled switch, the number of requests for the route is not limited.
Verify the result
Run the following command to send a test request:
curl -I http://121.196.XX.XX/demo/item/list //The IP address of the Ingress gateway.Sample response when the configured throttling policy is not enabled
HTTP/1.1 200 OK x-content-type-options: nosniff x-xss-protection: 1; mode=block cache-control: no-cache, no-store, max-age=0, must-revalidate pragma: no-cache expires: 0 x-frame-options: DENY content-type: application/json content-length: 86 date: Mon, 29 Nov 2021 08:28:00 GMT x-envoy-upstream-service-time: 4 server: istio-envoySample response when the configured throttling policy is enabled (returned HTTP status code: 429)
HTTP/1.1 429 Too Many Requests x-local-rate-limit: true content-length: 18 content-type: text/plain date: Mon, 29 Nov 2021 08:28:01 GMT server: istio-envoy