Alibaba Cloud Service Mesh:Enable autoscaling of pods based on the number of requests

Prerequisites

A Knative Service is created by using Knative on ASM. For more information, see Use Knative on ASM to deploy a serverless application.

Introduction to autoscaling

Knative Serving adds the Queue Proxy container to each pod. The Queue Proxy container sends concurrency metrics of the application containers to KPA. After KPA receives the metrics, KPA automatically adjusts the number of pods provisioned for a Deployment based on the number of concurrent requests and related autoscaling algorithms.

Concurrency and QPS

The concurrency refers to the number of requests received by a pod at the same time. Queries per second (QPS) refers to the number of requests that can be handled by a pod per second. QPS also shows the maximum throughput of a pod.

The increase in concurrency does not necessarily increase the QPS. In scenarios where the access load is heavy, if the concurrency is increased, the QPS may be decreased. This is because the system is overloaded and the CPU and memory usage is high, which downgrades the performance and increases the response latency.

Algorithms

KPA scales pods based on the average number of requests (or concurrent requests) received by each pod. By default, KPA scales pods based on the number of concurrent requests. Each pod can handle at most 100 requests concurrently. KPA also introduces the target utilization (target-utilization-percentage) annotation, which specifies a target utilization percentage value for auto scaling.

• Stable mode

  In stable mode, KPA counts the average number of concurrent requests across pods within the stable window. The default stable window is 60 seconds. Then, KPA adjusts the number of pods based on the average concurrency value to maintain the loads at a stable level.

• Panic mode

  In panic mode, KPA counts the average number of concurrent requests across pods within the panic window. The default panic window is 6 seconds. The panic window is calculated based on the following formula: Panic window = Stable window × panic-window-percentage. The value of panic-window-percentage is greater than 0 and smaller than 1 and the default is 0.1. When a request burst occurs and the current pod utilization exceeds the panic window, KPA increases the number of pods to handle the burst.

KPA configurations

To configure KPA, you must configure the config-autoscaler ConfigMap in the knative-serving namespace. Run the following command to query the default configurations in the config-autoscaler ConfigMap. The following content describes the key parameters.

kubectl -n knative-serving get cm config-autoscaler -o yaml

Expected output (Annotations in code are ignored.):

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-autoscaler
  namespace: knative-serving
data:
  _example:
    container-concurrency-target-default: "100"
    container-concurrency-target-percentage: "0.7"
    enable-scale-to-zero: "true"
    max-scale-up-rate: "1000"
    max-scale-down-rate: "2"
    panic-window-percentage: "10"
    panic-threshold-percentage: "200"
    scale-to-zero-grace-period: "30s"
    scale-to-zero-pod-retention-period: "0s"
    stable-window: "60s"
    target-burst-capacity: "200"
    requests-per-second-target-default: "200"

The settings of the parameters displayed under the _example field are default values. If you want to modify the settings of these parameters, copy the parameters that you want to modify under the _example field to the data field for modification.

Configure parameters related to the scale-to-zero feature

Set a concurrency target for KPA

Set the scale bounds

You can use the minScale and maxScale annotations to set the minimum and maximum number of pods that are provisioned for an application. This helps reduce cold starts and computing costs.

You can configure the minScale and maxScale parameters in the template section of the Revision configuration file.

spec:
  template:
    metadata:
      autoscaling.knative.dev/minScale: "2"
      autoscaling.knative.dev/maxScale: "10"

Scenario 1: Enable autoscaling by setting a concurrency target

In this scenario, an autoscale-go application is deployed in a cluster. This example shows how to enable KPA to perform autoscaling by setting a concurrency target.

1. Create the autoscale-go.yaml file and set the value of autoscaling.knative.dev/target to 10.

   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
     name: autoscale-go
     namespace: default
   spec:
     template:
       metadata:
         labels:
           app: autoscale-go
         annotations:
           autoscaling.knative.dev/target: "10"
       spec:
         containers:
           - image: registry.cn-hangzhou.aliyuncs.com/knative-sample/autoscale-go:0.1

2. Use kubectl to connect to the cluster and run the following command to deploy the autoscale-go application:

   kubectl apply -f autoscale-go.yaml

3. Log on to the ASM console console. Click the name of the desired ASM instance and choose ASM Gateways > Ingress Gateway. On the Ingress Gateway page, obtain the IP address from the Service address section.

4. Use the load testing tool hey to send 30 seconds of traffic maintaining 50 in-flight requests.

   For more information about how to install the load testing tool hey, see hey.

   Note

   Replace xxx.xxx.xxx.xxx with the gateway address that you actually access. For more information about how to obtain the gateway address, see Step 3: Query the gateway address in Use Knative on ASM to deploy a serverless application.

   hey -z 30s -c 50   -host "autoscale-go.default.example.com"   "http://xxx.xxx.xxx.xxx?sleep=100&prime=10000&bloat=5"

   Expected output:

   The expected output indicates that seven pods are added. When the number of concurrent requests that are handled by a pod exceeds a certain percentage (70% by default) of the concurrency target, Knative will create more pods. This prevents the concurrency target from being exceeded in the case that concurrent requests further increase.

Scenario 2: Enable autoscaling by setting scale bounds

Scale bounds control the minimum and maximum number of pods that can be provisioned for an application. In this scenario, an autoscale-go application is deployed in a cluster. This example shows how to enable autoscaling by setting scale bounds.

1. Create the autoscale-go.yaml file. Set the concurrency target to 10, minScale to 1, and maxScale to 3.

   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
     name: autoscale-go
     namespace: default
   spec:
     template:
       metadata:
         labels:
           app: autoscale-go
         annotations:
           autoscaling.knative.dev/target: "10"
           autoscaling.knative.dev/minScale: "1"
           autoscaling.knative.dev/maxScale: "3"
       spec:
         containers:
           - image: registry.cn-hangzhou.aliyuncs.com/knative-sample/autoscale-go:0.1

2. Use kubectl to connect to the cluster and run the following command to deploy the autoscale-go application:

   kubectl apply -f autoscale-go.yaml

3. Log on to the ASM console console. Click the name of the desired ASM instance and choose ASM Gateways > Ingress Gateway. On the Ingress Gateway page, obtain the IP address from the Service address section.

4. Use the load testing tool hey to send 30 seconds of traffic maintaining 50 in-flight requests.

   For more information about how to install the load testing tool hey, see hey.

   Note

   Replace xxx.xxx.xxx.xxx with the gateway address that you actually access. For more information about how to obtain the gateway address, see Step 3: Query the gateway address in Use Knative on ASM to deploy a serverless application.

   hey -z 30s -c 50   -host "autoscale-go.default.example.com"   "http://xxx.xx.xx.xxx?sleep=100&prime=10000&bloat=5"

   Expected output:

   The expected output indicates that a maximum of three pods are added and one pod is reserved when no traffic flows to the application. The autoscaling feature works as expected.

References

• When you need to securely access and manage microservices that are built by using Knative, you can use an ASM gateway to allow access to a Knative Service only over HTTPS. In addition, you can configure encrypted transmission for service endpoints to protect communications between the services, improving the security and reliability of the overall architecture. For more information, see Use an ASM gateway to access a Knative Service over HTTPS.

• If you face compatibility and stability challenges when you release a new version of your application during iterative development, you can perform a canary release based on traffic splitting for a Knative Service by using Knative on ASM. For more information, see Perform a canary release based on traffic splitting for a Knative Service by using Knative on ASM.

• You can set the threshold of the CPU metric for a Knative Service. This ensures that pods are automatically scaled to match the fluctuations of user traffic. For more information, see Use HPA in Knative.