Container Service for Kubernetes:FAQ about auto scaling

To prevent cluster-autoscaler from removing nodes, add the "cluster-autoscaler.kubernetes.io/scale-down-disabled": "true" annotation to the node configurations. Run the following command to add the annotation:

kubectl annotate node <nodename> cluster-autoscaler.kubernetes.io/scale-down-disabled=true

Check whether the following situations exist:

• The instance types in the scaling group cannot meet the resource request from pods. Some resources provided by the specified Elastic Compute Service (ECS) instance type are reserved or occupied for the following purposes:

  • Resources are used for virtualization or occupied by the operating system during instance creation. For more information, see the Why does a purchased instance have a memory size that is different from the memory size defined in the instance type? section of the "Instance FAQ" topic.

  • Resources are occupied to run components such as kubelet, kube-proxy, Terway, and container runtime. For more information, see Resource reservation policy.

  • By default, system components are installed for each node. Therefore, the requested pod resources must be less than the resource capacity of the instance type.

• Cross-zone scale-out activities cannot be triggered for pods that have limits on zones.

• The Resource Access Management (RAM) role does not have the permissions to manage the Kubernetes cluster. You must configure RAM roles for each Kubernetes cluster that is involved in the scale-out activity. For more information, see Enable node auto scaling.

• The following issues occur when you enable node auto scaling:

  • The instance fails to be added to the cluster and a timeout error occurs.

  • The node is not ready and a timeout error occurs.

  To ensure that nodes can be accurately scaled, cluster-autoscaler does not perform any scaling activities before it fixes the abnormal nodes.

Check whether the following situations exist:

• The requested resource threshold of each pod is higher than the specified scale-in threshold.

• Pods that belong to the kube-system namespace are running on the node.

• A scheduling policy forces the pods to run on the current node. Therefore, the pods cannot be scheduled to other nodes.

• PodDisruptionBudget is set for the pods on the node and the minimum value of PodDisruptionBudget is reached.

You can configure a pod to allow cluster-autoscaler to remove the node that hosts the pod or prevent cluster-autoscaler from removing the node that hosts the pod.

• To configure a pod to prevent cluster-autoscaler from removing the node that hosts the pod, add the "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" annotation to the pod configuration.

• To configure a pod to allow cluster-autoscaler to remove the node that hosts the pod, add the "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" annotation to the pod configuration.

If unknown is displayed in the current field, kube-controller-manager cannot access the data sources to collect resource metrics, and Horizontal Pod Autoscaler (HPA) fails to scale up or down.

Name:                                                  kubernetes-tutorial-deployment
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Mon, 10 Jun 2019 11:46:48  0530
Reference:                                             Deployment/kubernetes-tutorial-deployment
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 2%
Min replicas:                                          1
Max replicas:                                          4
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                      From                       Message
  ----     ------                   ----                     ----                       -------
  Warning  FailedGetResourceMetric  3m3s (x1009 over 4h18m)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

Possible causes:

• Cause 1: The data sources from which resource metrics are collected are unavailable. Run the kubectl top pod command to check whether the metric data of monitored pods is returned. If no metric data is returned, run the kubectl get apiservice command to check whether the metrics-server component is available. The following sample code provides an example of the returned data:

  View sample code

  NAME                                   SERVICE                      AVAILABLE   AGE
  v1.                                    Local                        True        29h
  v1.admissionregistration.k8s.io        Local                        True        29h
  v1.apiextensions.k8s.io                Local                        True        29h
  v1.apps                                Local                        True        29h
  v1.authentication.k8s.io               Local                        True        29h
  v1.authorization.k8s.io                Local                        True        29h
  v1.autoscaling                         Local                        True        29h
  v1.batch                               Local                        True        29h
  v1.coordination.k8s.io                 Local                        True        29h
  v1.monitoring.coreos.com               Local                        True        29h
  v1.networking.k8s.io                   Local                        True        29h
  v1.rbac.authorization.k8s.io           Local                        True        29h
  v1.scheduling.k8s.io                   Local                        True        29h
  v1.storage.k8s.io                      Local                        True        29h
  v1alpha1.argoproj.io                   Local                        True        29h
  v1alpha1.fedlearner.k8s.io             Local                        True        5h11m
  v1beta1.admissionregistration.k8s.io   Local                        True        29h
  v1beta1.alicloud.com                   Local                        True        29h
  v1beta1.apiextensions.k8s.io           Local                        True        29h
  v1beta1.apps                           Local                        True        29h
  v1beta1.authentication.k8s.io          Local                        True        29h
  v1beta1.authorization.k8s.io           Local                        True        29h
  v1beta1.batch                          Local                        True        29h
  v1beta1.certificates.k8s.io            Local                        True        29h
  v1beta1.coordination.k8s.io            Local                        True        29h
  v1beta1.events.k8s.io                  Local                        True        29h
  v1beta1.extensions                     Local                        True        29h
  ...
  [v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        29h]
  ...
  v1beta1.networking.k8s.io              Local                        True        29h
  v1beta1.node.k8s.io                    Local                        True        29h
  v1beta1.policy                         Local                        True        29h
  v1beta1.rbac.authorization.k8s.io      Local                        True        29h
  v1beta1.scheduling.k8s.io              Local                        True        29h
  v1beta1.storage.k8s.io                 Local                        True        29h
  v1beta2.apps                           Local                        True        29h
  v2beta1.autoscaling                    Local                        True        29h
  v2beta2.autoscaling                    Local                        True        29h

  If kube-system/metrics-server is not displayed in the SERVICE column of v1beta1.metrics.k8s.io, check whether metrics-server is overwritten by Prometheus Operator. If metrics-server is overwritten by Prometheus Operator, use the following YAML template to redeploy metrics-server:

  apiVersion: apiregistration.k8s.io/v1beta1
  kind: APIService
  metadata:
    name: v1beta1.metrics.k8s.io
  spec:
    service:
      name: metrics-server
      namespace: kube-system
    group: metrics.k8s.io
    version: v1beta1
    insecureSkipTLSVerify: true
    groupPriorityMinimum: 100
    versionPriority: 100

  If the issue persists, go to the Add-ons page of the cluster in the ACK console to check whether metrics-server is installed. For more information, see metrics-server.

• Cause 2: Metrics cannot be fetched during a rolling update or scale-out activity.

  By default, metrics-server collects metrics at intervals of 1 minute. However, metrics-server must wait a few minutes before it can collect metrics after a rolling update or scale-out activity. We recommend that you query metrics 2 minutes after a rolling update or scale-out activity is complete.

• Cause 3: The request field is not specified for the pod.

  By default, HPA obtains the CPU or memory usage of the pod by calculating the value of used resources/requested resources. If the requested resources are not specified in the pod configurations, HPA cannot calculate the resource usage. Therefore, you must make sure that the request field is specified in the resource parameter of the pod configurations.

• Cause 4: The metric name is incorrect. Check whether the metric name is correct. The metric name is case-sensitive. For example, if the cpu metric supported by HPA is accidentally written as CPU, unknown is displayed in the current field.

During a rolling update, kube-controller-manager performs zero filling on pods whose monitoring data cannot be collected. This may cause HPA to add an excessive number of pods. You can perform the following steps to fix this issue.

• Fix this issue for all workloads in the cluster:

  To fix this issue, we recommend that you update metrics-server to the latest version and add the following parameter to the startup settings of metrics-server.

  The following configuration takes effect on all workloads in the cluster.

  ## Add the following configuration to the startup settings of metrics-server. 
  --enable-hpa-rolling-update-skipped=true  

• Fix this issue for specific workloads. You can use one of the following methods to fix this issue for specific workloads:

  • Method 1: Add the following annotation to the template of a workload to skip HPA during rolling updates.

    ## Add the following annotation to the spec.template.metadata.annotations parameter of the workload configuration to skip HPA during rolling updates. 
    HPARollingUpdateSkipped: "true"

    View sample code

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment-basic
      labels:
        app: nginx
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: nginx
        template:
            metadata:
              labels:
                app: nginx
              annotations:
                HPARollingUpdateSkipped: "true"  # Skip HPA during rolling updates. 
            spec:
              containers:
              - name: nginx
                image: nginx:1.7.9
                ports:
                - containerPort: 80

  • Method 2: Add the following annotation to the template of a workload to skip the warm-up period before rolling updates.

    ## Add the following annotation to the spec.template.metadata.annotations parameter of the workload configuration to skip the warm-up period before rolling updates. 
    HPAScaleUpDelay: 3m # You can change the value based on your business requirements.

    View sample code

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment-basic
      labels:
        app: nginx
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: nginx
        template:
            metadata:
              labels:
                app: nginx
              annotations:
                HPAScaleUpDelay: 3m  # This setting indicates that HPA takes effect 3 minutes after the pods are created. Valid units: s and m. s indicates seconds and m indicates minutes. 
            spec:
              containers:
              - name: nginx
                image: nginx:1.7.9
                ports:
                - containerPort: 80

The following YAML template provides the sample pod configurations that prevent HPA from accidentally triggering scaling activities in this scenario.

## In this example, a Deployment is used.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-basic
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        HPAScaleUpDelay: 3m # This setting indicates that HPA takes effect 3 minutes after the pods are created. Valid units: s and m. s indicates seconds and m indicates minutes. 
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9 # Replace it with your exactly <image_name:tags>.
        ports:
        - containerPort: 80 

The number of matching pods may be inaccurate for the following reasons:

• A rolling update is in progress.

• Pods that do not belong to the object specified by the scaleTargetRef parameter have the label specified in the Selector field in the status section of the scale subresource. Run the following command to query the pods:

  kubectl get pods -n {Namespace} -l {Value of the Selector field in the status section of the subresource named scale}