If the default kubelet configuration cannot meet your business requirements, you can customize the kubelet parameters for all nodes in a node pool. For example, you can customize the kubelet parameters when you want to modify resource reservations to adjust the resource usage, customize the pod eviction thresholds to resolve the resource shortage issue, and adjust the topology management policy to improve system performance.
Limitations
For Container Service for Kubernetes (ACK) clusters, you can customize the kubelet parameters only for clusters that run Kubernetes 1.20 or later. For more information about how to update an ACK cluster, see Manually upgrade ACK clusters.
For ACK Lingjun clusters, you can customize the kubelet parameters only for clusters that run Kubernetes 1.22 or alter.
If your cluster version does not meet these requirements, exceptions may occur.
Usage notes
You must customize the kubelet parameters of nodes in batches. The customized kubelet parameters immediately take effect on existing nodes in the node pool. Nodes that are newly added to the node pool also use the customized kubelet parameters. To apply the customized kubelet parameters, the kubelet process will automatically restart. This may affect the nodes and workloads running in the node pool. We recommend that you perform this operation during off-peak hours.
If you do not set the evictionHard, kubeReserved, or systemReserved parameter, the system reserves resources based on the default resource reservation policy. For more information, see Resource reservation policy.
Changing the amount of reserved resources may reduce the amount of allocatable resources of a node. If the resource usage of a node is high, workloads on the node may be evicted.
We recommend that you do not use the CLI to customize kubelet parameters that are unavailable in the ACK console. This may pose severe stability risks. You are liable for the validity and compatibility of the content in the userdata file. If you use invalid configurations or deprecated configurations, the nodes may become unavailable.
Customizable kubelet parameters in the ACK console
Parameter | Description | Default | Recommended value |
allowedUnsafeSysctls | Specifies a list of allowed unsafe sysctl or sysctl patterns that end with asterisks ( Important Before you specify this parameter, assess the risks and make sure that your nodes are available after this parameter is configured. | N/A | Unsafe sysctl patterns with the following prefixes are supported:
|
containerLogMaxFiles | Specifies the maximum number of container log files. The value must be greater than or equal to 2 and the container runtime must be containerd. | 10 | [2, 10] |
containerLogMaxSize | Specifies the maximum size of a container log file before it is rotated. The container runtime must be containerd. | 100Mi | N/A |
cpuCFSQuota | Enables CPU CFS quota enforcement for containers that specify CPU limits. | true | Valid values:
|
cpuCFSQuotaPeriod | Specifies the CPU CFS quota period. Make sure that the CustomCPUCFSQuotaPeriod feature gate is enabled. | 100ms | The value range is from 1 millisecond to 1 second. |
cpuManagerPolicy | Specifies the CPU management policy that is used by the kubelet. | none | Valid values:
|
eventBurst | Specifies the maximum size of a burst of event creation. | 10 | Valid values: 1 to 100. The value must be equal to or greater than the value of |
eventRecordQPS | Specifies the maximum number of events that can be generated per second. | 5 | [1, 50] |
evictionHard | Specifies a set of hard eviction thresholds for pods. | imagefs.available<15%,memory.available<300Mi,nodefs.available<10%,nodefs.inodesFree<5% | N/A |
evictionSoft | Specifies a set of soft eviction thresholds for pods. | N/A | N/A |
evictionSoftGracePeriod | Specifies A set of grace periods for eviction thresholds. Note To configure this parameter, you must first configure the evictionSoft parameter. | N/A | N/A |
featureGates | Specifies a group of feature gates. Specify each feature gate in the
Important Before you specify this parameter, assess the risks and make sure that your nodes are available after this parameter is configured. | N/A | N/A |
imageGCHighThresholdPercent | Specifies the percentage of disk usage after which image garbage collection always runs. The value must be greater than the value of imageGCLowThresholdPercent. | 85 | [60, 95] |
imageGCLowThresholdPercent | Specifies the percentage of disk usage before which image garbage collection never runs. The value must be smaller than the value of imageGCHighThresholdPercent. | 80 | [30, 90] |
kubeAPIBurst | Specifies the maximum size of a burst of requests that can be sent to the Kubernetes API server per second. | 10 | Valid values: 1 to 100. The value must be equal to or greater than the value of |
kubeAPIQPS | Specifies the QPS used when kubelet communicates with the Kubernetes API server. | 5 | [1, 50] |
kubeReserved | Specifies a set of configurations to reserve resources for the Kubernetes system components. | The reserved resources are determined by the system by default. For more information, see Resource reservation policy. | N/A |
maxPods | Specifies the maximum number of pods that a node can run. | None. The default value depends on the physical resource configuration, such as the node specifications and container network design. | N/A |
memoryManagerPolicy | Specifies the name of the policy used by the memory manager. | None | Valid values:
|
podPidsLimit | Specifies the maximum number of PIDs that can be used by each pod. | 16384 | N/A |
readOnlyPort | Specifies the read-only port that the kubelet serves without authentication or authorization. |
| 0 For more information about the risks of enabling the kubelet read-only port 10255, see [Product Changes] Open the authentication port instead of the read-only port for the kubelet in ACK clusters that run Kubernetes versions earlier than 1.26. |
registryBurst | Specifies the maximum size of a burst of image pulling. | 10 | Valid values: 1 to 100. The value must be equal to or greater than the value of |
registryPullQPS | Specifies the maximum queries per second (QPS) of image pulls. | 5 | [1, 50] |
reservedMemory | Specifies a list of memory reservations for NUMA nodes. | N/A | N/A |
serializeImagePulls | Enables the kubelet to pull one image at a time. | False | Valid values:
|
systemReserved | Specifies a set of configurations to reserve resources for the system. | The reserved resources are determined by the system by default. For more information, see Resource reservation policy. | N/A |
topologyManagerPolicy | Specifies the name of the policy used by the topology manager. Due to the adoption of the NUMA architecture, data can be allocated to the same NUMA node to reduce cross-node access and improve system performance. The topology manager can make resource allocation decisions based on the topology. For more information, see Control Topology Management Policies on a node. | none |
|
ContainerLogMonitorInterval | The Kubernetes version of the cluster must be 1.30 or later. This feature is available only to users in the whitelist. To use it, submit a ticket. The interval for detecting the execution of container log rotation. | 10s | [3s, 60s] |
ContainerLogMaxWorkers | The Kubernetes version of the cluster must be 1.30 or later. This feature is available only to users in the whitelist. To use it, submit a ticket. The maximum number of concurrent workers performing log rotation. | 1 | [1, 20] |
For more information, see Customize kubelet parameters in the ACK console
To apply the customized kubelet parameters, the kubelet process will automatically restart. This may affect your businesses. We recommend that you perform this operation during off-peak hours.
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose .
On the Node Pools page, choose More > Configure kubelet in the Actions column of the node pool that you want to manage.
Read the usage notes on the page, click Custom Parameters to select the parameter that you want to customize, specify Maximum Number of Nodes to Repair per Batch, and then click Submit. Follow the instructions to complete the operation.
After you specify Maximum Number of Nodes to Repair per Batch, the kubelet updates the configurations of the nodes in the specified batches. We recommend that you set the number of batches to 10. You can view the progress of the update in the Event Rotation section. You can also pause, resume, or cancel the update.
You can pause the update and then verify the updated nodes. After you pause the update, the kubelet configuration of the nodes in the current batch will still be updated. The remaining batches of nodes are not updated until you resume the update
NoteWe recommend that you complete the update at the earliest opportunity. If the update remains paused for seven days, the system automatically cancels the update and deletes the related events and logs.
FAQ
Will ACK deprecate customized kubelet parameters?
Some parameters or feature gates are marked as deprecated or directly removed from new Kubernetes versions. If the custom kubelet parameters managed by Container Service for Kubernetes are no longer applicable to new Kubernetes versions, ACK will automatically deprecate these parameters during a node configuration update.
How do I use configuration files to manage the kubelet?
Container Service for Kubernetes adjusts the way how the kubelet configuration is managed based on best practices in the Kubernetes community. In Kubernetes versions later than 1.20, the deprecated kubelet startup commands (Command-Line Flags) will be replaced by the configuration file. For more information, Kubelet Configuration (v1beta1).
The original configuration and the configuration file are reserved on new nodes (newly added nodes and newly created nodes). Existing nodes are not affected. For more information about how to use the configuration file to manage these nodes, refer to Customizable kubelet parameters in the ACK console to trigger a configuration update in order to manage the configurations of all nodes in the node pool.
How do I customize kubelet parameters that are unavailable in the ACK console?
We recommend that you do not use the CLI to customize kubelet parameters that are unavailable in the ACK console. If you want to use the CLI, assess the risks and run the following command to modify the userdata of the node pool. After the userdata is modified, the customized configuration is automatically applied to nodes that are newly added to the node pool.
echo 'KUBELET_CUSTOMIZED_ARGS=""' > /etc/kubernetes/kubelet-customized-args.conf
systemctl daemon-reload
systemctl restart kubeletIn addition, do not modify the customizable kubelet parameters in the ACK console by modifying the userdata with the CLI in case configuration conflicts occur. This may result in unexpected consequences during node pool maintenance.
Using the CLI to customize kubelet parameters poses severe stability risks. You are liable for the validity and compatibility of the content in the userdata file. If you use invalid configurations or deprecated configurations, the nodes may become unavailable.
Related topics
For more information about parameters supported by managed node pools, see Overview of managed node pools and Create a node pool.
For more information about how to troubleshoot node, pod, and kubelet exceptions, see Troubleshoot node exceptions, Pod troubleshooting, and FAQ about nodes and node pools.