Customize the kubelet parameters of a node pool - Container Service for Kubernetes

If the default kubelet configuration cannot meet your business requirements, you can customize the kubelet parameters for all nodes in a node pool. For example, you can customize the kubelet parameters when you want to modify resource reservations to adjust the resource usage, customize the pod eviction thresholds to resolve the resource shortage issue, and adjust the topology management policy to improve system performance.

Limits

For Container Service for Kubernetes (ACK) clusters, you can customize the kubelet parameters only for clusters that run Kubernetes 1.20 or later. For more information about how to update an ACK cluster, see Manually update ACK clusters.
For ACK Lingjun clusters, you can customize the kubelet parameters only for clusters that run Kubernetes 1.22 or alter.

Usage notes

You must customize the kubelet parameters of nodes in batches. The customized kubelet parameters immediately take effect on existing nodes in the node pool. Nodes that are newly added to the node pool also use the customized kubelet parameters. To apply the customized kubelet parameters, the kubelet process will automatically restart. This may affect the nodes and workloads running in the node pool. We recommend that you perform this operation during off-peak hours.
If you do not set the evictionHard, kubeReserved, or systemReserved parameter, the system reserves resources based on the default resource reservation policy. For more information, see Resource reservation policy.
Changing the amount of reserved resources may reduce the amount of allocatable resources of a node. If the resource usage of a node is high, workloads on the node may be evicted.
We recommend that you do not use the CLI to customize kubelet parameters that are unavailable in the ACK console. This may pose severe stability risks. You are liable for the validity and compatibility of the content in the userdata file. If you use invalid configurations or deprecated configurations, the nodes may become unavailable.

Customizable kubelet parameters in the ACK console

Parameter	Description	Default	Recommended value
allowedUnsafeSysctls	Specifies a list of allowed unsafe sysctl or sysctl patterns that end with asterisks (``). Separate the patterns with commas (,). Important* Before you specify this parameter, assess the risks and make sure that your nodes are available after this parameter is configured.	N/A	Unsafe sysctl patterns with the following prefixes are supported: kernel.shm kernel.msg kernel.sem fs.mqueue. net.
containerLogMaxFiles	Specifies the maximum number of container log files. The value must be greater than or equal to 2 and the container runtime must be containerd.	10	[2, 10]
containerLogMaxSize	Specifies the maximum size of a container log file before it is rotated. The container runtime must be containerd.	100Mi	N/A
eventRecordQPS	Specifies the maximum number of events that can be generated per second.	5	[1, 50]
eventBurst	Specifies the maximum size of a burst of event creation.	10	Valid values: 1 to 100. The value must be equal to or greater than the value of `eventRecordQPS`.
featureGates	Specifies a group of feature gates. Specify each feature gate in the `key=value` format. For more information, see Feature Gates. Important Before you specify this parameter, assess the risks and make sure that your nodes are available after this parameter is configured.	N/A	N/A
kubeAPIQPS	Specifies the QPS used when kubelet communicates with the Kubernetes API server.	5	[1, 50]
kubeAPIBurst	Specifies the maximum size of a burst of requests that can be sent to the Kubernetes API server per second.	10	Valid values: 1 to 100. The value must be equal to or greater than the value of `kubeAPIQPS`.
registryPullQPS	Specifies the maximum queries per second (QPS) of image pulls.	5	[1, 50]
registryBurst	Specifies the maximum size of a burst of image pulling.	10	Valid values: 1 to 100. The value must be equal to or greater than the value of `registryPullQPS`.
serializeImagePulls	Enables the kubelet to pull one image at a time.	False	Valid values: False: This value is recommended. You can use the parallel mode to pull images to accelerate the launch speeds of pods. True: In this mode, the kubelet pulls one image at a time.
cpuManagerPolicy	Specifies the CPU management policy that is used by the kubelet.	none	Valid values: none static
evictionHard	Specifies a set of hard eviction thresholds for pods.	imagefs.available<15%,memory.available<300Mi,nodefs.available<10%,nodefs.inodesFree<5%	N/A
evictionSoft	Specifies a set of soft eviction thresholds for pods.	N/A	N/A
evictionSoftGracePeriod	Specifies A set of grace periods for eviction thresholds. Note To configure this parameter, you must first configure the evictionSoft parameter.	N/A	N/A
kubeReserved	Specifies a set of configurations to reserve resources for the Kubernetes system components.	The reserved resources are determined by the system by default. For more information, see Resource reservation policy.	N/A
maxPods	Specifies the maximum number of pods that a node can run.	None. The default value depends on the physical resource configuration, such as the node specifications and container network design.	N/A
systemReserved	Specifies a set of configurations to reserve resources for the system.	The reserved resources are determined by the system by default. For more information, see Resource reservation policy.	N/A
readOnlyPort	Specifies the read-only port that the kubelet serves without authentication or authorization.	The default value is 0 for clusters whose Kubernetes versions are 1.26 or later. The default value is 10255 for clusters whose Kubernetes versions are earlier than 1.26.	0 For more information about the risks of enabling the kubelet read-only port 10255, see [Product Changes] Open the authentication port instead of the read-only port for the kubelet in ACK clusters that run Kubernetes versions earlier than 1.26.
cpuCFSQuota	Enables CPU CFS quota enforcement for containers that specify CPU limits.	true	Valid values: true false
cpuCFSQuotaPeriod	Specifies the CPU CFS quota period. Make sure that the CustomCPUCFSQuotaPeriod feature gate is enabled.	100ms	The value range is from 1 millisecond to 1 second.
podPidsLimit	Specifies the maximum number of PIDs that can be used by each pod.	16384	N/A
memoryManagerPolicy	Specifies the name of the policy used by the memory manager.	None	Valid values: None Static To use this policy, you must complete the following operations: Enable the MemoryManager feature gate. Manually manage memory reservations (systemReserved, kubeReserved, evictionHard, and reservedMemory) and make sure that the configuration is valid: `kube-reserved + system-reserved + eviction-hard = sum(reserved-memory)`. Otherwise, your nodes will become unavailable. The following figure shows an example. For more information, see Reserved memory flag.
topologyManagerPolicy	Specifies the name of the policy used by the topology manager. Due to the adoption of the NUMA architecture, data can be allocated to the same NUMA node to reduce cross-node access and improve system performance. The topology manager can make resource allocation decisions based on the topology. For more information, see Control Topology Management Policies on a node.	none	none (default): The kubelet does not apply the topology management policy. restricted: The kubelet does not allow cross-pod resource allocation when the resource requests of pods on the same NUMA node cannot be fulfilled. best-effort: The kubelet allows cross-pod resource allocation when the resource requests of pods on the same NUMA node cannot be fulfilled. single-numa-node: The kubelet allows CPU and device resource allocation only for pods on the same NUMA node.
imageGCHighThresholdPercent	Specifies the percentage of disk usage after which image garbage collection always runs. The value must be greater than the value of imageGCLowThresholdPercent.	85	[60, 95]
imageGCLowThresholdPercent	Specifies the percentage of disk usage before which image garbage collection never runs. The value must be smaller than the value of imageGCHighThresholdPercent.	80	[30, 90]
reservedMemory	Specifies a list of memory reservations for NUMA nodes.	N/A	N/A

Customize kubelet parameters in the ACK console

To apply the customized kubelet parameters, the kubelet process will automatically restart. This may affect your businesses. We recommend that you perform this operation during off-peak hours.

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Nodes > Node Pools.
On the Node Pools page, choose More > Configure kubelet in the Actions column of the node pool that you want to manage.
Read the usage notes on the page, click Custom Parameters to select the parameter that you want to customize, specify Maximum Number of Nodes to Repair per Batch, and then click Submit. Follow the instructions to complete the operation.
After you specify Maximum Number of Nodes to Repair per Batch, the kubelet updates the configurations of the nodes in the specified batches. We recommend that you set the number of batches to 10. You can view the progress of the update in the Event Rotation section. You can also pause, resume, or cancel the update.
You can pause the update and then verify the updated nodes. After you pause the update, the kubelet configuration of the nodes in the current batch will still be updated. The remaining batches of nodes are not updated until you resume the update
Note
We recommend that you complete the update at the earliest opportunity. If the update remains paused for seven days, the system automatically cancels the update and deletes the related events and logs.

FAQ

Will ACK deprecate customized kubelet parameters?

Some parameters or feature gates are marked as deprecated or directly removed from new Kubernetes versions. If the custom kubelet parameters managed by Container Service for Kubernetes are no longer applicable to new Kubernetes versions, ACK will automatically deprecate these parameters during a node configuration update.

How do I use configuration files to manage the kubelet?

Container Service for Kubernetes adjusts the way how the kubelet configuration is managed based on best practices in the Kubernetes community. In Kubernetes versions later than 1.20, the deprecated kubelet startup commands (Command-Line Flags) will be replaced by the configuration file. For more information, Kubelet Configuration (v1beta1).

The original configuration and the configuration file are reserved on new nodes (newly added nodes and newly created nodes). Existing nodes are not affected. For more information about how to use the configuration file to manage these nodes, refer to Customizable kubelet parameters in the ACK console to trigger a configuration update in order to manage the configurations of all nodes in the node pool.

How do I customize kubelet parameters that are unavailable in the ACK console?

We recommend that you do not use the CLI to customize kubelet parameters that are unavailable in the ACK console. If you want to use the CLI, assess the risks and run the following command to modify the userdata of the node pool. After the userdata is modified, the customized configuration is automatically applied to nodes that are newly added to the node pool.

echo 'KUBELET_CUSTOMIZED_ARGS=""' > /etc/kubernetes/kubelet-customized-args.conf
systemctl daemon-reload
systemctl restart kubelet

In addition, do not modify the customizable kubelet parameters in the ACK console by modifying the userdata with the CLI in case configuration conflicts occur. This may result in unexpected consequences during node pool maintenance.

Important

Using the CLI to customize kubelet parameters poses severe stability risks. You are liable for the validity and compatibility of the content in the userdata file. If you use invalid configurations or deprecated configurations, the nodes may become unavailable.

Container Service for Kubernetes:Customize the kubelet parameters of a node pool