(Discontinued) Kubernetes 1.24 release notes - Container Service for Kubernetes

Container Service for Kubernetes (ACK) strictly abides by the terms of the Certified Kubernetes Conformance Program. This topic lists the changes that ACK has made to support Kubernetes 1.24.

Version updates

Components are updated and optimized by ACK to support Kubernetes 1.24.

Key component	Version	Description
Kubernetes	1.24.6-aliyun.1	Before you start the update, we recommend that you read and understand the version details in Version details. Dockershim is removed in Kubernetes 1.24. Therefore, Docker is not supported in Kubernetes 1.24. However, you can continue to use Docker images. When you create new nodes, we recommend that you use containerd as the container runtime. For more information, see Dockershim Removal FAQ. To ensure data security, the LegacyServiceAccountTokenNoAutoGeneration feature gate is enabled by default in Kubernetes 1.24. The Secret API does not automatically create Secrets to store the tokens of service accounts. To generate service account tokens, you must use the TokenRequest API. To enhance security, all tokens generated by using the TokenRequest API have a validity period by default. To generate a token that does not expire, refer to service-account-token-secrets. During the creation of an ACK dedicated cluster that runs Kubernetes 1.24, kubeadm no longer adds the `node-role.kubernetes.io/master` label when it adds control plane nodes to the cluster. Kubeadm adds only the `node-role.kubernetes.io/control-plane` label to new control plane nodes. However, the `node-role.kubernetes.io/master:NoSchedule` and `node-role.kubernetes.io/control-plane:NoSchedule` taints are both added to new control plane nodes. The `node-role.kubernetes.io/master:NoSchedule` taint will be removed in Kubernetes 1.25. In Kubernetes 1.24 and later, the logic based on which kube-proxy listens on NodePort Services is removed. After the logic is removed, TCP connections may occasionally fail if the port of a NodePort Service conflicts with the port range specified by the kernel parameter `net.ipv4.ip_local_port_range` of a node. This may lead to health check failures and service exceptions on the node. Before you update the Kubernetes version of your cluster to 1.24 or later, make sure that the ports of all NodePort Services in the cluster do not conflict with the port range specified by the kernel parameter `net.ipv4.ip_local_port_range` of each node. For more information, see Kubernetes community PR. CVE-2022-3172.
etcd	3.5.4	None
CoreDNS	v1.9.3.6-32932850-aliyun	The update does not affect your workloads. The following new features are provided: Affinity settings are optimized for CoreDNS scheduling. This allows you to deploy CoreDNS in a cluster where each node has auto scaling enabled. Caching for DNS resolution results of the ServError type is disabled. Pod anti-affinity settings based on hostnames are changed from `preferred` to `required`. This indicates that pod anti-affinity settings are forced to take effect within the topological domains of specified nodes. Custom parameters are supported. By default, log parsing is enabled.
CRI	containerd 1.5.13	None
CSI	v1.20.7-aafce42-aliyun	None
CNI	Flannel v0.15.1.13-941db231-aliyun	An init container that is used to install the Flannel plug-in is added. /var/run is used as the default directory to cache IP addresses. This prevents IP leaks that are caused by server restarts. Instances that use the ARM64 architecture are supported. The CVE-2022-28391 and CVE-2022-37434 vulnerabilities are fixed.
CNI	Terway	The Terway version must be later than 1.1.0.
NVIDIA Container Runtime	3.7.0	None
Ingress Controller	v1.2.0-aliyun.1	Ingress Controller 0.44.0 and earlier use the Ingress API v1beta1, which is retained in ACK clusters that run Kubernetes 1.22. As a result, Ingress Controller 0.44.0 and earlier cannot run as normal in clusters that run Kubernetes 1.24. To resolve this issue, update the NGINX Ingress controller to 1.2.0 or later before you update the Kubernetes version of your cluster from 1.22 to 1.24. The update may temporarily interrupt your workloads and cause compatibility issues with your workload configurations. We recommend that you evaluate the impact of the component update before you update the Kubernetes version to 1.24.

Version details

Major changes

Dockershim is removed in Kubernetes 1.24 and later versions. Therefore, Docker is not supported in Kubernetes 1.24 and later versions. However, you can continue to use Docker images. When you create new nodes, we recommend that you use containerd as the container runtime. For more information, see Dockershim Removal FAQ. For more information about the impacts and feedback of Dockershim removal, see GitHub issue.
In Kubernetes 1.24 and later versions, the response latency is reduced by 10 times for 99% of the API requests handled by kube-apiserver and the loads of kube-apiserver increase by about 25%. This is because Kubernetes 1.24 is compiled with Go 1.18, which has significant changes in its garbage collection algorithm. If the memory usage increase of kube-apiserver is not acceptable, you can mitigate the impacts by setting the GOGC environment variable. You can specify GOGC=63 to bring the memory usage of kube-aspiserver back to the original level.
To ensure data security, the LegacyServiceAccountTokenNoAutoGeneration feature gate is enabled by default in Kubernetes 1.24 and later versions. The Secret API does not automatically create Secrets to store the tokens of service accounts. To generate service account tokens, you must use the TokenRequest API. To enhance security, all tokens generated by using the TokenRequest API have a validity period by default. To generate a token that does not expire, refer to service-account-token-secrets.
During the creation of an ACK dedicated cluster that runs Kubernetes 1.24 or later, kubeadm no longer adds the node-role.kubernetes.io/master label when it adds control plane nodes to the cluster. Kubeadm adds only the node-role.kubernetes.io/control-plane label to new control plane nodes. However, the node-role.kubernetes.io/master:NoSchedule and node-role.kubernetes.io/control-plane:NoSchedule taints are both added to new control plane nodes. The node-role.kubernetes.io/master:NoSchedule taint will be removed in Kubernetes 1.25.

New features

In Kubernetes 1.23 and later versions, the structured logging feature is available for public preview. The log format of components such as kube-scheduler and kubelet are changed to the structured text format (key="value"). For example, GET /healthz: (57.126µs) 200 is changed to verb="GET" URI="/healthz" latency="57.126µs" resp=200. Regular expressions are no longer required for parsing unstructured logs. We recommend that you print logs in structured text format or JSON format. For more information, see introducing-structured-logs. By default, log data in JSON format is printed to stderr instead of stdout.
In Kubernetes 1.23 and later versions, if you configure the pod.Spec.OS parameter of a pod and the node label that indicates the node OS does not match the value of the pod.Spec.OS parameter, the kubelet of the node rejects the pod.
In Kubernetes 1.23 and later versions, the .spec.minReadySeconds parameter is supported by StatefulSets by default and the StatefulSetMinReadySeconds feature gate is available for public preview. For more information, see Minimum Ready Seconds for StatefulSets.
In Kubernetes 1.23 and later versions, the CSIDriver.Spec.StorageCapacity parameter of the Container Storage Interface (CSI) plug-in can be modified.
In Kubernetes 1.23 and later versions, the JobReadyPods feature is enabled. After this feature is enabled, the number of pods in Ready state is displayed in the status field of a Job.
In Kubernetes 1.23 and later versions, the CustomResourceValidationExpressions feature gate is enabled for CustomResourceDefinitions (CRDs). This feature gate is based on the x-kubernetes-validations extension and uses Common Expression Language (CEL) to validate CRDs.
In Kubernetes 1.23 and later versions, client-side binary files can be generated for Windows on ARM64. This is because Go 1.17 supports Windows on ARM64.
In Kubernetes 1.23 and later versions, client-go adds the following new workqueue state: Processing. When you shut down a workqueue in the Processing state, the shutdown process starts after all ongoing tasks in the workqueue are complete.
In Kubernetes 1.23 and later versions, the admission_webhook_request_total metric is added. This metric includes the following information as labels: the webhook name, the admission type, the requested operation, the HTTP status code, information about whether the request is rejected, and the namespace of the requested resource.
In Kubernetes 1.23 and later versions, the following metrics of the Kubernetes API server can reach General Availability (GA): controller_admission_duration_seconds, step_admission_duration_seconds, webhook_admission_duration_seconds, apiserver_current_inflight_requests, and apiserver_response_sizes. In addition, a new metric is added to monitor the LIST requests received by the Kubernetes API server.
In Kubernetes 1.23 and later versions, the following metrics of the scheduler can reach GA: pending_pods, preemption_attempts_total, preemption_victims, schedule_attempts_total, scheduling_attempt_duration_seconds (formerly known as e2e_scheduling_duration_seconds ), pod_scheduling_duration_seconds, pod_scheduling_attempts, framework_extension_point_duration_seconds, plugin_execution_duration_seconds, and queue_incoming_pods_total.
In Kubernetes 1.23 and later versions, all controllers are included in the health checks of kube-controller-manager.
In Kubernetes 1.24 and later versions, the CSIStorageCapacity API can be used to show the available storage capacity. This ensures that pods can be scheduled to nodes with sufficient storage capacity and avoids pod scheduling from being delayed by volume creation failures or volume mounting failures. For more information, see Storage Capacity Constraints for Pod Scheduling.
In Kubernetes 1.24 and later versions, gRPC probes are available for public preview and the GRPCContainerProbe feature gate is enabled by default. For more information, see Configure probes.
In Kubernetes 1.24 and later versions, the CSI plug-in is optimized. For example, the in-tree persistent volume (PV) deletion protection finalizer is supported and built-in storage plug-ins can be migrated to out-of-tree CSI drivers. For more information, see In-tree Storage Plugin to CSI Migration Design Doc.
In Kubernetes 1.24 and later versions, the following options are added to kube-proxy. This way, kube-proxy can run on Windows.
- --forward-healthcheck-vip: forwards the health check requests destined for the virtual IP address (VIP) of a Service to the health check Service of kube-proxy.
- --root-hnsendpoint-name: specifies the name of the Host Networking Service (HNS) endpoint for the root network namespace.
In Kubernetes 1.24 and later versions, a new optional parameter named timeZone is supported for CronJobs. After you enable the CronJobTimeZone feature, you can run CronJobs in specified time zones. This feature will be in public preview in Kubernetes 1.25.
In Kubernetes 1.24 and later versions, the following metrics are added or updated:
- The webhook_fail_open_count metric is used to monitor webhook failures.
- The sync_proxy_rules_no_local_endpoints_total metric of kube-proxy is used to monitor the number of Services that do not have internal endpoints.
- The kubelet_volume_stats_health_abnormal metric of kubelet is used to monitor the health status of volumes.
- The evictions_number metric is replaced by evictions_total.
In Kubernetes 1.24 and later versions, the maxUnavailable parameter is supported by StatefulSets. This parameter allows you to stop pods faster than before during rolling updates.
In Kubernetes 1.24 and later versions, OpenAPI V3 is enabled by default.
In Kubernetes 1.24 and later versions, certificates signed by using the SHA-1 hash algorithm cannot be verified by default. This is because Kubernetes 1.24 is compiled with Go 1.18.
In Kubernetes 1.24 and later versions, kubelet creates an iptables chain named KUBE-IPTABLES-HINT in the mangle table. Containerized components that need to modify iptables rules in the host network namespace can use this chain to check whether the system is using iptables-legacy or iptables-nft in a more reliable manner.
In Kubernetes 1.23 and 1.24, kubectl is optimized, new kubectl commands are supported, the readability of user help commands is improved, and fish and powershell are supported by code completion and hinting.
- By default, the log of the first container in a pod is displayed in the output of the kubectl logs command.
- IngressClasses are displayed in the output of the kubectl describe ingress command.
- Information about the embedded version of Kustomize is displayed in the output of the kubectl version command.
- Resource name hinting is supported by the kubectl get command. For example, pod name hints are automatically provided when you run the kubectl get pod pod1 <TAB> command.

Deprecated features

In Kubernetes 1.23 and later versions, FlexVolume is deprecated and out-of-tree CSI drivers are recommended. For more information, see Kubernetes Volume Plugin FAQ for Storage Vendors.
In Kubernetes 1.23 and later versions, specific klog command line flags are deprecated and Kubernetes will progressively simplify component logs. For more information, see System Logs.
In Kubernetes 1.23 and later versions, the --experimental-patches command line flag is deprecated in the kubeadm init, kubeadm join, and kubeadm upgrade commands. --patches and --config cannot be used at the same time.
In Kubernetes 1.23 and later versions, kube-log-runner is included in the tar package of the release to replace the deprecated --log-file parameter. For more information, see kube-log-runner.
In Kubernetes 1.23 and later versions, the scheduler_volume_scheduling_duration_seconds metric is deprecated.
In Kubernetes 1.23 and later versions, the deprecated apiserver_longrunning_gauge metric is replaced by the apiserver_longrunning_requests metric.
In Kubernetes 1.23 and later versions, when you run the kubectl --dry-run command, you must specify --dry-run=(server|client|none).
In Kubernetes 1.24 and later versions, the Service.Spec.LoadBalancerIP parameter is deprecated because this parameter does not support IPv4/IPv6 dual stack.
In Kubernetes 1.24 and later versions, the --address, --insecure-bind-address, --port, and --insecure-port=0 options of kube-apiserver are removed.
In Kubernetes 1.24 and later versions, the startup options --port=0 and --address of kube-controller-manager and kube-scheduler are removed.
In Kubernetes 1.24 and later versions, the --audit-log-version and --audit-webhook-version options of kube-apiserver support only the audit.k8s.io/v1 value. In addition, audit.k8s.io/v1[alpha|beta]1 is removed and only audit.k8s.io/v1 is supported.
In Kubernetes 1.24 and later versions, the startup option --network-plugin of kubelet is removed along with Dockershim. This option is Docker-specific and takes effect only when Docker is used as the container runtime.
In Kubernetes 1.24 and later versions, dynamic log cleanup is deprecated and removed. The dynamic log cleanup feature introduces a log filter that can be applied to the logs of all Kubernetes system components to prevent various types of sensitive information from being exposed through logs. This feature may block the logging process. For more information, see Dynamic log sanitization and KEP-1753.
The v1beta1 API version (deprecated in Kubernetes 1.20) of the VolumeSnapshot CRD is removed in Kubernetes 1.24. The v1 API version is used.
In Kubernetes 1.24 and later versions, the Service annotation tolerate-unready-endpoints (deprecated in Kubernetes 1.11) is removed and replaced by Service.spec.publishNotReadyAddresses.
In Kubernetes 1.24 and later versions, the metadata.clusterName parameter is deprecated. This parameter will be removed in the next release.
In Kubernetes 1.24 and later, the logic based on which kube-proxy listens on NodePort Services is removed. After the logic is removed, TCP connections may occasionally fail if the port of a NodePort Service conflicts with the port range specified by the kernel parameter net.ipv4.ip_local_port_range of a node. This may lead to health check failures and service exceptions on the node. Before you update the Kubernetes version of your cluster to 1.24 or later, make sure that the ports of all NodePort Services in the cluster do not conflict with the port range specified by the kernel parameter net.ipv4.ip_local_port_range of each node. For more information, see Kubernetes community PR.

API changes

In Kubernetes 1.23 and later versions, the following changes are applied to the Kubernetes API:
- The rbac.authorization.k8s.io/v1alpha1 API version is removed and replaced by rbac.authorization.k8s.io/v1.
- The scheduling.k8s.io/v1alpha1 API version is removed and replaced by scheduling.k8s.io/v1.
In Kubernetes 1.23 and later versions, HorizontalPodAutoscaler v2 can reach GA and the autoscaling/v2beta2 API version is deprecated.
In Kubernetes 1.23 and later versions, the Service.spec.ipFamilyPolicy parameter is required when you create or update IPv4/IPv6 dual-stack Services. When you use an IPv4/IPv6 dual-stack Service, you must set ipFamilyPolicy PreferDualStack or RequireDualStack.
In Kubernetes 1.23 and later versions, the code of the components that are configured based on LogFormatRegistry must be updated in order to use the logr v1.0.0 API. JSON logs are exported in the go-logr/zapr format. Some issues are fixed.
In Kubernetes 1.24 and later versions, the client.authentication.k8s.io/v1alpha1 API version is removed and replaced by the v1 API version.
In Kubernetes 1.24 and later versions, the node.k8s.io/v1alpha1 API version is removed and replaced by the v1 API version.
In Kubernetes 1.24 and later versions, the CSIStorageCapacity.storage.k8s.io API uses the v1 API version instead of the v1beta1 API version. The v1beta1 API version will be removed in Kubernetes 1.27.
In Kubernetes 1.24 and later versions, the networking.k8s.io/v1alpha1 API version is removed.

Feature gates

A feature can be in the Alpha, Beta or GA stage. An Alpha feature is disabled by default. A Beta feature is enabled by default. A GA feature is enabled and cannot be disabled. You can disable GA features in later Kubernetes versions. For more information, see Feature Gates. The following list describes the major changes of feature gates:

In Kubernetes 1.23 and later versions, the TTLAfterFinished feature gate can reach GA and is enabled by default. This feature gate uses time-to-live (TTL) controllers to clear resource objects that have finished execution.
In Kubernetes 1.23 and later versions, the StatefulSetAutoDeletePVC feature gate is supported to allow automatic deletion of persistent volume claims (PVCs) that are created by StatefulSet pods.
In Kubernetes 1.23 and later versions, the PodSecurity feature gate is available for public preview and is enabled by default. The PodSecurity feature gate is used to replace the deprecated PodSecurityPolicy admission controller.
In Kubernetes 1.23 and later versions, the IPv4/IPv6 Dual-stack Networking feature gate can reach GA and the IPv6DualStack feature gate is removed.
In Kubernetes 1.23 and later versions, the NodeLease feature gate switch is removed and the NodeLease feature gate is always enabled. This feature gate has reached GA in Kubernetes 1.17.
In Kubernetes 1.23 and later versions, the CSIVolumeFSGroupPolicy feature gate can reach GA and is always enabled by default.
In Kubernetes 1.23 and later versions, the GenericEphemeralVolume feature gate is enabled by default. For more information about how to use this feature gate, see Ephemeral Volumes. All features of common volumes are supported by generic ephemeral inline volumes. Generic ephemeral inline volumes can be provisioned by using third-party storage drivers that support persistent storage. Common volumes can be provisioned by using third-party storage drivers and can be restored from volume snapshots. Common volumes support storage capacity tracking.
In Kubernetes 1.23 and later versions, the IngressClassNamespacedParams feature gate can reach GA. This feature gate allows IngressClasses to reference namespace-scoped parameters. The - scope and namespace fields are added to the IngressClass.spec.parameters parameter.
In Kubernetes 1.23 and later versions, the StorageObjectInUseProtection feature gate postpones the deletion of PVs or PVCs if the PVs or PVCs are still in use. This feature gate has reached GA in Kubernetes 1.11 and will be removed in Kubernetes 1.25.
In Kubernetes 1.23 and later versions, the ConfigurableFSGroupPolicy feature gate can reach GA and the volume_fsgroup_recursive_apply metric is renamed as volume_apply_access_control. When you mount volumes to a pod, you can use the ConfigurableFSGroupPolicy feature gate to configure volume permissions and ownership change policies for volumes that match the fsGroup field. For more information, see https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods.
In Kubernetes 1.23 and later versions, the KubeletPodResourcesGetAllocatable feature gate is available for public preview and the GetAllocatableResources feature is enabled by default to optimize node resource allocation. For more information, see GetAllocatableResources gRPC endpoint.
In Kubernetes 1.23 and later versions, the WindowsHostProcessContainers feature gate is available for public preview and the Windows HostProcess containers are supported by default.
In Kubernetes 1.24 and later versions, the NonPreemptingPriority feature gate can reach GA.
In Kubernetes 1.24 and later versions, the ValidateProxyRedirects and StreamingProxyRedirects feature gates are deprecated.
In Kubernetes 1.24 and later versions, the JobReadyPods feature gate is available for public preview and is enabled by default. This feature gate allows you to track the number of Ready pods that are created by a Job. The number is recorded in the status field of the Job.
In Kubernetes 1.24 and later versions, the Indexed Jobs feature gate can reach GA and cannot be disabled.
In Kubernetes 1.24 and later versions, the SuspendJob feature gate can reach GA and will be removed in Kubernetes 1.26. This feature allows you to suspend and resume Jobs.
In Kubernetes 1.24 and later versions, the RemoveSelfLink feature gate can reach GA. This feature gate allows you to set the .metadata.selfLink field to an empty string for all objects and collections. This field has been deprecated in Kubernetes 1.16 and later versions. After this feature gate is enabled, the .metadata.selfLink field still belongs to the Kubernetes API.
In Kubernetes 1.24 and later versions, the PodAffinityNamespaceSelector feature gate can reach GA and will be removed in Kubernetes 1.26. This feature gate allows you to apply pod affinity settings across namespaces. This improves the performance of pod scheduling based on affinity rules.
In Kubernetes 1.24 and later versions, the AnyVolumeDataSource feature gate is available for public preview. This feature allows you to use a custom resource as the data source of a PVC.
In Kubernetes 1.24 and later versions, the CSRDuration feature gate can reach GA and the CertificateSigningRequest resource can be used to apply for X.509 certificates. The CSRDuration feature gate allows you to use an optional field named spec.expirationSeconds to specify the validity period of the certificate that you want to issue. The minimum valid value is 600.
In Kubernetes 1.24 and later versions, the ServerSideFieldValidation feature gate is available for public preview and is enabled by default. This feature gate performs resource validation on the server side instead of the client side. For example, resource validation is performed on the server side when you run the kubectl create or kubectl apply command.
In Kubernetes 1.24 and later versions, the DynamicKubeletConfig feature gate (deprecated in Kubernetes 1.22) is removed.
In Kubernetes 1.24 and later versions, the LegacyServiceAccountTokenNoAutoGeneration feature gate is enabled by default. When this feature gate is enabled, no Secrets are automatically generated for service accounts.
In Kubernetes 1.24 and later versions, the SetHostnameAsFQDN, ImmutableEphemeralVolumes and NamespaceDefaultLabelName feature gates are removed. These feature gates have reached GA in Kubernetes 1.22.
In Kubernetes 1.23 and 1.24, the following feature gates can reach GA: ConfigurableFSGroupPolicy, ControllerManagerLeaderMigration, CSIMigrationAzureDisk, CSIMigrationOpenStack, CSIStorageCapacity, CSIVolumeFSGroupPolicy, CSRDuration, CronJobControllerV2, DefaultPodTopologySpread, EfficientWatchResumption, ExpandCSIVolumes, ExpandInUsePersistentVolumes, ExpandPersistentVolumes, GenericEphemeralVolume, IPv6DualStack, IndexedJob, IngressClassNamespacedParams, NonPreemptingPriority, PodAffinityNamespaceSelector, PodOverhead, PreferNominatedNode, RemoveSelfLink, ServiceLBNodePortControl, ServiceLoadBalancerClass, SuspendJob, DynamicKubeletConfig, and TTLAfterFinished. These feature gates are enabled and cannot be disabled.

Container Service for Kubernetes:(Discontinued) Kubernetes 1.24 release notes