Container Compute Service:Kubernetes 1.28 release notes

Version updates

The following key components are updated and optimized by Alibaba Cloud Container Compute Service (ACS) to support Kubernetes 1.28.

Update notes

Terms

We recommend that you learn the following terms before you read this topic.

Major changes

• The scheduling logic of the scheduler is optimized in Kubernetes 1.28 to reduce invalid retries and improve performance.

  If your cluster uses a custom scheduler plug-in, we recommend that you optimize and update the plug-in to improve the performance. For more information, see Changes in the scheduling framework.

• The Kubernetes community provides the CSI migration solution to replace existing storage plug-in drivers with an off-tree driver that uses standard CSI APIs. The CSI migration feature is in the GA stage in Kubernetes 1.25. The storage.k8s.io/v1beta1 API and Elastic Block Service (EBS) plug-in are removed in Kubernetes 1.27. The CephFS volume plug-in code is removed and kubernetes.io/rbd is deprecated in Kubernetes 1.28. Instead, the CephFS CSI driver is used. In addition, you can no longer migrate Ceph RBD volumes to a plug-in that uses the off-tree CSI driver in Kubernetes 1.28.

• The following CVE vulnerabilities are fixed in 1.28.9-aliyun.1:

  • CVE-2023-45288

  • CVE-2024-3177

  • CVE-2024-24786

Functions and features

Kubernetes 1.28

• The non-graceful node shutdown feature has reached GA. When a node is shut down due to an exception such as a power shortage, the StatefulSet needs to create pods with the same name on another node to avoid business interruptions.

• The NodeOutOfServiceVolumeDetach feature gate has reached GA. After this feature is enabled, when a node is shut down due to an exception, volume detach operations are immediately performed for the terminated pods on the node. This allows pods on the out-of-service node to quickly recover on other nodes.

• The retroactive default StorageClass assignment feature has reached GA. Before this feature is introduced, if you create a PVC without the storageClassName field when no default StorageClass exists, the PVC remains in the Pending state. After this feature is introduced, when a default StorageClass is created, the PVC without the storageClassName field automatically uses the default StorageClass.

• Two features are introduced to avoid Job failures.

  • Compared with the deletionTimestamp feature gate that creates replacement pods immediately after pods are deleted, the JobPodReplacementPolicy feature gate (in the Alpha stage) creates replacement pods only after pods are assigned the Failed phase (status.phase: Failed). The policy prevents two pods from using the same index and node resources at the same time.

  • The JobBackoffLimitPerIndex feature gate (in the Alpha phase) allows you to set .spec.backoffLimitPerIndex to limit the maximum number of retries for pod failures per index. Before this feature is introduced, if the number of consecutive pod failures of an index reaches .spec.backoffLimit, the corresponding indexed Job fails.

• If the completion field of an indexed Job is set to a value greater than 100,000, the parallelism field of the Job is set to a value greater than 10,000, and large numbers of pods fail, pod terminal phase tracing may fail. To prevent this issue, warnings are displayed if you set the preceding fields to excessively large values when you create a Job.

• The reason and fieldPath fields are added to CustomResourceDefinition (CRD) validation rules to return the reason and field path when CRD validation fails. For more information, see CRD Validation Expression Language.

• Common Expression Language (CEL) expressions can be used in webhook matching requests. Up to 64 matching conditions are supported. For more information, see Matching requests: matchConditions.

• The SidecarContainers feature gate is introduced to allow you to specify the time when sidecar containers are launched. For example, you can launch log collection containers before other containers to improve the reliability of log collection. For more information, see Kubernetes v1.28: Introducing native sidecar containers. This feature has reached Alpha in Kubernetes 1.28 and is disabled by default.

• The .status.resizeStatus field of a PVC is replaced with the .status.allocatedResourceStatus map field to store the states of resources that are being resized for the PVC. For more information, see PersistentVolumeClaimStatus.

• Pod indexes (sequence numbers) are added as labels to pods created by indexed Jobs and StatefulSets.

• The ValidatingAdmissionPolicy feature gate (in the Beta phase) provides a declarative alternative to the method of validating admission webhooks to validate resource requests. The feature gate also allows you to use CEL expressions to write complex validation rules. The API server will validate resource requests against CEL expressions.

• The --concurrent-cron-job-syncs flag is added to the Kubernetes controller manager to set the concurrency of the CronJob controller and the --concurrent-job-syncs flag is added to set the concurrency of the Job controller. For more information, see --concurrent-cron-job-syncs and --concurrent-job-syncs.

• The API server is optimized:

  • The memory usage of getting a list (GetList) from the cache is reduced. For more information, see GetList test data.

  • The issue that the endpoint of a Kubernetes Service is not removed when only one replicated API server exists is fixed. This ensures that the endpoints of Kubernetes Services are removed during graceful shutdown.

  • The OpenAPI v2 controller is made lazy to aggregate information from CRDs and the OpenAPI v2 specifications are reduced. When no client sends requests to the OpenAPI v2, the CPU and memory usage of the API server is reduced. In addition, the efficiency of installing large numbers of CRDs is improved. However, this slows down the processing of first-time requests. We recommend that you update your client to a version that supports OpenAPI v3.

  • The Consistent Reads from Cache feature gate is introduced to allow you to use the watch cache to guarantee consistent reads for LIST requests.

  • A variety of metrics can be collected by calling the metrics API.

Deprecated features

Kubernetes 1.28

• The in-tree CephFS plug-in code is removed.

  We recommend that you use the CephFS CSI driver.

• Support for migrating Ceph RBD volumes to a plug-in that uses the off-tree CSI driver is deprecated and will be removed in later versions.

  We recommend that you complete the migration before the removal of the in-tree plug-in code.

• The RBD volume plug-in (kubernetes.io/rbd) is deprecated and will be removed in later versions.

  We recommend that you use the CephFS CSI driver.

• KMSv1 is deprecated. If you want to continue to use KMSv1, set --feature-gates=KMSv1=true. For more information, see Mark KMS v1beta1 as deprecated with no further fixes.

  We recommend that you use KMSv2.

• The --volume-host-cidr-denylist and --volume-host-allow-local-loopback flags in Kubernetes controller manager commands are deprecated.

• The --azure-container-registry-config flag in kubelet commands is deprecated.

  We recommend that you use the image-credential-provider-config and --image-credential-provider-bin-dir flags.

• You can no longer create Windows node pools.

Deprecated APIs

The CSIStorageCapacity API allows you to query the current available storage capacity to ensure that your pods are scheduled to a node with sufficient storage resources. The version of the storage.k8s.io/v1beta1API is deprecated in Kubernetes 1.24 and removed in Kubernetes 1.27.

We recommend that you use the storage.k8s.io/v1 version. This version is available in Kubernetes 1.24 and later versions. For more information, see Storage Capacity Constraints for Pod Scheduling KEP.

Feature gates

This section lists only the major changes. For more information, see Feature Gates.

Kubernetes 1.28

• When the NodeOutOfServiceVolumeDetach feature gate adds the node.kubernetes.io/out-of-service taint to mark a node as out-of-service, pods that do not match tolerations on the node are forcefully evicted and volumes are immediately detached. This feature gate has reached GA in Kubernetes 1.28 and is always enabled by default.

• The AdmissionWebhookMatchCondition feature gate is enabled by default to allow you to use CEL expressions as webhook matching conditions.

• The UnknownVersionInteroperabilityProxy feature gate has reached Alpha. This feature gate can send requests to the correct API server when multiple API server versions exist. For more information, see Mixed Version Proxy.

• The IPTablesOwnershipCleanup feature gate has reached GA. This feature gate causes the kubelet to no longer create KUBE-MARK-DROP and KUBE-MARK-MASQ iptables rules.

• The ConsistentListFromCache feature gate has reached Alpha. This feature gate allows the API server to use the watch cache to guarantee consistent reads for LIST requests.

• The ProbeTerminationGracePeriod feature gate has reached GA and is enabled by default. This feature gate allows you to use probe-level terminationGracePeriodSeconds.

• The following feature gates in the GA stage are removed: DelegateFSGroupToCSIDriver, DevicePlugins, KubeletCredentialProviders, MixedProtocolLBService, ServiceInternalTrafficPolicy, ServiceIPStaticSubrange, and EndpointSliceTerminatingCondition.

References

For more information about the release notes for Kubernetes 1.27 and Kubernetes 1.28, see CHANGELOG-1.27 and CHANGELOG-1.28.