Overview of CSI-based storage solutions - Container Service for Kubernetes

When you run workloads in Container Service for Kubernetes (ACK) clusters, you may have storage requirements such as persistent storage of application data, storage of sensitive and configuration data, and dynamical provision of storage resources. ACK clusters provide the container storage feature based on the Container Storage Interface (CSI) plug-in. This feature uses Alibaba Cloud storage services to provide and manage statically or dynamically provisioned volumes.

Supported volumes

The Kubernetes community recommends that you use the CSI plug-in. The container storage feature of ACK is implemented based on the CSI plug-in. This feature is integrated with Elastic Block Storage (EBS), File Storage NAS (NAS), Object Storage Service (OSS), and local disks. It is fully compatible with Kubernetes-native storage services, such as emptyDir volumes, hostPath volumes, Secrets, and ConfigMaps. The following figure shows the volumes supported by the CSI plug-in.

Alibaba Cloud ACK allows pods to be automatically associated with storage services such as Alibaba Cloud disks, NAS, OSS, and local volumes. The following table describes the key features, use scenarios, and billing rules of the volumes.

Note

The CSI plug-in allows you to mount statically and dynamically provisioned volumes. To mount a statically provisioned volume, you must manually modify and create a persistent volume (PV) and a persistent volume claim (PVC). If you need a large number of PVs and PVCs, you can create and mount dynamically provisioned volumes. Definitions of PV and PVC:

PV
A PV is a storage resource in a Kubernetes cluster. The lifecycle of a PV is independent of the lifecycle of the pod that has the PV mounted. Different types of PVs can be provisioned by using different types of StorageClasses.
PVC
A PVC is a request for storage in the cluster. PVs are node resources consumed by pods. PVCs are claims that consume PVs. When PVs are insufficient, PVCs can dynamically provision PVs.

Storage service	Statically provisioned volume	Dynamically provisioned volume	Default ACK storage	Key feature	Scenario	Billing
Alibaba Cloud disks	Supported	Supported	Yes	Non-shared storage. A disk can be mounted only to one node.	High I/O and low latency Disks are block storage devices and are suitable for scenarios that require high I/O performance and low latency, such as databases and middleware services. Non-data sharing A disk can be provisioned only for one pod. You can use disk volumes in scenarios that do not require data sharing. For more information, see Disk volume.	For more information about the billable items of disks, see Block storage devices. For more information about the pricing of disks, visit the ECS product page.
NAS	Supported	Supported	Yes	Shared storage that provides high performance and high throughput.	Data sharing NAS file systems allow multiple pods to access the same data. We recommend that you use NAS file systems if data needs to be shared. Big data analysis NAS file systems provide high throughput and meet the requirement of shared storage access when large numbers of jobs are involved. Web applications NAS file systems can provision storage for web applications and content management systems. Log storage We recommend that you use NAS volumes to store log data. For more information, see NAS volumes.	Billing of General-purpose NAS file systems
OSS	Supported	Supported	Yes	Shared storage that supports file systems in user space.	Read-only media files such as video files and images You can use OSS volumes to read the preceding types of files. Read-only configuration files of websites and applications ossfs provides limited network performance and can be used to read small files. Note OSS volumes are mounted by using ossfs, which is implemented as a file system in user space (FUSE). The write performance is limited when you use OSS volumes. We recommend that you use other volumes in scenarios that require high write performance. For more information, see OSS volumes.	Billing overview

Different Alibaba Cloud storage services have different limits on volume usage. For more information, see the following sections.

Limits

When you use the CSI plug-in in ACK clusters, take note of the limits on the CSI plug-in and Alibaba Cloud storage services.

Limits on cluster versions

When you use the CSI plug-in in an ACK cluster, make sure that the ACK cluster runs Kubernetes 1.14 or later and the --enable-controller-attach-detach parameter is set to true for kubelet. For more information about how to update an ACK cluster, see Manually upgrade ACK clusters.

Limits on the CSI plug-in

Alibaba Cloud CSI is open source in the Kubernetes community and supports ACK clusters. CSI does not seamlessly support non-ACK clusters such as clusters that are not deployed in an Alibaba Cloud environment and self-managed clusters in Alibaba Cloud due to reasons such as cluster configurations, permission management, and network differences. If your cluster is a non-ACK cluster, you can read the source code and modify the configurations of CSI to enable CSI to support the cluster. For more information, see alibaba-cloud-csi-driver.

Limits on volumes

Volume type	Limits
Disk volumes	For more information, see Limits.
NAS volumes	NAS is a shared storage service. A persistent volume claim (PVC) that is used to mount a NAS file system can be shared among pods. You cannot use the Container Storage Interface (CSI) plug-in to mount Server Message Block (SMB) file systems. We recommend that you use the NFSv3 file sharing protocol. You can mount a NAS volume only to ECS instances in the same virtual private cloud (VPC) as the NAS file system. General-purpose and Extreme NAS file systems have different limits such as the limits on mounting connectivity, the number of file systems, and file sharing protocols. For more information, see Limits. Before you use NAS volumes, we recommend that you update the CSI plug-in to the latest version. After a mount target is created, wait until the mount target changes to the Available state. Do not delete the mount target of a NAS file system before you unmount the NAS file system. Otherwise, an operating system hang issue may occur.
OSS volumes	An OSS bucket can be shared by multiple pods. The names of PVs used by each application must be unique. To mount a subdirectory in an OSS bucket, we recommend that you set the path field of the PV instead of using subPath. If subPath or subPathExpr is used in your business, to avoid mounting errors caused by permission issues, we recommend that you read What do I do if a mounting error occurs when I use subPath or subPathExpr to mount an OSS volume?. OSS volumes are FUSE file systems mounted by using ossfs. OSS volumes are suitable for scenarios where you need to read objects. For example, use OSS volumes when you need to read configuration files, video files, or images. For more information about the limits, see ossfs limits. OSS volumes are not suitable for scenarios where you need to write objects. We recommend that you use the SDK when you need to write objects. For more information about how to use the SDK to write objects, see Best practice for OSS read/write splitting. To directly write data to OSS volumes, we recommend that you use ossfs 1.91 or later. For more information, see Introduction to ossfs 1.91 and later versions and stress testing. When you use a CSI version earlier than 1.28, ossfs runs as a process on each node. It supports the following node OSs: CentOS, Alibaba Cloud Linux, ContainerOS, and Anolis OS. If the OS of the nodes is not supported, update CSI to run ossfs. You cannot perform the chmod or chown operation when OSS volumes are mounted to the root path. To perform these operations, modify the mp_umask setting. For more information, see How do I manage the permissions related to OSS volume mounting?.

Container storage features

The following table describes the storage features supported by different ACK clusters.

Storage type	Feature	ACK managed cluster and ACK dedicated cluster	ACK Serverless cluster	ACK cluster that supports sandboxed containers
EBS	Mounting and unmounting disks
	Online resizing
	Snapshot
	Container I/O monitoring
	File systems	XFS and ext4 are supported.	XFS and ext4 are supported.	XFS and ext4 are supported.
	Block devices and bare metal devices
	Data restoration from snapshots
	Disk queue settings
	Customer managed key (CMK)-based encryption and Bring Your Own Key (BYOK)-based encryption
	Multi-zone awareness
	Custom labels
	Cross-host migration
NAS	Creating, mounting, and unmounting NAS file systems
	Mounting and unmounting Samba file systems
	Recycle bin (CNFS)
	Subdirectories or shared directories of dynamically provisioned volumes (CNFS)
	CMK-based encryption (CNFS and Extreme NAS file systems)
	Quota limits (CNFS)	Only ACK managed clusters support this feature.
	Capacity and I/O monitoring (CNFS)
	Online resizing (CNFS)
OSS	Mounting and unmounting OSS buckets
OSS	BYOK-based encryption
Local storage	Linux Volume Manager (LVM)-managed block storage
	Automated volume groups
	LVM-managed capacity-aware scheduling
	Persistent memory (PMem) that is directly accessible
	LVM-managed PMem

CNFS

You can use the Container Network File System (CNFS) to manage container storage resources in ACK Pro clusters. CNFS can improve the performance of NAS and OSS volumes and enforce quality of service (QoS). CNFS allows ACK to create, delete, describe, mount, monitor, and scale individual file storage of Alibaba Cloud by using Kubernetes CustomResourcecDefinitions (CRDs). CNFS also provides features such as recycle bin, resource quota, and I/O performance monitoring of volumes. For more information about CNFS, see CNFS, Manage the lifecycles of NAS file systems, and Manage the lifecycle of OSS buckets.

CSI components

The CSI plug-in contains the csi-plugin and csi-provisioner components that are used to automatically create, mount, and unmount volumes. By default, the CSI components are deployed in ACK managed clusters and ACK dedicated clusters. For more information about CSI components and how to update the components, see Manage the CSI plug-in.

FAQ

How do I check the storage plug-in used by a cluster?

You can check the storage plug-in used by a cluster by checking node annotations in the ACK console or checking kubelet parameters in kubectl.

Check node annotations in the ACK console

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Nodes > Nodes.
On the Nodes page, find a node that you want to manage, click More in the Actions column, and then select Details.
On the Overview tab, check the annotations of the node. If the volumes.kubernetes.io/controller-managed-attach-detach: true annotation exists, the cluster uses the CSI plug-in. Otherwise, the cluster uses the FlexVolume plug-in.

Check kubelet parameters in kubectl

Run the following command to check kubelet parameters:

ps -ef | grep kubelet

Expected output:

--enable-controller-attach-detach=true

If the value of the --enable-controller-attach-detach parameter is true, the cluster uses the CSI plug-in.
If the value of the --enable-controller-attach-detach parameter is false, the cluster uses the FlexVolume plug-in.

How do I manually grant permissions to the CSI plug-in?

Before you can use the CSI plug-in to mount, unmount, create, and delete volumes, you must grant the plug-in the permissions to access other cloud resources. In most cases, the CSI plug-in is installed in the cluster by default and granted the relevant permissions. If you want to manually grant permissions to the CSI plug-in in your cluster, you can use an AccessKey pair or a Resource Access Management (RAM) role. By default, the system grants permissions to the CSI plug-in by using a RAM role.

Use an AccessKey pair
- Specify an AccessKey pair in the CSI deployment template.
- Create a Secret to pass an AccessKey pair as environment variables.
Use a RAM role: The CSI plug-in uses the AliyunCSManagedCsiRole role to access your resources of other Alibaba Cloud services. For more information, see ACK roles. For more information about how to grant permissions to RAM roles, see Grant permissions to a RAM role.
- ACK managed clusters: The token of the RAM role used by the CSI plug-in is stored in a Secret named addon.csi.token. To grant permissions to the CSI plug-in by using the RAM role and allow the plug-in to call API operations, you need to only mount the Secret to the plug-in.
- ACK dedicated clusters: The CSI plug-in uses the RAM role assigned to the Elastic Compute Node (ECS) node on which the pod resides.