By Kan Junbao (Junbao)
FlexVolume is a storage plugin that has been supported by Kubernetes since its earlier versions. At present, the Kubernetes community still uses FlexVolume as a storage plugin. FlexVolume is a non-containerized solution. The community introduced the Container Storage Interface (CSI) specification and will maintain and update the CSI specification in the long run.
Earlier versions of Alibaba Cloud Container Service for Kubernetes (ACK) support only FlexVolume. ACK now supports both FlexVolume and CSI. With the constant evolution of CSI, CSI has far surpassed FlexVolume in terms of orchestration capabilities. Therefore, the long-term plan for ACK is to eventually support only CSI, and gradually stop the support for FlexVolume.
This article shows you how to replace Alibaba Cloud NAS and OSS volumes mounted through FlexVolume with those of the CSI type.
To perform this operation, you must upgrade your ACK cluster to 1.16 or later.
If you have mounted a cloud disk to your cluster through FlexVolume, the solution described in this article is not applicable.
To migrate from FlexVolume drivers to CSI, you must restart the running pods and redefine persistent volumes (PVs) and persistent volume claims (PVCs).
To deploy the CSI plugin in your cluster, use a template provided on the following page (you can also select a later release from the dropdown list, for example, release-v1.0.6): https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/tree/release-v1.0.5/deploy/ack
Run the following commands to deploy CSI:
# kubectl apply -f csi-plugin.yaml
# kubectl apply -f csi-provisioner.yaml
When you deploy CSI, replace the address of the mirror to the region address of your cluster to improve the download speed. For example, replace registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.14.8.39-0d749258-aliyun
with registry-vpc.cn-beijing.aliyuncs.com/acs/csi-plugin:v1.14.8.39-0d749258-aliyun
.
After you complete the deployment, check whether the deployment is successful.
# kubectl get pod -nkube-system |grep csi
csi-plugin-5jqf6 9/9 Running 0 4m15s
csi-plugin-dt8fd 9/9 Running 0 4m15s
csi-plugin-nlnpn 9/9 Running 0 4m15s
csi-plugin-p5lv6 9/9 Running 0 4m15s
csi-plugin-s29wp 9/9 Running 0 4m15s
csi-provisioner-58cb59cf6c-5c9qj 8/8 Running 0 4m9s
csi-provisioner-58cb59cf6c-k96v2 8/8 Running 0 4m9s
Note: A StorageClass-related error may occur when you deploy csi-provisioner.
This error is caused because FlexVolume and Disk-Controller also have corresponding StorageClasses which have the same name as the CSI StorageClass.
To solve this error, delete the existing StorageClasses and then reapply csi-provisioner:
# kubectl get sc
NAME PROVISIONER AGE
alicloud-disk-available alicloud/disk 5h11m
alicloud-disk-efficiency alicloud/disk 5h11m
alicloud-disk-essd alicloud/disk 5h11m
alicloud-disk-ssd alicloud/disk 5h11m
# kubectl delete sc alicloud-disk-available alicloud-disk-efficiency alicloud-disk-essd alicloud-disk-ssd
storageclass.storage.k8s.io "alicloud-disk-available" deleted
storageclass.storage.k8s.io "alicloud-disk-efficiency" deleted
storageclass.storage.k8s.io "alicloud-disk-essd" deleted
storageclass.storage.k8s.io "alicloud-disk-ssd" deleted
# kubectl apply -f csi-provisioner.yaml
# kubectl get sc
NAME PROVISIONER AGE
alicloud-disk-available diskplugin.csi.alibabacloud.com 84s
alicloud-disk-efficiency diskplugin.csi.alibabacloud.com 84s
alicloud-disk-essd diskplugin.csi.alibabacloud.com 84s
alicloud-disk-ssd diskplugin.csi.alibabacloud.com 84s
alicloud-disk-topology diskplugin.csi.alibabacloud.com 5m18s
Procedure:
The following shows you how to migrate a NAS PVC in a deployment:
# kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
nas-static 2/2 2 2 6m21s
# kubectl get pod
NAME READY STATUS RESTARTS AGE
nas-static-7f9446758b-2qfxj 1/1 Running 0 11m
nas-static-7f9446758b-wcwc5 1/1 Running 0 11m
# kubectl describe pod nas-static-7f9446758b-2qfxj |grep ClaimName
ClaimName: nas-pvc
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nas-pvc Bound nas-pv 512Gi RWX nas 12m
## Log on to the node where the pod is located and view information about the mounting.
2564f49129-ysu87.cn-shenzhen.nas.aliyuncs.com:/aliyun on /var/lib/kubelet/pods/b9f14e39-86ed-4c39-b4b7-baba37b15bac/volumes/alicloud~nas/nas-pv type nfs4 (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,noresvport,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.7.208,local_lock=none,addr=192.168.1.152)
Step 1: Retain the existing PVC (nas-pvc) and PV (nas-pv).
# kubectl get pvc nas-pvc -oyaml > pvc.nas-pvc.yaml
# cat pvc.nas-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nas-pvc
namespace: default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 512Gi
selector:
matchLabels:
alicloud-pvname: nas-pv
storageClassName: nas
# kubectl get pv nas-pv -oyaml > pv.nas-pv.yaml
# cat pv.nas-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
labels:
alicloud-pvname: nas-pv
name: nas-pv
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 512Gi
flexVolume:
driver: alicloud/nas
options:
path: /aliyun
server: 2564f49129-ysu87.cn-shenzhen.nas.aliyuncs.com
vers: "3"
persistentVolumeReclaimPolicy: Retain
storageClassName: nas
Step 2: Create a PVC (nas-pvc-new) and a PV (nas-pv-new) of the CSI type.
# cat pvc.nas-pvc-new.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nas-pvc-new
namespace: default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 512Gi
selector:
matchLabels:
alicloud-pvname: nas-pv-new
storageClassName: nas
## Change the FlexVolume configuration to the CSI configuration.
## You can write the options in mountOptions.
# cat pv.nas-pv-new.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
labels:
alicloud-pvname: nas-pv-new
name: nas-pv-new
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 512Gi
csi:
driver: nasplugin.csi.alibabacloud.com
volumeHandle: nas-pv-new
volumeAttributes:
server: "2564f49129-ysu87.cn-shenzhen.nas.aliyuncs.com"
path: "/aliyun"
mountOptions:
- nolock,tcp,noresvport
- vers=3
persistentVolumeReclaimPolicy: Retain
storageClassName: nas
# kubectl apply -f pvc.nas-pvc-new.yaml
# kubectl apply -f pv.nas-pv-new.yaml
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nas-pvc Bound nas-pv 512Gi RWX nas 30m
nas-pvc-new Bound nas-pv-new 512Gi RWX nas 2s
Step 3: Update the application to apply the new CSI PVC.
# kubectl edit deploy nas-static
Replace the existing PVC with the new PVC.
volumes:
- name: pvc-nas
persistentVolumeClaim:
claimName: nas-pvc-new
## Check whether the pod has been restarted.
# kubectl get pod
NAME READY STATUS RESTARTS AGE
nas-static-6f7d965689-7sm5b 1/1 Running 0 70s
nas-static-6f7d965689-8npxw 1/1 Running 0 53s
## View information about the mounting.
2564f49129-ysu87.cn-shenzhen.nas.aliyuncs.com:/aliyun on /var/lib/kubelet/pods/ac02ea3f-125f-4b38-9bcf-9b117f62eaf0/volumes/kubernetes.io~csi/nas-pv-new/mount type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,noresvport,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.152,mountvers=3,mountport=2049,mountproto=tcp,local_lock=all,addr=192.168.1.152)
Through the preceding three steps, an NAS volume mounted through FlexVolume in a deployment is replaced with an NAS volume of the CSI type.
Procedure:
The following shows you how to migrate a dynamic PV in a StatefulSet:
# kubectl get sts
NAME READY AGE
web 2/2 3m11s
# kubectl get pod
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 3m14s
web-1 1/1 Running 0 118s
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nas-pvc-web-0 Bound default-nas-pvc-web-0-pvc-951911bd-b7f3-4551-8647-22f1f5d576b3 20Gi RWX alicloud-nas 3m32s
nas-pvc-web-1 Bound default-nas-pvc-web-1-pvc-95eee8f9-54f4-42dc-a34f-b554423220eb 20Gi RWX alicloud-nas 2m16s
# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
default-nas-pvc-web-0-pvc-951911bd-b7f3-4551-8647-22f1f5d576b3 20Gi RWX Delete Bound default/nas-pvc-web-0 alicloud-nas 2m34s
default-nas-pvc-web-1-pvc-95eee8f9-54f4-42dc-a34f-b554423220eb 20Gi RWX Delete Bound default/nas-pvc-web-1 alicloud-nas 2m14s
# kubectl get sc alicloud-nas
NAME PROVISIONER AGE
alicloud-nas alicloud/nas 106m
Step 1: Retain the existing StorageClass, PVC, and PV templates.
## Retain the PVC template.
# kubectl get pvc nas-pvc-web-0 -oyaml > origin/nas-pvc-web-0.yaml
# cat origin/nas-pvc-web-0.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
volume.beta.kubernetes.io/storage-provisioner: alicloud/nas
finalizers:
- kubernetes.io/pvc-protection
labels:
app: nginx
name: nas-pvc-web-0
namespace: default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 20Gi
storageClassName: alicloud-nas
# kubectl get pvc nas-pvc-web-1 -oyaml > origin/nas-pvc-web-1.yaml
## Retain the PV template.
# kubectl get pv default-nas-pvc-web-0-pvc-951911bd-b7f3-4551-8647-22f1f5d576b3 -oyaml > origin/default-nas-pvc-web-0-pvc-951911bd-b7f3-4551-8647-22f1f5d576b3.yaml
# cat origin/default-nas-pvc-web-0-pvc-951911bd-b7f3-4551-8647-22f1f5d576b3.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: alicloud/nas
labels:
createdby.aliyun.com: alicloud-nas-controller
version.controller.aliyun.com: v1.14.3-58bf821
name: default-nas-pvc-web-0-pvc-951911bd-b7f3-4551-8647-22f1f5d576b3
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 20Gi
flexVolume:
driver: alicloud/nas
options:
options: nolock,tcp,noresvport,vers=3
path: /root/default-nas-pvc-web-0-pvc-951911bd-b7f3-4551-8647-22f1f5d576b3
server: 2564f49129-ysu87.cn-shenzhen.nas.aliyuncs.com
vers: "3"
persistentVolumeReclaimPolicy: Delete
storageClassName: alicloud-nas
# kubectl get pv default-nas-pvc-web-1-pvc-95eee8f9-54f4-42dc-a34f-b554423220eb -oyaml > origin/default-nas-pvc-web-1-pvc-95eee8f9-54f4-42dc-a34f-b554423220eb.yaml
## Retain the StorageClass template.
# kubectl get sc alicloud-nas -oyaml > origin/alicloud-nas.yaml
# cat origin/alicloud-nas.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: alicloud-nas
mountOptions:
- nolock,tcp,noresvport
- vers=3
parameters:
archiveOnDelete: "true"
driver: flexvolume
server: 2564f49129-ysu87.cn-shenzhen.nas.aliyuncs.com:/root/
provisioner: alicloud/nas
reclaimPolicy: Delete
volumeBindingMode: Immediate
Step 2: Create a StorageClass, a PVC, and a PV of the CSI type.
## The existing PVC is nas-pvc-web-0, and the new PVC is nas-pvc-new-web-0. Here, nas-pvc is the name of the volume configured in the StatefulSet, and is now changed to nas-pvc-new.
## Set provisioner to nasplugin.csi.alibabacloud.com.
## Set name to nas-pvc-new-web-0.
## Set storageClassName to alicloud-nas-new.
## Set alicloud-pvname to nas-pvc-new-web-0-pv (You can choose any name. However, to improve readability and ensure global uniqueness of the PVC name, we recommend that you add pv or namespace to the end of the name.)
# cat nas-pvc-new-web-0.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
volume.beta.kubernetes.io/storage-provisioner: nasplugin.csi.alibabacloud.com
labels:
app: nginx
name: nas-pvc-new-web-0
namespace: default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 20Gi
storageClassName: alicloud-nas-new
selector:
matchLabels:
alicloud-pvname: nas-pvc-new-web-0-pv
## Replace the existing PV default-nas-pvc-web-0-pvc-951911bd-b7f3-4551-8647-22f1f5d576b3 with the following new PV:
## Set provisioned-by to nasplugin.csi.alibabacloud.com.
## Set labels to nas-pvc-new-web-0-pv.
## Set name to nas-pvc-new-web-0-pv.
## Change FlexVolume configurations to CSI configurations: Set volumeHandle to the name of the PV.
## Set storageClassName to alicloud-nas-new.
# cat nas-pvc-new-web-0-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: nasplugin.csi.alibabacloud.com
finalizers:
- kubernetes.io/pv-protection
labels:
alicloud-pvname: nas-pvc-new-web-0-pv
name: nas-pvc-new-web-0-pv
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 20Gi
csi:
driver: nasplugin.csi.alibabacloud.com
volumeHandle: nas-pvc-new-web-0-pv
volumeAttributes:
server: "2564f49129-ysu87.cn-shenzhen.nas.aliyuncs.com"
path: "/root/default-nas-pvc-web-0-pvc-951911bd-b7f3-4551-8647-22f1f5d576b3"
mountOptions:
- nolock,tcp,noresvport
- vers=3
persistentVolumeReclaimPolicy: Delete
storageClassName: alicloud-nas-new
Create additional PVCs and PVs following the preceding steps.
Replace the existing PVC nas-pvc-web-1 with the new PVC nas-pvc-new-web-1.
Replace the existing PV default-nas-pvc-web-1-pvc-95eee8f9-54f4-42dc-a34f-b554423220eb with nas-pvc-new-web-1-pv.
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nas-pvc-new-web-0 Bound nas-pvc-new-web-0-pv 20Gi RWX alicloud-nas-new 13m
nas-pvc-new-web-1 Bound nas-pvc-new-web-1-pv 20Gi RWX alicloud-nas-new 10m
Step 3: Update the application to change the volume to nas-pvc-new.
## Change the volume in the StatefulSet template from nas-pvc to nas-pvc-new.
volumeClaimTemplates:
- metadata:
name: nas-pvc-new
spec:
## Delete the existing StatefulSet (volumeClaimTemplates of the StatefulSet cannot be edited), and create a new one.
# kubectl delete sts web
# kubectl apply -f sts.yaml
Check the new application:
# kubectl get pod
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 9m37s
web-1 1/1 Running 0 9m20s
# kubectl describe pod web-0 | grep ClaimName
ClaimName: nas-pvc-new-web-0
# kubectl get pvc nas-pvc-new-web-0
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nas-pvc-new-web-0 Bound nas-pvc-new-web-0-pv 20Gi RWX alicloud-nas-new 17m
Procedure:
The following shows you how to migrate a static OSS PV or PVC in a deployment:
# kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-oss-deploy 1/1 1 1 5m37s
# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-oss-deploy-f49d8d6c4-df892 1/1 Running 0 6m1s
# kubectl describe pod nas-static-7f9446758b-2qfxj |grep ClaimName
ClaimName: pvc-oss
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-oss Bound pv-oss 5Gi RWX 7m23s
# kubectl get pv pv-oss -o yaml
flexVolume:
driver: alicloud/oss
options:
akId: xxxx
akSecret: xxx
bucket: oss-mount-test
otherOpts: -o max_stat_cache_size=0 -o allow_other
url: oss-cn-shanghai.aliyuncs.com
Step 1: Retain the existing PVC (pvc-oss) and PV (pv-oss).
# kubectl get pvc pvc-oss -oyaml > flex.oss-pvc.yaml
# cat flex.oss-pvc.yaml
apiVersion: v1
items:
- apiVersion: v1
kind: PersistentVolumeClaim
name: pvc-oss
namespace: default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
volumeMode: Filesystem
volumeName: pv-oss
status:
accessModes:
- ReadWriteMany
capacity:
storage: 5Gi
phase: Bound
kind: List
metadata:
resourceVersion: ""
selfLink: ""
# kubectl get pv pv-oss -oyaml > flex.oss-pv.yaml
# cat flex.oss-pv.yaml
apiVersion: v1
items:
- apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-oss
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 5Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: pvc-oss
namespace: default
flexVolume:
driver: alicloud/oss
options:
akId: xxx
akSecret: xxx
bucket: oss-mount-test
otherOpts: -o max_stat_cache_size=0 -o allow_other
url: oss-cn-shanghai.aliyuncs.com
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
status:
phase: Bound
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Step 2: Create a PVC (pvc-oss-new) and a PV (pv-oss-new) of the CSI type.
# cat csi.oss-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: oss-csi-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
selector:
matchLabels:
alicloud-pvname: oss-csi-pv
# cat csi.oss-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: oss-csi-pv
labels:
alicloud-pvname: oss-csi-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
csi:
driver: ossplugin.csi.alibabacloud.com
# set volumeHandle same value pv name
volumeHandle: oss-csi-pv
volumeAttributes:
bucket: "oss-mount-test"
url: "oss-cn-shanghai.aliyuncs.com"
otherOpts: "-o max_stat_cache_size=0 -o allow_other"
akId: "***"
akSecret: "***"
path: "/"
# kubectl apply -f csi.oss-pv.yaml
# kubectl apply -f csi.oss-pvc.yaml
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
oss-csi-pvc Bound oss-csi-pv 5Gi RWO 7m15s
pvc-oss Bound pv-oss 5Gi RWX 52m
Step 3: Update the application to apply the new CSI PVC.
# kubectl edit deploy nginx-oss-deploy
volumes:
- name: oss1
persistentVolumeClaim:
claimName: oss-csi-pvc
// Check whether the pod has been restarted.
# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-oss-deploy-d84b9bb86-nxbf5 1/1 Running 0 21s
Through the preceding three steps, an OSS volume mounted through FlexVolume in a deployment is replaced with an OSS volume of the CSI type.
Solution for OSS volumes where secrets are configured:
Procedure
Step 1: Acquire the secret used in the FlexVolume deployment.
# kubectl get secret osssecret -o yaml
apiVersion: v1
data:
akId: xxxx
akSecret: xxxx
kind: Secret
metadata:
name: oss-secret
namespace: default
type: alicloud/oss
Step 2: Use the secret to create a static OSS PVC and PV of the CSI type.
# kubectl apply -f secret-pvc-pv.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: oss-csi-pvc-secret
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
selector:
matchLabels:
alicloud-pvname: oss-csi-pv-secret
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: oss-csi-pv-secret
labels:
alicloud-pvname: oss-csi-pv-secret
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
csi:
driver: ossplugin.csi.alibabacloud.com
volumeHandle: oss-csi-pv-secret // The specified value must be the same as the name of the PV.
nodePublishSecretRef:
name: oss-secret
namespace: default
volumeAttributes:
bucket: "oss"
url: "oss-cn-hangzhou.aliyuncs.com"
otherOpts: "-o max_stat_cache_size=0 -o allow_other"
# kubectl get pvc oss-csi-pvc-secret
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
oss-csi-pvc-secret Bound oss-csi-pv-secret 5Gi RWX 3m35s
Step 3: Change the deployment reference to the PVC that you created in the previous step.
# kubectl edit deploy nginx-oss-deploy
volumes:
- name: oss1
persistentVolumeClaim:
claimName: oss-csi-pvc-secret
// Check whether the pod has been restarted.
# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-oss-deploy-d84bxxrs6-nxbf5 1/1 Running 0 2s
Evolution of the Frontend Architecture and Serverless Practices in DataWorks
Sensitive Data Discovery and Protection: An All-In-One Data Security Solution
2,599 posts | 762 followers
FollowAlibaba Developer - June 17, 2020
Alibaba Developer - February 1, 2021
Alibaba Container Service - November 7, 2024
Alibaba Developer - November 5, 2020
Alibaba Developer - August 18, 2020
Alibaba Developer - June 30, 2020
2,599 posts | 762 followers
FollowProvides scalable, distributed, and high-performance block storage and object storage services in a software-defined manner.
Learn MoreAn encrypted and secure cloud storage service which stores, processes and accesses massive amounts of data from anywhere in the world
Learn MoreSimple, scalable, on-demand and reliable network attached storage for use with ECS instances, HPC and Container Service.
Learn MorePlan and optimize your storage budget with flexible storage services
Learn MoreMore Posts by Alibaba Clouder