Your disk space may become insufficient as your business develops and application data grows. To resolve this issue, you can define one or more expansion policies to automatically expand a disk volume when the usage of the disk volume exceeds a specified threshold value. This topic describes how to automatically expand a disk volume.
Prerequisites
The storage-operator component is installed in the cluster and the component version is v1.18.8.28-18cca7b-aliyun or later. For more information, see storage-operator.
Usage notes
Only ACK clusters that run Kubernetes 1.16 and later are support. You can expand disks but cannot shrink disks.
You can expand only disks that are mounted to pods.
You can expand only disks that can be resized. For more information, see ResizeDisk.
The maximum interval between two expansion operations is 2 minutes. The disk expansion process requires 1 minute. Make sure that you do not exhaust the disk space within 3 minutes.
The maximum capacity of a disk is 2,000 GiB.
Step 1: Enable automatic scale-up
The default storage-auto-expander in the storage-operator component manages automatic storage expansion. To enable automatic scale-up, run the following command to modify the ConfigMap file of the storage-operator component.
kubectl patch configmap/storage-operator \
-n kube-system \
--type merge \
-p '{"data":{"storage-auto-expander":"{\"imageRep\":\"acs/storage-auto-expander\",\"imageTag\":\"\",\"install\":\"true\",\"template\":\"/acs/templates/storage-auto-expander/install.yaml\",\"type\":\"deployment\"}"}}'Step 2: Configure an auto expansion policy
Configure an auto expansion policy in StorageClass.
We recommend that you create an alicloud-disk-topology-alltype StorageClass in each ACK cluster. The StorageClass can automatically select disk types to avoid disk creation failures caused by instance type limits or insufficient disk stock in the specified zone.
If your cluster runs an earlier Kubernetes version, you can use the following method to manually create an alicloud-disk-topology-alltype StorageClass .
Use the following template to create a file named storageclass.yaml.
allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: alicloud-disk-topology-alltype parameters: type: cloud_essd,cloud_ssd,cloud_efficiency provisioner: diskplugin.csi.alibabacloud.com reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumerRun the following command to create a StorageClass:
kubectl create -f storageclass.yaml
Use a custom resource definition (CRD) to create an auto expansion policy.
Use the following template to create a file named StorageAutoScalerPolicy.yaml:
apiVersion: storage.alibabacloud.com/v1alpha1 kind: StorageAutoScalerPolicy metadata: name: hybrid-expand-policy spec: pvcSelector: matchLabels: app: mysql namespaces: - default - mysql conditions: - name: condition1 key: volume-capacity-used-percentage operator: Gt values: - "80" actions: - name: action1 type: volume-expand params: scale: 50Gi limits: 100Gi - name: action2 type: volume-expand params: scale: 50% limits: 300GiParameter
Description
pvcSelector
Selects a persistent volume claim (PVC) based on labels. In this example, mysql is used.
namespaces
Specifies the namespace of the PVC. If multiple namespaces are specified, the logical operator between the namespaces is OR. Default value: default. In this example, default and mysql are specified.
conditions
Specifies the conditions that trigger the action. The logical operator between multiple conditions is AND. Each condition includes the following parameters:
key: specifies the type of the metric.
volume-capacity-used-percentage: specifies that the storage usage threshold is expressed in percentage.
operator: specifies the operator, which can be Gt (greater than), Lt (less than), Eq (equal to), or Ne (not equal to). The parameter value is not case-sensitive.
values: specifies the threshold value.
In the example, the condition specifies that the action is triggered when the capacity usage of the PVC exceeds 80%.
actions
Specifies the operations that are performed when the preceding conditions are met. Multiple operations are allowed. Each action includes the following parameters:
type: specifies the type of action. Valid value: volume-expand.
scale: specifies the size of the storage that you want to add to the volume. Valid units: GiB and percentage (%).
limits: specifies the maximum capacity of the PVC if the action is performed.
If multiple actions are specified in the actions section, the first action that meets the limits is performed. The other actions are skipped. In this example, if action1 meets the limits, action1 is performed and action2 is skipped.
In this example, action1 specifies the disk is expanded by 50 GiB if the disk size is no greater than 100 GiB. The maximum disk size is 100 GiB.
In this example, action2 specifies that the disk is expanded by 50% if the disk size is greater than 100 GiB but smaller than 300 GiB. The maximum disk size is 300 GiB.
Run the following command to create an auto expansion policy:
kubectl create -f StorageAutoScalerPolicy.yaml
Create a StatefulSet.
Use the following template to create a file named StatefulSet.yaml.
apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql spec: selector: matchLabels: app: mysql serviceName: mysql replicas: 3 template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:5.7 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-pass key: password ports: - containerPort: 80 name: mysql volumeMounts: - name: pvc-disk mountPath: /data volumeClaimTemplates: - metadata: name: pvc-disk labels: app: mysql spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "alicloud-disk-topology-alltype" resources: requests: storage: 25Gi --- apiVersion: v1 kind: Secret metadata: name: mysql-pass type: Opaque data: username: dGVz**** password: dGVzdDEt****Run the following command to create a StatefulSet:
kubectl create -f StatefulSet.yaml
Step 3: Verify automatic disk expansion
Write data to the mounted directory to increase the disk usage to more than 80%.
Run the following command to write data to the mounted directory:
dd if=<Data path> of=<Mounted directory>Run the following command to query the disk details:
df -h | grep d-****1Expected output:
Filesystem Size Used Avail Use% Mounted on /dev/vde 25G 24G 1.5G 95% /var/lib/kubelet/plugins/kubernetes.io/csi/pv/d-****1/globalmount
Run the following command to query events that are related to disk expansion:
Disk expansion is triggered when the disk usage exceeds 80%. In this case, action1 meets the limits. Therefore, action1 is performed to expand the disk.
kubectl get eventsExpected output:
101s Warning StartExpand persistentvolumeclaim/pvc-disk-mysql-1 Start to expand of pvc pvc-disk-mysql-1 from 25Gi to 75Gi, usedCapacityPercentage:94%, freeSize:1472MB. 101s Warning ExternalExpanding persistentvolumeclaim/pvc-disk-mysql-1 Ignoring the PVC: did't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC. 101s Normal Resizing persistentvolumeclaim/pvc-disk-mysql-1 External resizer is resizing volume d-****1 97s Normal FileSystemResizeRequired persistentvolumeclaim/pvc-disk-mysql-1 Require file system resize of volume on node 96s Warning SkipExpand persistentvolumeclaim/pvc-disk-mysql-1 Pvc pvc-disk-mysql-1 is expanding status from 25Gi to 75Gi, this action action2 will skip.Run the following command to query the capacity of the PVC:
kubectl get pvcExpected output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-disk-mysql-0 Bound d-****0 25Gi RWO alicloud-disk-topology-alltype 22m pvc-disk-mysql-1 Bound d-****1 75Gi RWO alicloud-disk-topology-alltype 21m pvc-disk-mysql-2 Bound d-****2 25Gi RWO alicloud-disk-topology-alltype 21mThe expected output shows that the disk
d-****1is expanded to 75 GiB.Repeat Step 1 to increase the disk usage to more than 80%. This expands the disk again.
Run the following command to query events that are related to disk expansion:
kubectl get eventsExpected output:
7m22s Warning StartExpand persistentvolumeclaim/pvc-disk-mysql-1 Start to expand of pvc pvc-disk-mysql-1 from 100Gi to 150Gi, usedCapacityPercentage:95%, freeSize:3732MB. 5m2s Warning ExternalExpanding persistentvolumeclaim/pvc-disk-mysql-1 Ignoring the PVC: did't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC. 2m4s Normal Resizing persistentvolumeclaim/pvc-disk-mysql-1 External resizer is resizing volume d-****1 3m4s Normal FileSystemResizeRequired persistentvolumeclaim/pvc-disk-mysql-1 Require file system resize of volume on node 5m59s Warning SkipExpand persistentvolumeclaim/pvc-disk-mysql-1 Pvc pvc-disk-mysql-1 is expanding status from 100Gi to 150Gi, this action action1 will skip.The limits parameter of action1 specifies that the maximum disk capacity is 100 GiB. Therefore, the disk size is expanded from 75 GiB to 100 GiB, as shown in the expected output.
Repeat Step 1 to increase the disk usage to more than 80%. This expands the disk a third time.
Run the following command to query events that are related to disk expansion:
kubectl get eventsExpected output:
7m22s Warning StartExpand persistentvolumeclaim/pvc-disk-mysql-1 Start to expand of pvc pvc-disk-mysql-1 from 100Gi to 150Gi, usedCapacityPercentage:95%, freeSize:3732MB. 5m2s Warning ExternalExpanding persistentvolumeclaim/pvc-disk-mysql-1 Ignoring the PVC: did't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC. 2m4s Normal Resizing persistentvolumeclaim/pvc-disk-mysql-1 External resizer is resizing volume d-****1 3m4s Normal FileSystemResizeRequired persistentvolumeclaim/pvc-disk-mysql-1 Require file system resize of volume on node 5m59s Warning SkipExpand persistentvolumeclaim/pvc-disk-mysql-1 Pvc pvc-disk-mysql-1 is expanding status from 100Gi to 150Gi, this action action1 will skip.The limits parameter of action1 is 100 GiB. Therefore, this action is not performed. The expected output shows that action2 is performed to expand the disk size from 100 GiB to 150 GiB.
Repeat Step 1 to increase the disk usage to more than 80%. This expands the disk a fourth time.
Run the following command to query events that are related to disk expansion:
kubectl get eventsExpected output:
0s Warning StartExpand persistentvolumeclaim/pvc-disk-mysql-1 Start to expand of pvc pvc-disk-mysql-1 from 150Gi to 225Gi, usedCapacityPercentage:94%, freeSize:7637MB. 0s Warning ExternalExpanding persistentvolumeclaim/pvc-disk-mysql-1 Ignoring the PVC: did't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC. 0s Normal Resizing persistentvolumeclaim/pvc-disk-mysql-1 External resizer is resizing volume d-****1 0s Normal FileSystemResizeRequired persistentvolumeclaim/pvc-disk-mysql-1 Require file system resize of volume on node 0s Warning SkipExpand persistentvolumeclaim/pvc-disk-mysql-1 Pvc pvc-disk-mysql-1 is expanding status from 150Gi to 225Gi, this action action1 will skip.The expected output shows that action2 is performed to expand the disk size from 150 GiB to 225 GiB.
Repeat Step 1 to increase the disk usage to more than 80%. This expands the disk a fifth time.
Run the following command to query events that are related to disk expansion:
kubectl get eventsExpected output:
0s Warning StartExpand persistentvolumeclaim/pvc-disk-mysql-1 Start to expand of pvc pvc-disk-mysql-1 from 225Gi to 300Gi, usedCapacityPercentage:94%, freeSize:7637MB. 0s Warning ExternalExpanding persistentvolumeclaim/pvc-disk-mysql-1 Ignoring the PVC: did't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC. 0s Normal Resizing persistentvolumeclaim/pvc-disk-mysql-1 External resizer is resizing volume d-****1 0s Normal FileSystemResizeRequired persistentvolumeclaim/pvc-disk-mysql-1 Require file system resize of volume on node 0s Warning FileSystemResizeSuccessful persistentvolumeclaim/pvc-disk-mysql-1 MountVolume.NodeExpandVolume succeeded for volume "d-****1"The limits parameter of action2 specifies that the maximum disk capacity is 300 GiB. Therefore, the disk size is expanded from 225 GiB to 300 GiB, as shown in the expected output.
Disk expansion is not triggered when the disk usage exceeds 80% again.
Reference
If you have questions when you use disk volumes, see FAQ about disk volumes for troubleshooting.