Your disk space may become insufficient as your business develops and application data grows. To resolve this issue, you can expand your disk volume. In a Container Service for Kubernetes (ACK) cluster that runs Kubernetes earlier than 1.16, you cannot set the cluster to automatically expand disk volumes. You must manually expand disk volumes. This topic describes how to manually expand a disk volume.
Usage notes
- Limits
Only disks no larger than 32 TiB can be expanded.
- Data backup
Before you expand a disk volume, you must back up the disk data by creating a snapshot of the disk. This prevents data loss when you expand the disk volume.
- Applicable scope
The PV is a statically provisioned disk volume.
- Method 1: Expand the disk volume without service interruption: If the I/O throughput of a disk is high when you expand the file system of the disk, an I/O error may occur in the file system. However, you do not need to restart the application if you choose this method.
- Method 2: Expand a disk volume with service interruption: After the application is stopped, the disk I/O operations are stopped. This ensures data security when you expand the file system of the disk. Your service will be temporarily interrupted if you choose this method.
View disk usage information
A stateful application named web is used in this example to demonstrate how to expand a disk volume by using the preceding two methods. You can perform the following steps to view disk usage information.
- Run the following command to query the pods that are provisioned for the web application:
kubectl get pod | grep web
Expected output:
NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 11h web-1 1/1 Running 0 11h
- Run the following command to query the PVCs that are created for the web application:
kubectl get pvc | grep web
Expected output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE disk-ssd-web-0 Bound d-0jlhaq*** 20Gi RWO alicloud-disk-essd 11h disk-ssd-web-1 Bound d-0jl0j5*** 20Gi RWO alicloud-disk-essd 11h
- Run the following command to query the PVs that are created for the web application:
kubectl get pv | grep web
Expected output:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE d-0jl0j5*** 20Gi RWO Delete Bound default/disk-ssd-web-1 alicloud-disk-essd 11h d-0jlhaq*** 20Gi RWO Delete Bound default/disk-ssd-web-0 alicloud-disk-essd 11h
The output indicates that two disks named
d-0jl0j5***
andd-0jlhaq***
are used by the web application. Both disks are 20 GiB in size. The disks are mounted to two pods separately.
For more information about how to deploy a stateful application, see Use a StatefulSet to create a stateful application.
Method 1: Expand the disk volume without service interruption
Find the corresponding disk based on the PV information, manually expand the disk, and then connect to the node to which the disk is mounted and expand the file system. The following example demonstrates how to expand both disks to 30 GiB.
Step 1: Expand the disks
- Log on to the ECS console.
- In the left-side navigation pane, choose .
- Find the disks named d-0jl0j5*** and d-0jlhaq*** and check the status of the disks. Then, choose in the Actions column for each disk.
- On the Resize Disks page, select Online Resizing, and enter the size to which you want to expand the disk in the Size after Resize section. In this example, set the size to 30 GiB. Note The specified value of Size after Resize must be greater than the current disk capacity.
- Read and select ECS Terms of Service, confirm the fee, and then click Confirm. For more information, see Expand disks.
- Read the notes, click I have read the notes. Resize, and then complete the payment. On the Disks page, check whether the disk is expanded.
Step 2: Expand the file systems
After the disks are expanded, you must expand the file systems. Otherwise, the storage that the application can use is still 20 GiB.
- If an unpartitioned disk is mounted as a PV, you cannot manually create partitions for the disk. Otherwise, the file system may be damaged and data loss may occur.
- If a partitioned disk is mounted as a PV, you must expand the file system after you expand the partitioned disk. For more information, see Extend the partitions and file systems of disks on a Linux instance and Extend the partitions and file systems of a disk on a Windows instance.
- View the ECS instances to which the disks are mounted.
- Connect to the ECS instance to which the disk is mounted and obtain the driver letter of the disk. For more information about how to connect to an ECS instance, see Methods used to connect to ECS instances.
You can use the following methods to obtain the driver letter of the disk.
- Obtain the driver letter of the disk.
- Run the following command to query the driver letter of the disk named d-0jlhaq***:
# Query {pv-name} mount |grep d-0jlhaq***
Expected output:
/dev/vdc on /var/lib/kubelet/plugins/kubernetes.io/csi/pv/d-0jlhaq***/globalmount type ext4 (rw,relatime) /dev/vdc on /var/lib/kubelet/pods/a26d174f-***/volumes/kubernetes.io~csi/d-0jlhaq***/mount type ext4 (rw,relatime)
The output indicates that the driver letter of the
d-0jlhaq***
disk is /dev/vdc.
- Run the following command to expand the file system:
resize2fs /dev/vdc
Expected output:
resize2fs 1.43.5 (04-Aug-2017) Filesystem at /dev/vdc is mounted on /var/lib/kubelet/plugins/kubernetes.io/csi/pv/d-0jlhaq***/globalmount; on-line resizing required old_desc_blocks = 3, new_desc_blocks = 4 The filesystem on /dev/vdc is now 7864320 (4k) blocks long.
- Run the following command to check whether the file system is expanded:
lsblk /dev/vdc
Expected output:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vdc 254:32 0 30G 0 disk /var/lib/kubelet/pods/a26d174f-***/volumes/kubernetes.io~csi/d-0jlhaq***/mount
The output indicates that the size of the vdc file system is expanded to 30 GiB.
Method 2: Expand a disk volume with service interruption
You can stop the application by deleting the StatefulSet or set the value of Replica to 0. Then, you can manually expand the disks and restart the application. The following example demonstrates how to expand both disks to 30 GiB.
Step 1: Delete the pods that are provisioned for the application
- Run the following command to scale the number of pods to 0:
kubectl scale sts web --replicas=0
Expected output:
statefulset.apps/web scaled
- Run the following command to check whether the pods are deleted:
kubectl get pod | grep web
No output is returned. This indicates that the web application is stopped.
Step 2: Expand the disks
- Log on to the ECS console.
- In the left-side navigation pane, choose .
- Find the disks named d-0jl0j5*** and d-0jlhaq*** and check the states of the disks. Then, choose in the Actions column for each disk.
- On the Resize Disks page, select a method to resize the disk, and specify the size to which you want to expand the disk.
- If the disk is in the Unattached state, do not select Online Resizing on the Resize Disks page. Specify the size to which you want to expand the disk in the Size after Resize section. In this example, set the value to 30 GiB.
- If the disk is in the In Use state, select Online Resizing on the Resize Disks page, and specify the size to which you want to expand the disk in the Size after Resize section.
Note The specified value of Size after Resize must be greater than the current disk capacity. - Read and select ECS Terms of Service, confirm the fee, and then click Confirm. For more information, see Step 1: Resize a disk to extend its capacity.
- Read the notes, click I have read the notes. Resize, and then complete the payment.
Step 3: Expand the file systems
After the disks are expanded, you must expand the file systems. Otherwise, the storage that the application can use is still 20 GiB.
- If an unpartitioned disk is mounted as a PV, you cannot manually create partitions for the disk. Otherwise, the file system may be damaged and data loss may occur.
- If a partitioned disk is mounted as a PV, you must expand the file system after you expand the partitioned disk. For more information, see Extend the partitions and file systems of disks on a Linux instance and Extend the partitions and file systems of a disk on a Windows instance.
- Optional:Mount the disks to an ECS instance. Note You must mount the disks to an ECS instance before you can expand the file systems.After the disk is mounted, the status of the disk changes to In Use.
- Connect to the ECS instance to which the disk is mounted and obtain the driver letter of the disk. For more information about how to connect to an ECS instance, see Methods used to connect to ECS instances.
- Run the following command to obtain the driver letter of the disk:
for device in `ls /sys/block | grep vd`; do cat /sys/block/$device/serial | grep 0jlhaq*** && echo $device; done
Note The ID of the expanded disk isd-0jlhaq***
.0jlhaq***
is the string that followsd-
. - Optional:If you cannot obtain the driver letter of the disk by running the preceding command, perform the following operations:
- Unmount the disk and run the
ls /dev/vd*
command to query the list of disks. - Mount the disk and run the
ls /dev/vd*
command to query the list of disks. - Compare the lists that are returned. The disk that appears only in the second list is the one that you mounted.
- Unmount the disk and run the
- Run the following command to obtain the driver letter of the disk:
- Run the following command to expand the file system:
resize2fs /dev/vdb
Note /dev/vdc is the driver letter of the disk obtained in the Step 2.Expected output:
resize2fs 1.43.5 (04-Aug-2017) Resizing the filesystem on /dev/vdb to 7864320 (4k) blocks. The filesystem on /dev/vdb is now 7864320 (4k) blocks long.
- Check whether the file system is expanded.
Step 4: Restart the application
- Run the following command to scale the number of pods to 2:
kubectl scale sts web --replicas=2
Expected output:
statefulset.apps/web scaled
- Run the following command to check whether the pod has recovered:
kubectl get pod | grep web
Expected output:
NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 74s web-1 1/1 Running 0 42s
- Run the following command to query the size of the specified file system:
kubectl exec web-0 -- df /data
Expected output:
Filesystem 1K-blocks Used Available Use% Mounted on /dev/vdb 30832548 45036 30771128 1% /data
The output indicates that the size of the /dev/vdb file system is expanded to 30 GiB.
What can I do if the following message appears after I run the resize2fs
command?
resize of device /dev/xxx failed: exit status 1 resize2fs output: resize2fs xxx(version)
Please run `e2fsck -f /dev/xxx` first
The file systems are not consistent, which causes I/O errors.
SolutionRun the e2fsck -f /dev/xxx
command and then expand the file systems.