All Products
Search
Document Center

Container Service for Kubernetes:Manually expand a disk volume

Last Updated:Aug 09, 2024

Your disk space may become insufficient as your business develops and application data grows. To resolve this issue, you can expand your disk volume. In a Container Service for Kubernetes (ACK) cluster that runs Kubernetes earlier than 1.16, you cannot set the cluster to automatically expand disk volumes. You must manually expand disk volumes. This topic describes how to manually expand a disk volume.

Usage notes

To expand a disk volume, you must expand the corresponding disk on the Elastic Compute Service (ECS) side. The expansion does not affect resources in the cluster. This means that the capacity displayed in the persistent volume claim (PVC) and persistent volume (PV) does not change.

  • Limits

    • Only disks no larger than 32 TiB can be expanded.

    • To ensure that the disk capacity values displayed in the PV and PVC are same as the actual disk capacity, we recommend that you perform the steps in the Expand a disk volume without service interruptions topic if your cluster runs Kubernetes 1.16 or later.

  • Data backup

    Before you expand a disk volume, you must back up the disk data by creating a snapshot of the disk. This prevents data loss when you expand the disk volume.

  • Applicable scope

    The PV to be expanded is a statically provisioned disk volume.

To ensure the stability of disk volumes, ACK allows you to use the following methods to expand a disk volume:

View disk usage information

A stateful application named MySQL is used in this example to demonstrate how to expand a disk volume by using the preceding two methods. You can perform the following steps to view disk usage information.

  1. Run the following command to query the pods that are provisioned for the MySQL application:

    kubectl get pod | grep mysql

    Expected output:

    NAME                               READY   STATUS    RESTARTS   AGE
    mysql-0                            1/1     Running   0          11h
    mysql-1                            1/1     Running   0          11h
  2. Run the following command to query the PVCs that are created for the MySQL application:

    kubectl get pvc 

    Expected output:

    NAME             STATUS   VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS                     AGE
    disk-pvc-1       Bound    d-0jlhaq***   20Gi       RWO            alicloud-disk-topology-alltype   11h
    disk-pvc-2       Bound    d-0jl0j5***   20Gi       RWO            alicloud-disk-topology-alltype   11h
  3. Run the following command to query the PVs that are created for the MySQL application:

    kubectl get pv

    Expected output:

    NAME          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS                   REASON    AGE
    d-0jl0j5***   20Gi       RWO            Delete           Bound    default/disk-pvc-1    alicloud-disk-topology-alltype            11h
    d-0jlhaq***   20Gi       RWO            Delete           Bound    default/disk-pvc-1    alicloud-disk-topology-alltype            11h

    The output indicates that two disks named d-0jl0j5*** and d-0jlhaq*** are used by the MySQL application. Both disks are 20 GiB in size. The disks are mounted to two pods separately.

For more information about how to deploy a stateful application, see Use a StatefulSet to create a stateful application.

Method 1: Expand the disk volume without service interruption

Find the corresponding disk based on the PV information, manually expand the disk, and then connect to the node to which the disk is mounted and expand the file system. The following example demonstrates how to expand both disks to 30 GiB.

Step 1: Expand the disks

  1. Log on to the ECS console.

  2. In the left-side navigation pane, choose Storage & Snapshots > Block Storage (Disks) .

  3. Find the disks named d-0jl0j5*** and d-0jlhaq*** and check the status of the disks. Then, choose More > Resize Disk in the Actions column for each disk.

  4. On the Resize Disks page, select Online Resizing, and enter the size to which you want to expand the disk in the Size after Resize section. In this example, set the size to 30 GiB.

    Note

    The specified value of Size after Resize must be greater than the current disk capacity.

  5. Read and select ECS Terms of Service, confirm the fee, and then click Confirm.

    For more information, see Expand disks.

  6. Read the notes, click I have read the notes. Resize, and then complete the payment.

    On the Disks page, check whether the disk is expanded.

Step 2: Expand the file systems

After the disks are expanded, you must expand the file systems. Otherwise, the storage that the application can use is still 20 GiB.

Important

This step is intended for unpartitioned disks that are used in Kubernetes. We recommend that you do not use partitioned disks in Kubernetes.

  1. View the ECS instances to which the disks are mounted.

    1. Log on to the ECS console.

    2. In the left-side navigation pane, choose Storage & Snapshots > Block Storage (Disks) .

    3. Find the disks named d-0jl0j5*** and d-0jlhaq***, and click the name of each disk.

    4. In the Associated Instances list of the Basic Information tab, click the instance to which the disk is mounted. Then, view information in the Network Information section of the Instance Details tab.

      Note

      You can also view the ECS instances to which the disks are mounted in the ACK console. For more information, see View pods.

  2. Connect to the ECS instance to which the disk is mounted and obtain the driver letter of the disk.

    For more information about how to connect to an ECS instance, see Methods used to connect to ECS instances.

    You can use the following methods to obtain the driver letter of the disk.

    • Obtain the driver letter of the disk.

    • Run the following command to query the driver letter of the disk named d-0jlhaq***:

      # Query {pv-name}.
      mount |grep d-0jlhaq*** 

      Expected output:

      /dev/vdc on /var/lib/kubelet/plugins/kubernetes.io/csi/pv/d-0jlhaq***/globalmount type ext4 (rw,relatime)
      /dev/vdc on /var/lib/kubelet/pods/a26d174f-***/volumes/kubernetes.io~csi/d-0jlhaq***/mount type ext4 (rw,relatime)

      The output indicates that the driver letter of the d-0jlhaq*** disk is /dev/vdc.

  3. Run the following command to expand the file system:

    resize2fs /dev/vdc
    Note

    /dev/vdc is the driver letter obtained in Step 2.

    Expected output:

    resize2fs 1.43.5 (04-Aug-2017)
    Filesystem at /dev/vdc is mounted on /var/lib/kubelet/plugins/kubernetes.io/csi/pv/d-0jlhaq***/globalmount; on-line resizing required
    old_desc_blocks = 3, new_desc_blocks = 4
    The filesystem on /dev/vdc is now 7864320 (4k) blocks long.
  4. Run the following command to check whether the file system is expanded:

    lsblk /dev/vdc

    Expected output:

    NAME MAJ:MIN  RM SIZE RO TYPE MOUNTPOINT
    vdc  254:32   0  30G  0  disk /var/lib/kubelet/pods/a26d174f-***/volumes/kubernetes.io~csi/d-0jlhaq***/mount

    The output indicates that the size of the vdc file system is expanded to 30 GiB.

Method 2: Expand a disk volume with service interruption

You can stop the application by deleting the StatefulSet or set the value of Replica to 0. Then, you can manually expand the disks and restart the application. The following example demonstrates how to expand both disks to 30 GiB.

Step 1: Delete the pods that are provisioned for the application

  1. Run the following command to scale the number of pods to 0:

    kubectl scale sts mysql --replicas=0

    Expected output:

    statefulset.apps/mysql scaled
  2. Run the following command to check whether the pods are deleted:

    kubectl get pod | grep mysql

    No output is returned. This indicates that the MySQL application is stopped.

Step 2: Expand the disks

  1. Log on to the ECS console.

  2. In the left-side navigation pane, choose Storage & Snapshots > Block Storage (Disks) .

  3. Find the disks named d-0jl0j5*** and d-0jlhaq*** and check the states of the disks. Then, choose More > Resize Disk in the Actions column for each disk.

  4. On the Resize Disks page, select a method to resize the disk, and specify the size to which you want to expand the disk.

    • If the disk is in the Unattached state, do not select Online Resizing on the Resize Disks page. Specify the size to which you want to expand the disk in the Size after Resize section. In this example, set the value to 30 GiB.

    • If the disk is in the In Use state, select Online Resizing on the Resize Disks page, and specify the size to which you want to expand the disk in the Size after Resize section.

    Note

    The specified value of Size after Resize must be greater than the current disk capacity.

  5. Read and select ECS Terms of Service, confirm the fee, and then click Confirm.

  6. Read the notes, click I have read the notes. Resize, and then complete the payment.

Step 3: Expand the file systems

After the disks are expanded, you must expand the file systems. Otherwise, the storage that the application can use is still 20 GiB.

Important

This step is intended for unpartitioned disks that are used in Kubernetes. We recommend that you do not use partitioned disks in Kubernetes.

  1. Optional: Mount the disks to an ECS instance.

    Note

    You must mount the disks to an ECS instance before you can expand the file systems.

    1. Log on to the ECS console.

    2. In the left-side navigation pane, choose Storage & Snapshots > Block Storage (Disks) .

    3. For a disk that is in the Unattached state, you can choose More > Attach in the Actions column of the disk to mount the disk.

    4. In the Attach Disk dialog box, select an ECS instance from the drop-down list, and configure the settings that correspond to releasing disks.

      Parameter

      Description

      Destination Instance

      Select the instance to which you want to mount the disk.

      Release Disk with Instance

      If you select this option, the disk is automatically released when the instance is released. If you do not select this option, the disk is retained when its associated instance is released.

      Note

      If the disk that you want to mount is a system disk that was unmounted from another instance, the instance specified by Release Disk with Instance refers to the instance from which the disk was unmounted, not the current instance.

      Delete Automatic Snapshots While Releasing Disk

      If you select this option, the automatic snapshots of the disk are released when the disk is released. To retain the snapshots, do not select this option.

    5. Click Attach.

    After the disk is mounted, the status of the disk changes to In Use.

  2. Connect to the ECS instance to which the disk is mounted and obtain the driver letter of the disk.

    For more information about how to connect to an ECS instance, see Methods used to connect to ECS instances.

    • Run the following command to obtain the driver letter of the disk:

      for device in `ls /sys/block | grep vd`; do 
        cat /sys/block/$device/serial | grep 0jlhaq*** && echo $device; 
      done
      Note

      The ID of the expanded disk is d-0jlhaq***. 0jlhaq*** is the string that follows d-.

    • Optional: If you cannot obtain the driver letter of the disk by running the preceding command, perform the following operations:

      1. Unmount the disk and run the ls /dev/vd* command to query the list of disks.

      2. Mount the disk and run the ls /dev/vd* command to query the list of disks.

      3. Compare the lists that are returned. The disk that appears only in the second list is the one that you mounted.

  3. Run the following command to expand the file system:

    resize2fs /dev/vdb
    Note

    /dev/vdc is the driver letter of the disk obtained in the Step 2.

    Expected output:

    resize2fs 1.43.5 (04-Aug-2017)
    Resizing the filesystem on /dev/vdb to 7864320 (4k) blocks.
    The filesystem on /dev/vdb is now 7864320 (4k) blocks long.
  4. Check whether the file system is expanded.

    1. Run the following command to create a temporary folder named /mnt/disk/ and mount the disk to the folder:

      mkdir /mnt/disk
      mount /dev/vdb /mnt/disk/
    2. Run the following command to query the size of the specified file system:

      df /mnt/disk/

      Expected output:

      Filesystem     1K-blocks  Used Available Use% Mounted on
      /dev/vdb        30832548 45036  30771128   1% /mnt/disk

      The output indicates that the /dev/vdb folder can use at most 30 GiB of storage. This indicates that the file system is expanded.

    3. Run the following command to unmount the disk from the temporary folder:

      umount /mnt/disk

Step 4: Restart the application

  1. Run the following command to scale the number of pods to 2:

    kubectl scale sts mysql --replicas=2

    Expected output:

    statefulset.apps/mysql scaled
  2. Run the following command to check whether the pod has recovered:

    kubectl get pod | grep mysql

    Expected output:

    NAME         READY   STATUS    RESTARTS   AGE
    mysql-0        1/1     Running   0          74s
    mysql-1        1/1     Running   0          42s
  3. Run the following command to query the size of the specified file system:

    kubectl exec mysql-0 -- df /data

    Expected output:

    Filesystem     1K-blocks  Used Available Use% Mounted on
    /dev/vdb        30832548 45036  30771128   1% /data

    The output indicates that the size of the /dev/vdb file system is expanded to 30 GiB.

What can I do if the following message appears after I run the resize2fs command?

resize of device /dev/xxx failed: exit status 1 resize2fs output: resize2fs xxx(version)
Please run `e2fsck -f /dev/xxx` first

Cause

The file systems are not consistent, which causes I/O errors.

Solution

Run the e2fsck -f /dev/xxx command and then expand the file systems.