The storage monitoring feature of Container Service for Kubernetes (ACK) uses Managed Service for Prometheus to monitor the usage of disks and File Storage NAS (NAS) volumes on the node side. This topic describes how to use csi-plugin to monitor storage resources on the node side and provides an example on how to configure alert rules.
Prerequisites
Only csi-plugin v1.18.8.46-afb19e46-aliyun and later versions support both disk monitoring and NAS file system monitoring. Make sure that the YAML file and image of csi-plugin are updated to the latest version. You can update csi-plugin to the latest version in the ACK console. For more information, see Upgrade csi-plugin by using the ACK console.
Background information
Disk volumes or NAS volumes are commonly used to store data in ACK clusters. By default, the latest version of csi-plugin is automatically installed in ACK clusters to monitor persistent volume claims (PVCs) with the storage monitoring feature. Managed Service for Prometheus is used to monitor the status of PVCs and configure the capacity threshold and IOPS threshold. This way, alerts can be generated when the usage of the PVC is abnormal.
Update csi-plugin by using the ACK console
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose .
Click the Storage tab, find csi-plugin and click Upgrade.
How to monitor storage resources
Only disks and NAS file systems can be monitored on the node side. To view the monitoring data that is provided by Managed Service for Prometheus, perform the following steps:
To view storage resource monitoring data, you need to first enable Managed Service for Prometheus. For more information, see Enable Managed Service for Prometheus.
After the storage monitoring feature is enabled, the component will automatically report metrics to Managed Service for Prometheus. These metrics are considered custom metrics. You may need to pay for custom metrics. We recommend that you read and understand the Billing overview topic of Managed Service for Prometheus before you enable the storage monitoring feature. The fee varies based on the size of your cluster and the number of applications deployed in the cluster. You can use the resource usage feature to monitor and manage the usage of your resources.
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, choose .
Click the Storage Monitoring tab and then click the CSI - Nodes tab to view the node dashboard.
Filters
Filter
Description
StorageType
The type of storage resource. A value of disk indicates disks. A value of nas indicates NAS file systems.
Namespace
The namespace to which the PVC belongs
PVC
The name of the PVC.
Metrics
Metric
Description
Unit
IOPS
The read/write IOPS of the PVC
ops/s (number of reads/writes per second)
Latency(avg)
The average latency at which the PVC processes read/write requests
ms (milliseconds)
ThroughPut
The throughput of the PVC
GB
IO Count In Queue
The number of I/O requests in the queue
Count
Used Capacity
The used storage of the PVC
GB
Total Capacity
The storage capacity of the PVC
GB
Free Capacity
The available storage of the PVC
GB
Examples of disk monitoring
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, choose .
On the Prometheus Monitoring page, click the Storage Monitoring tab and then click the CSI - Nodes tab.
Specify StorageType, Namespace, and PVC name of the disk volume. Then, you can view the current capacity of the disk volume in the Total Capacity section.
In this example, set StorageType to disk, Namespace to default, and PVC to pvc-disk-static-create1.
The preceding figure shows that the total capacity of the disk is 39.25 GiB. 48.2 MiB has been used and 39.18 GiB is available. Write data into the disk.
Run the following command to query the name of the application to which the disk is mounted:
kubectl get podExpected output:
NAME READY STATUS RESTARTS AGE deployment-disk-static-create1-67f4b5cfdc-p5q7t 1/1 Running 0 20mRun the following command to log on to the application:
kubectl exec deployment-disk-static-create1-67f4b5cfdc-p5q7t -ti shRun the following command to write 20 GiB of data into the /data directory where the disk is mounted:
cd /data /usr/bin/fio -direct=1 -iodepth=256 -rw=randwrite -ioengine=libaio -bs=4k -numjobs=1 -runtime=100 -group_reporting -name=Rand_Write_Testing -filename=./test -size=20G
On the Prometheus Monitoring page, click the Storage Monitoring tab and then click the CSI - Nodes tab to view the monitoring data of the disk volume.
The preceding figure shows that the write IOPS is 4,000, the throughput is 15 MB/s, 20 GiB of data is written, the remaining disk space is 19 GiB, and the average latency is 67 milliseconds.
Examples of NAS file system monitoring
To use NAS file system monitoring, the persistent volume (PV) from which metrics are collected must meet the following requirements:
The NAS file system is managed by Container Network File System (CNFS) and the NAS file system is of the Capacity type.
The PV is a dynamically provisioned volume created by using a StorageClass.
Write data into the directory to which the NAS volume is mounted.
Run the following command to query the name of the application to which the NAS volume is mounted:
kubectl get podExpected output:
NAME READY STATUS RESTARTS AGE cnfs-nas-deployment-84f6cdf6cc-wvjjq 1/1 Running 0 20mRun the following command to log on to the application:
kubectl exec cnfs-nas-deployment-84f6cdf6cc-wvjjq -ti shRun the following command to write 20 GiB of data into the /data directory where the NAS volume is mounted:
cd /data /usr/bin/fio -direct=1 -iodepth=256 -rw=randwrite -ioengine=libaio -bs=4k -numjobs=1 -runtime=100 -group_reporting -name=Rand_Write_Testing -filename=./test -size=20G
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, choose .
On the Prometheus Monitoring page, click the Storage Monitoring tab and then click the CSI - Nodes tab.
Specify StorageType, Namespace, and PVC name of the NAS volume. Then, you can view the current capacity of the NAS volume in the Total Capacity section.
In this example, set StorageType to nas, Namespace to default, and PVC to cnfs-nas-pvc.
The preceding figure shows that the write IOPS is approximately 15,000, the throughput increases from 10 MB/s to 60 MB/s, and the average latency is 15 ms.
Related topics
For more information about how to expand disk volumes, see Expand disk volumes and Expand a NAS volume.
For more information about how to query the storage status of an ACK cluster, see Use storage-operator to monitor the storage resources of an ACK cluster.