NAS performance troubleshooting tool - File Storage NAS - Alibaba Cloud Documentation Center

The nasiostat tool is developed by the Alibaba Cloud File Storage NAS (NAS) team to collect real-time metrics such as the latency of Network File System (NFS) operations on Linux clients.

Introduction

In the Linux client kernel, when the NFS client performs NFS operations initiated at the application layer, metrics such as the operation latency are recorded in the /proc/self/mountstats file of the client. The nasiostat tool is designed to read and parse the file to display the metric results in a more understandable way. Therefore, the tool is very lightweight at run time and does not occupy too much CPU or memory resources.

Metrics

Metric	Description
ops/s	The number of I/O or metadata operations completed per second.
kB/s	The throughput per second.
kB/op	The average block size of all I/O or metadata operations per second.
avg RTT (ms)	The average duration from the time that an NFS client sends an I/O or metadata request until the time the NFS client receives a response. The duration includes the following parts: the duration that the I/O or metadata request is transmitted over the network (T2), the duration that the NAS server processes the request (T3), and the duration that the response reaches the NFS client (T4).
avg exe (ms)	The duration from the time that the NFS client creates the I/O or metadata request in its kernel until the request is completed. The duration includes the following parts: the duration that the request is queued at the kernel of the NFS client (T1), T2, T3, and T4.
avg queue (ms)	The duration that the I/O or metadata request is queued at the kernel of the NFS client (T1).

avg RTT, avg exe, and avg queue

Download nasiostat

Download the script of the nasiostat tool.
- If your Elastic Compute Service (ECS) instance is accessible over the Internet, run the following command to download the script of the nasiostat tool:
```
wget https://raw.githubusercontent.com/aliyun/nas-client-tools/master/linux_client/nasiostat -O /tmp/nasiostat
```
- If your ECS instance is inaccessible over the Internet, you can download the script of the nasiostat tool to your local computer and then upload it to the ECS instance.
  1. Download nasiostat to your local computer.
  2. Upload the downloaded nasiostat script to the ECS instance. For more information, see Upload a file to a Linux instance.
Run the following command to grant the execute permissions to the nasiostat script:
```
sudo chmod +x /tmp/nasiostat
```

View the performance of NFS operations

Run the following command to view the performance of NFS operations:

/tmp/nasiostat 1 10 -A write,read,getattr,setattr,access,rename,lookup,create,readdir,readdirplus,remove <NAS mount directory>

The following list describes the parameters. Replace the values of these parameters with the actual values.

1: the frequency at which data samples are collected. Unit: seconds.
10: the number of times that data samples are collected.
-A: the sampled NFS operations, such as write, read, and getattr. For more information about common NFS operations, see Common NFS commands.
<NAS mount directory>: the mount directory where the NAS file system is mounted. You can run the mount -l | grep nas command to view the mount directory.
Take 1ca404****.cn-hangzhou.nas.aliyuncs.com:/ /mnt as an example:
- 1ca404****.cn-hangzhou.nas.aliyuncs.com: the domain name of the mount target. To view the domain name, perform the following steps: Log on to the NAS console. In the left-side navigation pane, click File System List. On the File System List page, find the file system and click Manage in the Actions column. On the page that appears, click Mount Targets. On the Mount Targets tab, view the domain name of the mount target. For more information, see View the domain name of a mount target.
- /: the directory of the NAS file system.
- /mnt: the mount directory that resides on the current ECS instance.

Sample output:

           ops/s     concurrency      bklogqueue       sendqueue       pendqueue
         777.085               2               0               1               1
op                 ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)  avg queue (ms)
write              0.722         738.623        1023.263        0 (0.0%)         155.534         823.704         667.887
read             776.335        3296.389           4.246        0 (0.0%)           7.971           8.085           0.091
getattr            0.005           0.001           0.218       1 (14.3%)           2.857           2.857           0.000
setattr            0.001           0.000           0.262        0 (0.0%)           1.000           1.000           0.000
access             0.004           0.001           0.230        0 (0.0%)           0.200           0.200           0.000
rename             0.000           0.000           0.000        0 (0.0%)           0.000           0.000           0.000
lookup             0.001           0.000           0.234        0 (0.0%)           0.000           0.000           0.000
create             0.001           0.000           0.379        0 (0.0%)           1.000           1.000           0.000
readdir            0.000           0.000           0.000        0 (0.0%)           0.000           0.000           0.000
readdirplus        0.000           0.000           0.000        0 (0.0%)           0.000           0.000           0.000
remove             0.000           0.000           0.000        0 (0.0%)           0.000           0.000           0.000

Common NFS commands

Command	Description
getattr	Obtains file attributes. This command is used to retrieve the attributes of a specified file system object. The object is identified by the file handle returned by the server in the response of a LOOKUP, CREATE, MKDIR, SYMLINK, MKNOD, or READDIRPLUS procedure (or from the MOUNT service described elsewhere).
setattr	Changes file attributes. This command is used to change one or more attributes of a file system object on the server. The SETATTR operation may not be atomic. A failed SETATTR operation may partially change the attributes of a file. Changing the file size by using the SETATTR operation indirectly changes mtime.
access	Checks access permissions. This command is used to check whether a user (identified by credentials in the request) has the permissions to access a file system object. The client encodes the set of permissions to check as a bitmask. The server checks the permissions encoded in the bitmask. Normally, the server returns the NFS3_OK status and a bitmask that contains the permissions granted to the client.
lookup	Searches for a file name. This command searches directories for a specified file name and returns the file handle of the corresponding file system object.
read	Reads data from a file. This command reads data from a file based on the file handle.
write	Writes data to a file. This command writes data to a file based on the file handle.
create	Creates a file. This command is used to create a file. NFS v3 introduces the MKNOD procedure to handle the creation of device files and FIFO files.
mkdir	Creates a directory. This command is used to create a subdirectory. In many server implementations, an MKDIR operation cannot be performed on file names that contain only periods (`.` or `..`). In this case, the server returns `NFS3ERR_EXIST`.
remove	Deletes a file. This command is used to remove an entry from a directory. If the entry in the directory is the last reference to the corresponding file system object, the object may be destroyed. Typically, the REMOVE operation is used to delete non-directory objects, whereas the RMDIR operation is used to delete directories.
rename	Renames a file or directory. The RENAME operation moves a file or directory from its original location to a new location and renames the file or directory. The source and destination directories must be on the same file system and server to ensure the consistency and integrity of the operation.
readdir	Reads a directory. The READDIR operation retrieves a variable number of entries from a directory in sequence, and returns the file name and file identifier of each entry. Meanwhile, the READDIR operation provides information to allow the client to obtain additional directory entries in subsequent READDIR requests. For example, the `ls -l` command calls this operation to traverse a directory.
readirplus	Reads directory extensions. The READDIRPLUS operation retrieves a variable number of entries from a directory and returns the complete information about each entry. Meanwhile, the READDIRPLUS operation provides information to allow the client to obtain additional directory entries in subsequent READDIRPLUS requests. READDIRPLUS differs from READDIR only in the amount of information returned. The READDIR operation returns only the file name and file identifier (fileid) for each entry. The READDIRPLUS operation returns the file name, file identifier (fileid), attributes (including the file identifier), and file handle for each entry. For example, the `ls -l` command calls this operation to traverse a directory.

Exception scenarios

Scenario 1: The avg exe metric is high, the avg RTT metric is also high, whereas the avg queue metric is low

The bottleneck does not lie in the NFS client. If the NAS server responds slowly to requests or the network latency is high, submit a ticket.

Scenario 2: The avg exe metric is high, the avg RTT metric is low, and the avg queue metric is high

The bottleneck lies in the NFS client. Requests are queued on the NFS client. We recommend that you increase the throughput of a single ECS instance to access NAS. For more information, see How do I increase the throughput of accessing NAS on Linux?