If the No space left on device
error message is returned when you create a file or application on a Linux Elastic Compute Service (ECS) instance, the disk space on the instance is insufficient. This topic describes the causes of the issue of insufficient disk space and how to resolve the issue.
Possible causes
In most cases, the issue of insufficient disk space occurs due to the following causes:
The space usage of disk partitions reaches 100%.
The inode usage of disk partitions reaches 100%.
Leftover files (zombies) exist.
NoteFile handles are still occupied by deleted files. In this case, disk space that is consumed by the deleted files cannot be released.
Mount points are overwritten.
NoteFor example, the file system of a disk partition that manages a large number of files is mounted on a directory. The directory is considered a mount point. If the file system of another disk partition is mounted on the mount point, the mount point is overwritten by the new file system. However, applications in your system may still read data from and write data to the original file system. In this case, an error message that indicates the issue of insufficient disk space may be returned. If you run the
df
ordu
command to check the disk space usage of files and directories, no information is returned. This is because thedf
ordu
command returns information about the space usage of the disk partition that corresponds to the current mount point.The upper limit on inotify watches is reached.
The inotify API provides a mechanism for monitoring file system events in Linux. You can use inotify to monitor file changes in file systems in real time. This error is not caused by insufficient disk space. This topic describes the error to help you troubleshoot issues.
Troubleshooting methods and solutions
To resolve the issue, perform the following operations based on the cause of the issue:
1. The space usage of disk partitions reaches 100%
To resolve this issue, you can delete files or directories that consume a large amount of disk space, resize existing disks, or create more disks.
Query the current usage of disk space and find files that consume a large amount of disk space.
Connect to the ECS instance.
For more information, see Connect to a Linux instance by using a password or key.
Run the following command to query the usage of disk space:
df -h
The sample command output shown in the following figure is displayed. In this example, the space usage of the /dev/xvda1 partition is 15%.
Run the following commands to go to the root directory and identify which directory consumes the largest amount of disk space:
cd / du -sh *
The sample command output shown in the following figure is displayed. The command output indicates that the
/usr
directory consumes the largest amount of disk space. Then, you need to further identify which file or directory consumes the largest amount of disk space in the/usr
directory. Perform operations based on the actual scenario.Run the following commands in sequence to identify which directory consumes the largest amount of disk space in a hierarchical manner.
For example, the preceding command output indicates that the
/usr
directory consumes the largest amount of disk space. Then, you need to further identify which file or directory consumes the largest amount of disk space in the/usr
directory.cd /usr du -sh *
The sample command output shown in the following figure is displayed. The command output indicates that the
local
directory consumes the largest amount of disk space in the /usr directory. Then, you need to further identify which file or directory consumes the largest amount of disk space in thelocal
directory. Repeat the operation until you find the ultimate file or directory that consumes the largest amount of disk space.
Perform the following operations based on the business scenario:
Delete files or directories that consume a large amount of disk space and are no longer needed based on your business requirements.
Resize existing disks or create more disks based on your business requirements if your business does not allow you to delete files from disks or disk space cannot be released by deleting files. For more information, see Overview of disk resizing, Create a disk, and Attach a data disk.
2. The inode usage of disk partitions reaches 100%
Each file or directory in a file system is identified by a unique inode. A specific number of inodes are pre-allocated to a disk partition for files and directories when the disk partition is formatted. However, if the file system of the disk partition stores a large number of small files or directories, inode resources may become insufficient. If all inodes of a disk partition are allocated to files or directories, no new files or directories can be created on the disk partition regardless of whether the disk partition has free space. This causes the issue that the inode usage reaches 100%. To resolve the issue, you can delete unnecessary files or directories to release inodes or increase the number of inodes.
Query the inode usage.
Connect to the ECS instance.
For more information, see Connect to a Linux instance by using a password or key.
Run the following command to query the inode usage:
df -i
If the inode usage reaches or is about to reach 100%, use one of the following methods to resolve the issue.
Delete files or directories that use a large number of inodes
Increase the number of inodes
3. Leftover files (zombies) exist
If no exceptions are detected for the space usage and inode usage of disk partitions, the issue of insufficient disk space on the instance may be caused by a large number of leftover files. If specific files are deleted and displayed in the deleted state but are still used by processes, the disk space that is occupied by the files cannot be released or queried by running the df
or du
command. An excessively large number of leftover files occupy a large amount of disk space. Perform the following operations to query and delete leftover files:
Check whether leftover files exist.
Connect to the ECS instance.
For more information, see Connect to a Linux instance by using a password or key.
If lsof is not installed on the operating system, run one of the following commands to install lsof.
Alibaba Cloud Linux and CentOS:
yum install -y lsof
Debian and Ubuntu:
apt-get install -y lsof
Run the following command to query the disk space usage of leftover files:
lsof |grep delete | sort -k7 -rn | more
The sample command output shown in the following figure is displayed. The sizes of files that are in the deleted state are displayed in the seventh column. Check whether the sum of the sizes is close to the unexpected usage of disk space. If the sum is close to the unexpected usage, leftover files consume the disk space.
If leftover files exist, use one of the following methods to release the file handles and clear the leftover files to release the disk space:
Restart the instance
After the instance restarts, the system terminates running processes and releases the handles of deleted files.
ImportantInstance restarts may interrupt services. We recommend that you restart instances during an appropriate period of time.
Run the kill command
After you run the
lsof
command, the process IDs (PIDs) of running processes that correspond to the leftover files are usually displayed in the second column. You can specify a PID in thekill
command to terminate the corresponding process.Run the following command to list the PIDs of processes:
lsof |grep delete
Run the following command to terminate a process that corresponds to a leftover file based on your business requirements:
kill <PID>
ImportantWhen processes are terminated, services that run on instances may be affected. Proceed with caution.
4. Mount points are overwritten
If the issue persists after you troubleshoot the issue based on the preceding operations, check whether the issue is caused by overwritten mount points. Use the following method to check whether mount points are overwritten.
The following figure shows that the space usage of the /dev/vda1 system disk whose size is 30 GB reached 95%. After you run the du
command, you can see that the /home
directory consumes 24 GB of the disk space.
After you mount /dev/vdb1 on the /home
directory, the space usage of the /dev/vda1 system disk is still 95%. However, in the root partition of the system disk, only the /usr
directory consumes more than 1 GB of disk space and the /home
directory consumes only 20 KB instead of 24 GB of disk space based on the previous check. You cannot identify which directory consumes the largest amount of disk space. In this case, the issue may be caused by overwritten mount points.
To resolve this issue, unmount the disk partition and check the space usage of the original mount point.
The unmount operation may interrupt your application services. We recommend that you unmount partitions during an appropriate period of time.
5. The upper limit on inotify watches is reached
If an error message similar to tail: cannot watch '...': No space left on device
is returned when you run commands such as tail -f
, the upper limit on inotify watches is reached. To resolve the issue, you can increase the upper limit on inotify watches.
Run the following command to view the upper limit on inotify watches:
cat /proc/sys/fs/inotify/max_user_watches
Run the following command to change the upper limit on inotify watches:
sudo sysctl fs.inotify.max_user_watches=<New upper limit>
Replace
<New upper limit>
with the value of the upper limit that you want to specify for inotify watches.NoteIf you increase the upper limit on inotify watches, inotify watches may occupy a larger amount of system memory. When you change the upper limit on inotify watches, evaluate the memory and performance of the system and possible impact. You can run the
man 7 inotify
command to learn more about inotify watches and the related settings.
Optimize storage to prevent frequent full disk issues
If your disk space is frequently exhausted or the demands for data storage space are rapidly growing, you can optimize storage based on your business requirements to prevent frequent full disk issues.
If your disk space stores a large number of files, such as images and videos, and no high-concurrency read or write operations are performed on the disks, you can use Object Storage Service (OSS). OSS is a large-capacity, secure, cost-effective, and highly reliable cloud storage service that can automatically expand storage space based on the volume of data. You can use ossfs to mount OSS buckets to ECS instances. Applications can manage objects in OSS buckets in the same manner as on-premises files without the need to modify code. For information about how to use ossfs to mount an OSS bucket to a local directory on a Linux system, see ossfs.
If your business requires high-concurrency read and write operations and data sharing, you can use File Storage NAS to store files. NAS provides simple and scalable file systems that allow high-performance and high-concurrency shared storage, and can be used together with ECS. NAS can automatically expand the storage space as the data volume increases without the need for manual operations. For more information, see Mount a file system on a Linux ECS instance.
If you store a large number of log files on disks, you can transfer the log files to Simple Log Service. This facilitates log query and reduces disk usage. For more information, see Getting Started.