High-performance file storage services
Cloud Parallel File Storage (CPFS) is a fully-managed parallel file system provided by Alibaba Cloud. You can use CPFS in scenarios where high-performance computing is required.
CPFS provides a unified namespace and supports concurrent access of hundreds of machines. CPFS also provides an I/O throughput of tens of GB/s and millions of IOPS to ensure a sub-millisecond latency.
CPFS file systems can be used in I/O-intensive business that requires high throughput and high IOPS, and involves a large amount of data, such as AI training, autonomous driving, genomics computing, film rendering, oil exploration, weather forecast, and emulation for electronic design automation (EDA) software.
Specifications
400 MB/s/TiB Baseline (Invitational Preview of CPFS for Lingjun)
-
![]()
IOPS: up to 30 million
-
![]()
I/O throughput: up to 2 TB/s
-
![]()
Average latency for reading 4 KiB files from a
single-socket server: 0.25 ms -
![]()
Average latency for writing 4 KiB files to a
single-socket server: 0.6 ms
IOPS: up to 30 million
200 MB/s/TiB Baseline
-
![]()
IOPS: up to 2.8 million
-
![]()
I/O throughput: up to 100,000 Mbit/s
-
![]()
Average latency for reading 4 KiB files from a
single-socket server: 0.4 ms -
![]()
Average latency for writing 4 KiB files to a
single-socket server: 0.6 ms
Supports data migration between OSS and CPFS
100 MB/s/TiB Baseline
-
![]()
IOPS: up to 2.8 million
-
![]()
I/O throughput: up to 100,000 Mbit/s
-
![]()
Average latency for reading 4 KiB files from a
single-socket server: 0.6 ms -
![]()
Average latency for writing 4 KiB files to a
single-socket server: 0.8 ms
Supports data migration between OSS and CPFS
Benefits
High Performance
Provides an I/O throughput of 2 TB/s and 30 million IOPS based on a distributed concurrent architecture.
Scalable
Supports auto scaling and linearly improved performance
Integration with OSS Data Lake
Supports the data communication between OSS and reduces data storage costs.
Easy to Use
Allows you to create a high-performance file system within a few minutes.
Features
High-performance Computing File Storage
CPFS supports the standard Portable Operating System Interface (POSIX), MPI-IO, and Network File System (NFS) protocols. You can effectively execute your high-performance computing programs without calling API operations or optimizing performance.
Compatible with the POSIX, MPI-IO, and NFS protocols
CPFS is compatible with the POSIX, MPI-IO, and NFS protocols. You can directly use an ECS instance or the Container Storage Interface (CSI) interface plug-in to read data from or write data to CPFS without modifications on your high-performance programs.
High throughput and low latency
CPFS uses a new generation of parallel architecture to enable the concurrent I/O throughput of clients and storage nodes. CPFS also provides linearly improved I/O performance by aggregating clusters and sub-millisecond latency based on all-flash SSD storage.
Improved performance in concurrent access
CPFS uses a distributed parallel architecture to store metadata and data, which supports millions of IPOS for a large number of small files. CPFS can be concurrently accessed by thousands of servers and is suitable for the training of multiple multi-GPU servers.
Deep Integration with Alibaba Cloud Computing Services
CPFS can be mounted to computing platforms, such as Elastic High Performance Computing(E-HPC), Container Service for Kubernetes (ACK), and WUYING Workspace. You can build a high-performance computing and analytic platform in the cloud within a few minutes.
Deep integration with E-HPC
You can quickly create a CPFS file system and build a complete high-performance computing platform by using the E-HPC console.
Compatible with ACK
CPFS can be mounted to ACK. This way, ACK can use the CSI plug-in to use the high I/O performance of CPFS and enable the large-scale auto scaling of computing power.
Compatible with WUYING Workspace
CPFS can be directly mounted to WUYING Workspace over NFS. This allows you to share data between development and debugging terminals and high-performance computing clusters.
Common Scenarios
-
Description
Alibaba Cloud combines multiple storage services, such as CPFS, NAS, OSS, and ESSD and computing cluster services such as Elastic GPU Service, SCC, and ECS Bare Metal Instance by containerized deployment. This ensures the scalability of resources and high performance of I/O throughput and IOPS.
-
Requirements and Pain Points
The data streams of AI models are complex. Bottlenecks exist in the I/O performance of the storage system. High-performance computing and storage are required in AI training and inference. The efficient and cost-effective data lifecycle management is required to manage the rapidly growing data volume. -
Solutions and Benefits
CPFS supports the concurrent access of large amounts of data in AI training. This helps improve AI training performance. CPFS integrates multiple computing cluster services to provide high-performance I/O throughput and IOPS. CPFS enables the communication between data centers and services on Alibaba Cloud and manages the data flows and lifecycle to provide high-performance and cost-effective computing power. -
Effects
The high I/O performance of CPFS accelerates the efficiency of reading small files during AI training. This enhances the iteration efficiency of your business model and meets your growing needs for high capacity and performance. CPFS uses OSS to implement tiered storage of cold and hot data. This is suitable for the long-term storage of large amounts of data in a cost-effective manner.
-
Description
In the model training of autonomous driving, tens of TB data is generated for each test vehicle per day. The data must be quickly analyzed and stored in a cost-effective manner.
-
Requirements and Pain Points
Focus on business innovation and reduce IT spending. Process large amounts of data in an efficient manner. -
Solutions and Benefits
Data is uploaded to OSS over an Express Connect circuit. Data is imported from OSS to CPFS and used to train models in the cloud based on the processing capabilities of GPUs and CPFS. Then, the data is archived to OSS for long-term storage. -
Effects
The high performance and scalability of CPFS can meet your growing needs for high capacity and performance. CPFS uses OSS to implement tiered storage of cold and hot data. This is suitable for the long-term storage of large amounts of data in a cost-effective manner.
Related Services
-
Description
CPFS provides an I/O throughput of dozens of Gbit/s and millions of IOPS. You can scale up storage space online without business interruptions. Data is uploaded to the cloud over an Express Connect circuit to ensure data security.
-
Requirements and Pain Points
Accessing hot data at rates of dozens of Gbit/s requires high I/O throughput performance. The shared storage system need to be accessed without modifying closed source commercial rendering applications. The requirements on computing, data storage, and performance vary based on projects. The costs of compute-intensive business are high and must be controlled in each stage. -
Solutions and Benefits
CPFS provides millions of IOPS and an I/O throughput of dozens of Gbit/s that can be used to accelerate the frontend caching. The high scalability of CPFS allows you to scale up disk space and upgrade caching based on your business requirements without affecting your applications. CPFS is cost-effective. You can configure the storage space and caching performance based on you business requirements without the need of replicas or copies. CPFS uses your data to provide best performance and helps you reduce the TCO. -
Effects
The high-performance throughput and IOPS of CPFS greatly improve the execution efficiency of rendering tasks, simplify daily O&M, and reduce overall costs.
-
Description
The data flows between CPFS and OSS, and between CPFS and NAS are layered. CPFS combines this benefit in storage and the computing power of ECS Bare Metal Instance, Elastic GPU Service, and ACK to provide solutions for scenarios in which high-performance computing and storage are required, such as AI, supercomputing, simulation, and bioformatics.
-
Requirements and Pain Points
Bottlenecks exist in the performance of I/O throughput due to large amounts of data. Traditional file storage systems do not support the elastic scaling of performance and storage space. Heavy O&M workloads are required in data management and performance maintenance. -
Solutions and Benefits
The high-performance storage of CPFS improves the usage of computing resources. CPFS can help you upgrade specifications and scale up space storage in a more cost-effective manner. CPFS connects to other cloud storage services to separately store cold and hot data. This helps you reduce costs. -
Effects
CPFS provides high-performance I/O throughput and IOPS to help you effectively cope with the heavy read and write workloads of thousands of nodes, improve computing efficiency, and significantly reduce overall business costs.
