This topic describes how to migrate data between File Storage NAS (NAS) file systems based on the tutorial for migrating data between local file systems.
Overview
Alibaba Cloud Data Online Migration provides tunnels for data migration between different storage services. This topic describes how to use Data Online Migration to migrate data between NAS file systems.
Data Online Migration does not guarantee the consistency between source data and migrated data during data migration. Make sure that you verify the migrated data at the destination data address after a migration task is complete. If you delete the data at the source data address before you verify the migrated data at the destination data address, you are liable for the losses and consequences caused by any data loss.
Migration flowchart
Before you begin
Create ECS instances on which agents are to be deployed
Each Elastic Compute Service (ECS) instance that you create must have at least 4 CPU cores, 8 GB of memory, and 20 GB of free disk space.
An ECS instance that supports baseline bandwidth and burstable bandwidth may not provide stable bandwidth in actual cases. To be conservative, you can calculate the average bandwidth that such an ECS instance can provide based on the following formula: Average bandwidth = 1.5 × Baseline bandwidth.
The network bandwidth of an ECS instance is the sum of the downstream bandwidth and upstream bandwidth. An agent that runs on an ECS instance uses both the downstream bandwidth and upstream bandwidth. Therefore, the migration speed on a single ECS instance equals to half the average bandwidth.
If the migration speed on a single ECS instance does not reach the migration speed that you expect, you can use multiple ECS instances. The number of ECS instances that are required for a tunnel is calculated based on the following formula: Number of ECS instances required for a tunnel = Expected migration speed/Migration speed on a single ECS instance. When you migrate data between NAS file systems, the source and destination data addresses must be separately associated with a tunnel.
When you create ECS instances, make sure that the source NAS file system can be mounted to the source ECS instances and the destination NAS file system can be mounted to the destination ECS instances.
Manually mount NAS file systems
If multiple ECS instances are created for a tunnel associated with a NAS file system, the NAS file system must be mounted to the same local directory on the ECS instances. For example, if two ECS instances are created for the source tunnel, and the source NAS file system is mounted to /mnt/src/ on the first ECS instance, the NAS file system must also be mounted to /mnt/src/ on the second ECS instance.
For more information about the commands and parameters for mounting a NAS file system, see the usage notes in the NAS console.
Make sure that you mount a NAS file system by using the commands and parameters to the specified directory on an ECS instance and the mounted NAS file system is available during data migration. Otherwise, data may be written to a local disk instead of the NAS file system. If you have any questions about mounting a NAS file system, contact the technical support of NAS.
Migration tutorial
You can migrate data between NAS file systems in the same way as you migrate data between local file systems. For more information about the tutorial, see Migrate data between local file systems.
The migration speed is affected by multiple factors, including the network condition, the file system performance, and the directory structure. For example, the migration speed is affected by the length and latency of the network link, the read and write speed of NAS, whether files are stored in a single directory, the number and levels of directories in the source NAS file system, and the average file size. The number of ECS instances that you calculate for a tunnel is for reference only. The actual migration speed may fail to meet your expectation.
Examples
Example 1: Migrate data between NAS file systems that reside in the same region and VPC
In this example, the source NAS file system resides in the China (Beijing) region, the mount target of the source NAS file system is created in the vpc-nas virtual private cloud (VPC), and the vsw-nas vSwitch is used. The source NAS file system stores 200 TB of data to be migrated. The average file size is about 500 KB. The destination NAS file system also resides in the China (Beijing) region, the mount target of the destination NAS file system is created in the vpc-nas VPC, and the vsw-nas vSwitch is used. The expected migration speed is 2 Gbit/s.
Perform the following steps:
Determine the configurations of the ECS instances on which the agents are to be deployed. For example, you can create ECS instances of the ecs.c7.xlarge type. Each ECS instance of this type has 4 CPU cores, 8 GB of memory, 3 Gbit/s of baseline bandwidth, and 12.5 Gbit/s of burstable bandwidth.
The burstable bandwidth is unstable. To ensure sufficient bandwidth, you can calculate the average bandwidth based on the following formula: Average bandwidth = 1.5 × Baseline bandwidth = 4.5 Gbit/s.
The network bandwidth of an ECS instance is the sum of the downstream bandwidth and upstream bandwidth. An agent that runs on an ECS instance uses the downstream bandwidth to obtain data from the source NAS file system and uses the upstream bandwidth to push the obtained data to the server of Data Online Migration. Therefore, the migration speed on a single ECS instance is calculated based on the following formula: Migration speed on a single ECS instance = Average bandwidth/2 = 4.5 Gbit/s/2 = 2.25 Gbit/s.
Calculate the number of ECS instances that are required for a tunnel based on the following formula: Number of ECS instances required for a tunnel = Expected migration speed/Migration speed on a single ECS instance = 2 Gbit/s/2.25 Gbit/s = 0.9 ≈ 1. Therefore, you must create one ECS instance for the source NAS file system and another for the destination NAS file system.
In the ECS console, create one ECS instance of the ecs.c7.xlarge type for the source NAS file system and another of the same type for the destination NAS file system. Select the China (Beijing) region, the vpc-nas VPC, and the vsw-nas vSwitch when you create the ECS instances. Name the ECS instances ECS-1 and ECS-2.
Log on to ECS-1 as an administrator, create the /mnt/src/ directory, and then mount the source NAS file system to the directory. Log on to ECS-2 as an administrator, create the /mnt/dest/ directory, and then mount the destination NAS file system to the directory.
ImportantMake sure that the NAS file systems are mounted to the specified directories and the NAS file systems are available during data migration.
Migrate data in the Data Online Migration console in the China (Beijing) region. For more information, see Migrate data between local file systems.
Create a source tunnel. Create a source agent. Set the Network Type parameter to Leased Line/VPN (VPC), and associate the agent with the source tunnel. Deploy the agent on ECS-1.
Create a destination tunnel. Create a destination agent. Set the Network Type parameter to Leased Line/VPN (VPC), and associate the agent with the destination tunnel. Deploy the agent on ECS-2.
Create a data address for the source NAS file system. Associate the data address with the source tunnel, and select the source agent. Set the Directory to Be Migrated parameter to /mnt/src/.
Create a data address for the destination NAS file system. Associate the data address with the destination tunnel, and select the destination agent. Set the Directory to Be Migrated parameter to /mnt/dest/.
Create a migration task by using the source and destination data addresses.
Example 2: Migrate data between NAS file systems that reside in the same region but different VPCs
In this example, the source NAS file system resides in the China (Beijing) region, the mount target of the source NAS file system is created in the vpc-nas-1 VPC, and the vsw-nas-1 vSwitch is used. The source NAS file system stores 200 TB of data to be migrated. The average file size is about 500 KB. The destination NAS file system also resides in the China (Beijing) region, the mount target of the destination NAS file system is created in the vpc-nas-2 VPC, and the vsw-nas-2 vSwitch is used. The destination NAS file system is empty. The expected migration speed is 2 Gbit/s.
Perform the following steps:
Determine the configurations of the ECS instances on which the agents are to be deployed. For example, you can create ECS instances of the ecs.c7.xlarge type. Each ECS instance of this type has 4 CPU cores, 8 GB of memory, 3 Gbit/s of baseline bandwidth, and 12.5 Gbit/s of burstable bandwidth.
The burstable bandwidth is unstable. To ensure sufficient bandwidth, you can calculate the average bandwidth based on the following formula: Average bandwidth = 1.5 × Baseline bandwidth = 4.5 Gbit/s.
The network bandwidth of an ECS instance is the sum of the downstream bandwidth and upstream bandwidth. An agent that runs on an ECS instance uses the downstream bandwidth to obtain data from the source NAS file system and uses the upstream bandwidth to push the obtained data to the server of Data Online Migration. Therefore, the migration speed on a single ECS instance is calculated based on the following formula: Migration speed on a single ECS instance = Average bandwidth/2 = 4.5 Gbit/s/2 = 2.25 Gbit/s.
Calculate the number of ECS instances that are required for a tunnel based on the following formula: Number of ECS instances required for a tunnel = Expected migration speed/Migration speed on a single ECS instance = 2 Gbit/s/2.25 Gbit/s = 0.9 ≈ 1. Therefore, you must create one ECS instance for the source NAS file system and another for the destination NAS file system.
In the ECS console, create one ECS instance of the ecs.c7.xlarge type for the source NAS file system. Select the China (Beijing) region, the vpc-nas-1 VPC, and the vsw-nas-1 vSwitch when you create the ECS instance. Name the ECS instance ECS-1. Similarly, create one ECS instance of the ecs.c7.xlarge type for the destination NAS file system. Select the China (Beijing) region, the vpc-nas-2 VPC, and the vsw-nas-2 vSwitch when you create the ECS instance. Name the ECS instance ECS-2.
Log on to ECS-1 as an administrator, create the /mnt/src/ directory, and then mount the source NAS file system to the directory. Log on to ECS-2 as an administrator, create the /mnt/dest/ directory, and then mount the destination NAS file system to the directory.
ImportantMake sure that the NAS file systems are mounted to the specified directories and the NAS file systems are available during data migration.
Migrate data in the Data Online Migration console in the China (Beijing) region. For more information, see Migrate data between local file systems.
Create a source tunnel. Create a source agent. Set the Network Type parameter to Leased Line/VPN (VPC), and associate the agent with the source tunnel. Deploy the agent on ECS-1.
Create a destination tunnel. Create a destination agent. Set the Network Type parameter to Leased Line/VPN (VPC), and associate the agent with the destination tunnel. Deploy the agent on ECS-2.
Create a data address for the source NAS file system. Associate the data address with the source tunnel, and select the source agent. Set the Directory to Be Migrated parameter to /mnt/src/.
Create a data address for the destination NAS file system. Associate the data address with the destination tunnel, and select the destination agent. Set the Directory to Be Migrated parameter to /mnt/dest/.
Create a migration task by using the source and destination data addresses.
Example 3: Migrate data between NAS file systems that reside in different regions and different VPCs
In this example, the source NAS file system resides in the China (Beijing) region, the mount target of the source NAS file system is created in the vpc-nas-1 VPC, and the vsw-nas-1 vSwitch is used. The source NAS file system stores 200 TB of data to be migrated. The average file size is about 500 KB. The destination NAS file system resides in the China (Hangzhou) region, the mount target of the destination NAS file system is created in the vpc-nas-2 VPC, and the vsw-nas-2 vSwitch is used. The destination NAS file system is empty. The expected migration speed is 10 Gbit/s.
Perform the following steps:
Determine the configurations of the ECS instances on which the agents are to be deployed. For example, you can create ECS instances of the ecs.c7.xlarge type. Each ECS instance of this type has 4 CPU cores, 8 GB of memory, 3 Gbit/s of baseline bandwidth, and 12.5 Gbit/s of burstable bandwidth.
The burstable bandwidth is unstable. To ensure sufficient bandwidth, you can calculate the average bandwidth based on the following formula: Average bandwidth = 1.5 × Baseline bandwidth = 4.5 Gbit/s.
The network bandwidth of an ECS instance is the sum of the downstream bandwidth and upstream bandwidth. An agent that runs on an ECS instance uses the downstream bandwidth to obtain data from the source NAS file system and uses the upstream bandwidth to push the obtained data to the server of Data Online Migration. Therefore, the migration speed on a single ECS instance is calculated based on the following formula: Migration speed on a single ECS instance = Average bandwidth/2 = 4.5 Gbit/s/2 = 2.25 Gbit/s.
Calculate the number of ECS instances that are required for a tunnel based on the following formula: Number of ECS instances required for a tunnel = Expected migration speed/Migration speed on a single ECS instance = 10 Gbit/s/2.25 Gbit/s = 4.4 ≈ 5. Therefore, you need to create five ECS instances for the source NAS file system and another five ECS instances for the destination NAS file system.
In the ECS console, create five ECS instances of the ecs.c7.xlarge type for the source NAS file system. Select the China (Beijing) region, the vpc-nas-1 VPC, and the vsw-nas-1 vSwitch when you create the ECS instances. Name the ECS instances ECS-Src1, ECS-Src2, ECS-Src3, ECS-Src4, and ECS-Src5. Similarly, create five ECS instances of the ecs.c7.xlarge type for the destination NAS file system. Select the China (Beijing) region, the vpc-nas-2 VPC, and the vsw-nas-2 vSwitch when you create the ECS instances. Name the ECS instances ECS-Dst1, ECS-Dst2, ECS-Dst3, ECS-Dst4, and ECS-Dst5.
Separately log on to ECS-Src1, ECS-Src2, ECS-Src3, ECS-Src4, and ECS-Src5 as an administrator, create the /mnt/src/ directory, and then mount the source NAS file system to the directory. Separately log on to ECS-Dst1, ECS-Dst2, ECS-Dst3, ECS-Dst4, and ECS-Dst5 as an administrator, create the /mnt/dest/ directory, and then mount the destination NAS file system to the directory.
ImportantMake sure that the NAS file systems are mounted to the specified directories and the NAS file systems are available during data migration.
Migrate data between the source and destination NAS file systems. For more information, see Migrate data between local file systems. Select the China (Beijing) or China (Hangzhou) region in the Data Online Migration console. In this example, the China (Beijing) region is selected.
Create a source tunnel. Create five source agents. Set the Network Type parameter to Leased Line/VPN (VPC), and associate the agents with the source tunnel. Separately deploy the five agents on ECS-Src1, ECS-Src2, ECS-Src3, ECS-Src4, and ECS-Src5.
Create a destination tunnel. Create five destination agents. Set the Network Type parameter to Internet, and associate the agents with the destination tunnel. Separately deploy the five agents on ECS-Dst1, ECS-Dst2, ECS-Dst3, ECS-Dst4, and ECS-Dst5.
Create a data address for the source NAS file system. Associate the data address with the source tunnel, and select the source agents. Set the Directory to Be Migrated parameter to /mnt/src/.
Create a data address for the destination NAS file system. Associate the data address with the destination tunnel, and select the destination agents. Set the Directory to Be Migrated parameter to /mnt/dest/.
Create a migration task by using the source and destination data addresses.
Usage notes
You must start the agents as an administrator.
The agents associated with the source data address must have the permissions to read all files and subdirectories in the directories of the source NAS file system. Otherwise, the agents fail to read data from the NAS file system, and the data migration fails.
The agents associated with the destination data address must have the read and write permissions on all files and subdirectories in the directories of the destination NAS file system. Otherwise, the agents fail to read data from and write data to the NAS file system,and the data migration fails.
If a NAS file system has a large number of directories and each directory contains a small number of files, the files may be scanned at a low speed. For example, if the source NAS file system stores 1 million files that are distributed in 500,000 directories, the files are scanned at a lower speed than that if the files are distributed in 10,000 directories.
If more than 200,000 files are stored in the same directory, the migration speed may be low due to the concurrency limit of the operating system kernel of the ECS instances on which the agents are deployed and the concurrency limit of the NAS file systems.
If an excessively large number of files are stored in a directory or the directory is being modified, the NAS file system may give a slow response or not respond when you perform a list operation on the directory. In this case, the migration task may be interrupted. For more information, see the "Why does a file system give a slow response or does not respond when I run the ls command?" section of the FAQ about the performance of NAS file systems topic.