File Storage NAS:Migrate data to a NAS file system

Benefits

• Supports multiple operating system platforms.

• Provides a graphical user interface (GUI).

Procedure

1. Install an SFTP client on your local operating system.

   Various SFTP clients are available. This topic uses FileZilla as an example. Download and install an SFTP client compatible with your local operating system.

2. Establish a connection between the SFTP client and the transit ECS instance.

   1. Open the FileZilla client, configure the following parameters, and click Quickconnect to establish a connection.

      

      Parameter	Description
      Host	The public IP address of the transit node ECS instance. Example: 192.0.2.1
      Username	The username of the transit node ECS instance. The user must have read and write permissions on the NAS file system directory. Example: root
      Password	The login password of the transit node ECS instance. Example: the login password of the root user. To reset a forgotten login password, see Reset the logon password of an instance.
      Port	The SFTP port number. Default: 22

      Note: The default username of a Linux ECS instance is root or ecs-user. The default username of a Windows ECS instance is administrator.

   2. In the right pane, enter the mount path of the NAS file system (such as /mnt) in the Remote site field. Press Enter to view the files in the NAS file system.

      

3. Upload the data.

   Drag files or folders from the left pane (local) to the right pane (server) to upload the data.

Benefits

• Preserves file metadata, such as owner and permission information.

• Supports incremental data synchronization.

• Enables scheduled backups using crontab on Linux or macOS.

Step 1: Install rsync

Linux:

Note: For other Linux distributions, use the corresponding package manager.

macOS:

Download and install the homebrew package manager. Then run:

brew install rsync

Windows:

Download and install the Cygwin environment. During installation, search for and install the rsync tool, or manually download, compile, and install rsync.

Note: Open the SSH port (TCP 22) in the VPC security group.

Step 2: Upload data

Run the following command to upload a local folder to Alibaba Cloud NAS using incremental synchronization:

rsync -avP DirToSync/ root@192.0.2.0:/mnt/DirToSync/

Important: The source path must end with a forward slash (/). Otherwise, the data paths will be incorrect after synchronization.

Concurrent copy:

To concurrently copy data to a NAS file system, run:

threads=<number_of_threads>
src=<source_path/>
dest=<destination_path/>
rsync -av -f"+ */" -f"- *" $src $dest && (cd $src && find . -type f | xargs -n1 -P$threads -I% rsync -av % $dest/%)

Example: 10 threads, source path /abc, destination path /mnt1:

threads=10
src=/abc/
dest=/mnt1/
rsync -av -f"+ */" -f"- *" $src $dest && (cd $src && find . -type f | xargs -n1 -P$threads -I% rsync -av % $dest/%)

Step 3 (Optional): Configure scheduled backups

On a local Linux or macOS system, use rsync and crontab to configure scheduled upload and backup tasks.

Linux:

1. Create a passwordless SSH connection from your local system to the ECS instance. For more information, see Connect to a Linux instance using OpenSSH or Xshell.

   Confirm the connection is successful:

   ssh -i ~/.ssh/ecs.pem root@1.2.3.4

   Note: ~/.ssh/ecs.pem is the path to the key file on your local system.

2. Configure crontab.

   Run crontab -e to open the editor and add:

   0 23 * * * rsync -av -e "ssh -i ~/.ssh/ecs.pem" ~/Documents/ root@192.0.2.0:/mnt/Documents/

   This configuration automatically uploads and backs up data from the Documents folder to Alibaba Cloud NAS at 23:00 daily. Modify the parameters as needed.

macOS:

1. Configure access permissions for the /usr/sbin/cron directory.

   Go to System Settings > Security & Privacy > Privacy > Full Disk Access. Click the lock icon, click +, select the Macintosh HD folder, press Cmd+Shift+. to display hidden folders, and select /usr/sbin/cron.

2. Create a passwordless SSH connection (same as Linux above).

3. Configure crontab (same as Linux above).

Note: If rsync performs poorly during data migration, use the fpsync tool for multi-threaded migration. See Appendix: Migrate data using fpsync.

Procedure

1. Log on to the Alibaba Cloud ECS instance used for data migration.

2. Migrate the data.

   Run the following command to migrate data from the source file system (drive Z) to the destination file system (drive Y):

   robocopy Z:\ Y:\ /e /w:5 /z /mt:32

   Note: Only data in the specified folder is migrated. The folder itself is not included.

   Parameter	Description
   /mt	Sets the number of concurrent threads. Default: 8. Valid values: 1-128. This example uses 32 threads.
   /w	Sets the interval in seconds between retries
   /z	Enables resumable transfer
   /e	Copies all subdirectories, including empty ones
   /copyall	Copies all file information (data, properties, timestamps, ACL, owner, auditing)

   Note: To accelerate migration of large data volumes (such as hundreds of millions of small files exceeding 10 TB), install the latest Python program on a Windows ECS instance. For more information, see 如何加速迁移数据至NAS SMB协议文件系统.

3. Verify the migration result.

   Run the following command to verify consistency between the source and destination file systems:

   ROBOCOPY Z:\ Y:\ /e /l /ns /njs /njh /ndl /fp /log:reconcile.txt

   Parameter	Description
   /e	Lists only folders, including empty ones
   /l	Records differences without modifying or copying files
   /fp	Includes full file paths in the log (only necessary if /ndl is omitted)
   /ns	Excludes file sizes from the log
   /ndl	Excludes folders from the log
   /njs	Excludes the job summary
   /njh	Excludes the job header
   /log:reconcile.txt	Writes the migration result to reconcile.txt. Overwrites existing logs.

Benefits

• Supports multiple operating system platforms

• Provides a graphical user interface

Procedure

Configure the IIS FTP service on the ECS instance and configure the FTP client on your local system. For more information, see Set up the Windows IIS web service.

Step 1: Download and install fpsync

wget -N https://github.com/martymac/fpart/archive/fpart-1.1.0.tar.gz -P /tmp
tar -C /tmp/ -xvf /tmp/fpart-1.1.0.tar.gz
cd /tmp/fpart-fpart-1.1.0
sudo yum install -y automake libtool
autoreconf -i
./configure
make
sudo make install
sudo yum install parallel -y
printf "will cite" | parallel --bibtex
sudo yum install -y rsync

Step 2: Copy the entire folder

fpsync -n 10 -f 10000 /data/src/ /data/dst/

Note: For more information about the fpsync tool, see fpsync tool.