Object Storage Service:Synchronize local files to OSS

Usage notes

• To synchronize local files to OSS, you must have the oss:PutObject, oss:ListObjects, and oss:DeleteObject permissions. For more information, see Attach a custom policy to a RAM user.

• Binary name

  For ossutil 1.6.16 and later, you can directly use ossutil as the binary name in the command line. You do not need to update the binary name based on the operating system. For ossutil earlier than 1.6.16, you need to update the binary name based on the operating system. For more information, see ossutil command reference.

• Number of local files to synchronize

  When you run the sync command without including the --delete option, no limit is imposed on the number of local files that you can synchronize at a time. If you include the --delete option when you run the sync command, you can synchronize up to 1 million local files at a time. If the number of local files that you want to synchronize exceeds 1 million, the over max sync numbers 1000000. error message is returned.

• Differences between sync and cp

  • The sync command recursively synchronizes all objects and subdirectories in a specific directory. The cp command recursively synchronizes files only if the -r option is specified.

  • If you run the sync command to synchronize data from a local source to an OSS bucket, you can add the --delete option to the command to delete all files that do not exist in the local source but exist in the bucket. This way, only the synchronized files are retained in the bucket. The cp command does not support the --delete option.

  • The sync command does not support the --version-id option. You cannot use the sync command to synchronize previous versions of objects in buckets for which versioning is enabled. The cp command supports the --version-id option.

  Except for the preceding differences, you can use the sync and cp commands in a similar manner. For more information about how to run the cp command, see Upload objects.

Command syntax

ossutil sync file_url cloud_url
[-f --force]
[-u --update]
[--delete]
[--enable-symlink-dir]
[--disable-all-symlink]
[--disable-ignore-error]
[--only-current-dir]
[--output-dir <value>]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--encoding-type <value>]
[--snapshot-path <value>]
[--include <value>]
[--exclude <value>]
[--meta <value>]
[--acl <value>]
[--maxupspeed <value>]
[--disable-crc64]
[--payer <value>]
[-j, --job <value>]
[--parallel <value>]
[--retry-times <value>]
[--tagging <value>]

The following table describes the parameters and options in the syntax.

Examples

Two files named d.txt and e.png are stored in the localfolder directory of the local root directory. The destfolder directory of a bucket named examplebucket contains two objects named a.txt and b.txt and a subdirectory named C. The following structure shows the files and directories on your computer and OSS before synchronization:

Local root directory                  examplebucket
    └── localfolder         └── destfolder/
            ├── d.txt               ├── a.txt
            ├── e.png               ├── b.txt
                                    └── C/

The following examples show how to run the sync command to synchronize local files to OSS in different scenarios.

• Synchronize the localfolder directory from your computer to the examplebucket bucket

  ossutil sync localfolder/  oss://examplebucket/destfolder/

  After you run the preceding command, the d.txt and e.png objects are added to the destfolder directory of the examplebucket bucket. The following structure shows the files and directories on your computer and OSS after synchronization:

  Local root directory                  examplebucket
      └── localfolder         └── destfolder/
              ├── d.txt               ├── a.txt
              ├── e.png               ├── b.txt
                                      ├── d.txt 
                                      ├── e.png 
                                      └── C/
                          

• Synchronize the .txt object from the localfolder directory to the examplebucket bucket

  ossutil sync localfolder/  oss://examplebucket/destfolder/ --include "*.txt"

  After you run the preceding command, the d.txt object is added to the destfolder directory of the examplebucket bucket. The following structure shows the files and directories on your computer and OSS after synchronization:

  Local root directory                  examplebucket
      └── localfolder         └── destfolder/
              ├── d.txt               ├── a.txt
              ├── e.png               ├── b.txt
                                      ├── d.txt 
                                      └── C/
                          

• Synchronize non-TXT objects from the localfolder directory to the examplebucket bucket

  ossutil sync localfolder/  oss://examplebucket/destfolder/ --exclude "*.txt"

  After you run the preceding command, the e.png object is added to the destfolder directory of the examplebucket bucket. The following structure shows the files and directories on your computer and OSS after synchronization:

  Local root directory                  examplebucket
      └── localfolder         └── destfolder/
              ├── d.txt               ├── a.txt
              ├── e.png               ├── b.txt
                                      ├── e.png 
                                      └── C/
                          

• Synchronize the localfolder directory from your computer to the examplebucket bucket and retain only the synchronized objects in the destfolder directory of the examplebucket bucket

  Add the --delete option in the command to delete all objects that do not exist in the localfolder directory from the destfolder directory. This way, only the synchronized objects are retained in the destfolder directory.

  ossutil sync localfolder/  oss://examplebucket/destfolder/ --delete

  After you run the preceding command, the localfolder directory is synchronized from your computer to the examplebucket bucket. The a.txt and b.txt objects and the C subdirectory in the destfolder directory of the examplebucket bucket are deleted. Only the synchronized d.txt and e.png objects are retained in the destfolder directory of the examplebucket bucket. The following structure shows the files and directories on your computer and OSS after synchronization:

  Local root directory                  examplebucket
      └── localfolder         └── destfolder/
              ├── d.txt                ├── d.txt
              ├── e.png                ├── e.png

• Synchronize the localfolder directory from your computer to the examplebucket bucket without a prompt for confirmation

  By default, when you synchronize a local directory to an OSS bucket, if the OSS bucket contains objects that have the same name as the files in the local directory, OSS prompts you to confirm whether you want to overwrite the existing objects in the bucket. Example:

  ossutil sync localfolder/ oss://examplebucket/destfolder/
  cp: overwrite "oss://examplebucket/destfolder/d.txt"(y or N)?

  If you confirm that the objects in the examplebucket bucket can be overwritten, you can add the -f,--force option to the command to forcibly overwrite the existing objects in the bucket. Example:

  ossutil sync localfolder/ oss://examplebucket/destfolder/ -f

  After you run the preceding command, the d.txt and e.png objects are added to the destfolder directory of the examplebucket bucket. The following structure shows the files and directories on your computer and OSS after synchronization:

  Local root directory                  examplebucket
      └── localfolder         └── destfolder/
              ├── d.txt               ├── a.txt
              ├── e.png               ├── b.txt
                                      ├── d.txt 
                                      ├── e.png
                                      └── C/
                      

• If the preceding commands are run successfully, the following command output is returned and indicates the number of synchronized objects, the sizes of the synchronized objects, and the amount of time required to synchronize the objects:

  Succeed: Total num: 2, size: 750,081. OK num: 2(upload 2 files).
  
  average speed 1641000(byte/s)

Common options

If you use ossutil to switch to a bucket that is located in another region, add the -e option to specify the endpoint of the region in which the bucket is located. If you use ossutil to switch to a bucket that belongs to another Alibaba Cloud account, add the -i option to specify the AccessKey ID of the specified account, and add the -k option to specify the AccessKey secret of the specified account.

ossutil sync srcfolder/ oss://examplebucket/testfolder/ -e oss-cn-shanghai.aliyuncs.com -i LTAI4Fw2NbDUCV8zYUzA****  -k 67DLVBkH7EamOjy2W5RVAHUY9H****

For more information about common options, see Common options.