All Products
Search

This Product

    Object Storage Service:High-level commands

    Document Center

    Object Storage Service:High-level commands

    Last Updated:Aug 02, 2024

    This topic describes high-level commands in ossutil.

    Commands

    Command set

    Supported option

    All high-level commands

    • --encoding-type string: the encoding method of the object name or file name. Valid value: url.

    • --request-payer string: the payer of the request. If pay-by-requester is enabled, specify this option. Valid value: requester.

    High-level commands that support batch processing

    • --start-after/--end-with: specifies the range of object names that you want to query. The range is a left-open, right-closed interval.

    • filter: specifies filter rules for objects and files by name, path, size, time, and metadata (only for objects). For more information, see Filter options.

    • --limited-num: limits the number of results in the response.

    • --recursive/-r: specifies that the operation is recursively performed on all files or objects, and all subdirectories in the root directory.

    • --dirs/-d: matches files or objects only in the root directory and ignores subdirectories in the root directory.

      Note

      If the option is applied to objects, a delimiter is used to simulate a directory, and all objects with the specified name prefix are scanned. The amount of time used to scan objects is directly proportional to the number of the objects with the specified name prefix.

    • --force/-f: forces the operation without requiring confirmation.

    • --list-objects: lists objects by calling the ListObjects operation.

    High-level commands that support filtering by destination object

    • --update: performs the operation only if the last modified time of the source file or object is later than that of the destination object or file.

    • --size-only: performs the operation only if the source file or object has a different size from the destination object or file.

    • --checksum: specifies that the operation is performed only when one of the following conditions is met: a. the size of the source object is different from that of the destination object; b. the CRC-64 of the source object is different from that of the destination object, but the size of the source object is the same as that of the destination object. If the source or destination object does not have a CRC-64 value, the CRC-64 of the source object is different from that of the destination object.

    High-level commands that apply to individual objects

    --version-id string: the version ID of the object.

    Local options

    Local option

    Description

    --encoding-type string

    The encoding method used to encode the object or file name. Valid value: url.

    --request-payer string

    The payer of the request. If pay-by-requester is enabled, specify this option. Valid value: requester.

    --start-after string

    Matches objects whose names are alphabetically after the value of the --start-after option.

    --end-with string

    Matches objects whose names are alphabetically before or the same as the value of the --end-with option.

    --page-size int

    The maximum number of objects to return. Default value: 1000. Valid values: 1 to 1000.

    --include stringArray

    The inclusion rule for the path or object name.

    --include-from stringArray

    Reads inclusion rules from a file.

    --exclude stringArray

    The exclusion rule for the path or object name.

    --exclude-from stringArray

    Reads exclusion rules from a file.

    --filter stringArray

    The filter rule for paths or object names.

    --filter-from stringArray

    Reads filter rules from a file.

    --files-from stringArray

    Reads a list of source object names from a file, with empty lines or comment lines ignored.

    --files-from-raw stringArray

    Reads a list of source object names from a file, without ignoring empty lines and comment lines.

    --max-age Duration

    Excludes objects whose last modified time is earlier than the value of this option. The default unit is s (seconds). You can suffix the numerical value with a unit: ms for milliseconds, s for seconds, m for minutes, h for hours, d for days, w for weeks, M for months, or y for years.

    --min-age Duration

    Excludes objects whose last modified time is later than the value of this option. The default unit is s (seconds). You can suffix the numerical value with a unit: ms for milliseconds, s for seconds, m for minutes, h for hours, d for days, w for weeks, M for months, or y for years.

    --max-mtime Time

    Excludes objects whose last modified time is later than the value of this option. The time must be in UTC.

    --min-mtime Time

    Excludes objects whose last modified time is earlier than the value of this option. The time must be in UTC.

    --max-size SizeSuffix

    The maximum size of a file or object that can be transferred. By default, the size is in bytes (B). You can suffix the numerical value with a 1024-based unit: B for byte, K for Kibibyte, M for Mebibyte, G for Gibibyte, T for Tebibyte, or P for Pebibyte.

    --min-size SizeSuffix

    The minimum size of a file or object that can be transferred. By default, the size is in bytes (B). You can suffix the numerical value with a 1024-based unit: B for byte, K for Kibibyte, M for Mebibyte, G for Gibibyte, T for Tebibyte, or P for Pebibyte.

    --metadata-exclude stringArray

    The exclusion rule for object metadata.

    --metadata-include stringArray

    The inclusion rule for object metadata.

    --metadata-filter stringArray

    The filter rule for object metadata.

    --metadata-filter-from stringArray

    Reads metadata filter rules from a file.

    --checksum

    Specifies that only when one of the following conditions is met, the source object is copied: a. the size of the source object is different from that of the destination object; b. the CRC-64 of the source object is different from that of the destination object, but the size of the source object is the same as that of the destination object.

    --size-only

    Copies a source file or object only if it has a different file size from the destination object or file.

    -u, --update

    Copies a source file or object only if its last modified time is later than that of its destination object or file.

    -r, --recursive

    Specifies that the command performs the operation recursively. If this option is specified, the command performs the operation on all matching objects in the bucket. If this option is not specified, the command performs the operation only on the specified objects in the specified path.

    -d, --dirs

    Matches files or objects and subdirectories in the current directory.

    -f, --force

    Forces the operation without requiring confirmation.

    --list-objects

    Lists objects by calling the ListObjects operation.

    --no-progress

    Hides the progress bar.

    -j, --job int

    The number of concurrent jobs for processing multiple files or objects. Default value: 3.

    --checkers int

    The number of checkers that can run in parallel. Default value: 16.

    --parallel int

    The number of concurrent jobs for processing a single file or object.

    --part-size SizeSuffix

    The part size. By default, a part size is automatically calculated based on the object size. The part size ranges from 100 KiB to 5 GiB.

    --bigfile-threshold int

    The object size threshold for using multipart upload, download, or copy. Default value: 104857600.

    Options related to object attributes

    --version-id string

    The version ID of the object.

    --storage-class string

    The storage class of the object. Valid values: Standard, IA, Archive, ColdArchive, and DeepColdArchive.

    --acl string

    The access control list (ACL) of the object. Valid values: private, public-read, public-read-write, and default.

    --cache-control string

    The caching behavior in web browsers when the object is downloaded.

    --content-disposition string

    The display mode for the object.

    --content-encoding string

    The encoding method used to encode the object.

    --content-type string

    The content type of the object.

    --expires string

    The absolute expiration time of the cached content.

    --copy-props string

    The attributes to copy from the source object. Valid values: none, metadata, and default.

    --tagging strings

    The object tags in key=value pairs.

    --tagging-directive string

    The method used to configure the tags of the destination object. Valid values: COPY and REPLACE.

    --metadata strings

    The user metadata in key=value pairs.

    --metadata-directive string

    The method used to set metadata for the destination object. Valid values: COPY and REPLACE.

    Filter options

    Filter options are applicable to commands that support batch processing, such as ls, cp, sync, and rm. Filter options match files or objects by path, file or object name, path list, file list, object list, last modified time, and file or object size. Metadata-based filtering applies only to objects.

    Filter options can be used only in command lines. They cannot be specified in the configuration file.

    You can use the ls command to test your filters. You can also use the --dry-run/-n option to perform a test run without making any changes.

    Filter by path, file name, or object name

    You can use the --include and --exclude options to include and exclude files or objects by name or path, as in the --include="*.txt" and --exclude="*" rules.

    You can specify inclusion and exclusion rules directly in command lines or by writing them to a rule file. You can combine the --include and --exclude options based on your requirements and specify them multiple times in a single command. All occurrences of the options define a list of inclusion and exclusion rules, which match files or objects based on the following principles:

    • The order of the options matters. The rules are applied from the left to the right in command lines and top to bottom in rule files.

    • A detected file, object, or path is checked immediately for inclusion or exclusion.

    • Rule match checking occurs in order. If a file, object, or path is a match for an inclusion or exclusion rule, it is no longer checked against the subsequent rules.

    • If a file, object or path does not match any specified rules, the file, object, or path is included.

    • If a directory in a local file system is excluded, all subdirectories and files in the directory are excluded.

    • Objects are flatly stored in a bucket and are checked one by one for inclusion or exclusion.

    ossutil uses glob matching, in which the following matching patterns are common:

    • A matching pattern with a leading forward slash (/) searches for matches directly within the specified path. A matching pattern without a leading forward slash (/) searches for matches at any depth within the path.

    • A matching pattern with a trailing forward slash (/) matches only directories. A matching pattern without a trailing forward slash (/) matches directories and files.

    • A forward slash (/) in the middle of a matching pattern is a path separator. Example: subdir/test.txt.

    • The * wildcard character matches any sequence of characters of arbitrary length, excluding /.

    • The ** element in the end of a matching pattern matches any sequence of characters of arbitrary length, including /. For example, dir/** matches the path dir/subdir/xx/test.txt.

    • The ? character matches any single character, excluding /.

    • The square brackets ([]) matches any single character within the defined character class. For example, [a-z] matches a lowercase letter from a to z, and [0-9] matches a digit from 0 to 9.

    • By default, a pattern must completely align with the end of the path or file name. For example, the foo pattern matches foo and dir/foo, but does not match dir/foo1 or foo/xx.

    • Matching is based on the integrity of the specified file name, object name, or directory name and does not support name splitting. For example, the foo pattern does not match xxx/afoo, and abc/foo does not match subdir/aabc/foo.

    Note

    ossutil uses the forward slash (/) instead of the backslash (\) as path separators on Windows.

    The following table describes filter rule syntax and options that you can use to filter files or objects by path or name.

    Item

    Description

    Filter rule syntax

    • rule pattern: glob matching pattern

    • rule filename: the path to the file that contains glob matching patterns

    • rule modifiers pattern: glob matching pattern with a directive. The + directive indicates inclusion. The - directive indicates exclusion.

    Filter options

    • --include

    • --include-from

    • --exclude

    • --exclude-from

    • --filter

    • --filter-from

    The following table describes the filter options.

    Option

    Description

    --include

    --exclude

    Specify a matching pattern after the option name. Each option is followed by only one matching pattern (also called a rule). You can specify multiple matching patterns in a command by repeating the options, as in the --include "*.txt" --include "*.js" --exclude "*.jpg" settings.

    --filter

    Specify a matching pattern that contains a directive. Each option is followed by only one matching pattern. You can specify multiple matching patterns in a command by repeating the option, as the in --filter "+ *.txt" --filter "+ *.js" --filter "- *.txt" settings.

    --include-from

    --exclude-from

    Specify the path to a rule file after the option name. Each glob matching pattern is placed in a separate line in the rule file. You can specify multiple rule files in a command by repeating the options.

    The rule file that follows the --include-from option contains only inclusion rules. The rule file that follows the --exclude-from option contains only exclusion rules.

    For example, you can specify the following filters in a command: --include-from include-file.txt --exclude-from exclude-file.txt,

    where the include-file.txt file contains the following patterns:

     *.txt
 *.js

    and the exclude-file.txt file contains the *.jpg pattern.

    --filter-from

    Specify the path to a rule file after the option name. Each directive-based glob matching pattern is placed in a separate line in the rule file.

    You can specify multiple rule files in a command by repeating the option.

    For example, you can specify --filter-from filter-file.txt

    that contains the following patterns.

     # a filter rule file
 + *.txt
 + *.js
 - **

    Notes:

    • Matching patterns in a rule file are loaded and handled from top to bottom. Empty lines and comment lines that start with the number sign (#) or semicolon (;) are ignored.

    • If the file path is set to -, data is loaded from the standard input. You can specify only one "-" in a command.

    • If the --include or --include-from option is specified, ossutil automatically adds the - ** pattern to the bottom of the rule list defined by the option.

    • If the --filter option is used, ossutil does not automatically add the - ** pattern to the bottom of the rule list defined by the option.

    For example, if you want to use the --filter option to include only files whose extension is .jpg, you must specify the --filter "+ *.jpg" --filter "- **" filters.

    Note

    You cannot use --filter or --filter-from together with --include, --include-from, --exclude, or --exclude-from.

    Filter by path list, file list, or object list

    You can use the options described in the following table to specify a file that contains a list of paths, object names, or file names to filter.

    Option

    Description

    --files-from

    Reads paths, file names, or object names from the specified list file. You can specify multiple list files in a command by repeating the option.

    When ossutil reads data from the list file, it ignores leading and trailing spaces in each line, and lines that start with the number sign (#) or semicolon (;).

    --files-from-raw

    Reads paths, file names, or object names from the specified list file. You can specify multiple list files in a command by repeating the option.

    When ossutil reads data from the list file, it does not ignore leading or trailing spaces in each line, or lines that start with the number sign (#) or semicolon (;).

    Note

    If you filter objects or files by using a list of paths, file names, or object names, other types of filters specified in the command are ignored.

    Filter by last modified time

    You can filter files or objects by last modified time.

    You can specify relative and absolute time filters. The following table describes the options that filter files or objects by last modified time.

    Option

    Description

    --min-age

    Excludes objects whose last modified time is later than the value of this option. The default unit is s (seconds). You can suffix the numerical value with a unit of time. For example, 1h indicates one hour.

    --max-age

    Excludes objects whose last modified time is earlier than the value of this option. The default unit is s (seconds). You can suffix the numerical value with a unit of time. For example, 1h indicates one hour.

    --min-mtime

    Excludes objects whose last modified time is earlier than the value of this option. The time must be in UTC. Example: 2006-01-02T15:04:05.

    --max-mtime

    Excludes objects whose last modified time is later than the value of this option. The time must be in UTC. Example: 2006-01-02T15:04:05.

    Notes:

    • The --min-age and --max-age options allow you to suffix the numerical value with a unit of time: ms for milliseconds, s for seconds, m for minutes, h for hours, d for days, w for weeks, M for months, and y for years.

    • The numerical value before a unit can be a decimal, as in 1.5d (one and a half days).

    The following sample command lists files or objects that were last modified more than three days ago. 

    ossutil ls oss://bucket/ --min-age 3d

    Filter by data size

    You can filter files or objects by their data size to include files or objects whose sizes are within the specified range.

    The following table describes the options that you can use to filter files and objects by data size.

    Option

    Description

    --min-size

    The minimum size of a file or object that can be included. By default, the size is in bytes (B). You can specify a 1024-based size unit after the numerical value, as in 1K (1,024 bytes).

    --max-size

    The maximum size of a file or object that can be included. By default, the size is in bytes (B). You can suffix the numerical value with a 1024-based unit, as in 1K (1,024 bytes).

    Notes:

    • You can specify one of the following short-form size unit: B for byte, K for Kibibyte, M for Mebibyte, G for Gibibyte, T for Tebibyte, or P for Pebibyte.

    • The numerical value before a unit can be a decimal, as in 0.5K.

    The following sample command lists files or objects that do not exceed 1 MiB in size: 

    ossutil ls oss://bucket/ --max-size 1M

    Filter by metadata

    Metadata-based filtering applies only to objects in a bucket. You can filter objects by metadata in a similar way you filter files or objects by path, file name, or object name.

    Metadata-based filtering uses glob matching and requires key=value representation of metadata filter rules.

    You can use the following options to filter objects by metadata:

    • --metadata-include

    • --metadata-exclude

    • --metadata-filter

    • --metadata-filter-from

    You can filter objects by using the following metadata items:

    • Storage class. To filter objects by storage class, specify x-oss-storage-class=value, where the value can be Standard, IA, Archive, ColdArchive, or DeepColdArchive.

    • Object type. To filter objects by object type, specify x-oss-object-type=value, where the value can be Normal, Multipart, Appendable, or Symlink.

    • Restoration status. To filter objects by restoration status, specify x-oss-restore=value, where the value is the actual restoration status that you want to filter by.

    • Content type. To filter objects by content type, specify content-type=value, where the value is the actual content type that you want to filter by.

    • User metadata. To filter objects by user metadata, specify x-oss-meta-aaa=value, where aaa is the custom attribute name in lowercase and value is the metadata value.

    The following sample command lists objects in the Archive, Cold Archive, and Deep Cold Archive storage classes in the specified bucket: 

    ossutil ls oss://bucket/ --metadata-include "x-oss-storage-class=*Archive"