All Products
Search
Document Center

Function Compute:Use Function Compute to package and download multiple objects from OSS

Last Updated:Feb 02, 2024

When you download objects from Object Storage Service (OSS), it may be inconvenient to download them one by one if the objects are small in size but large in number. You can use Function Compute to package and download multiple objects from OSS to your on-premises computer at a time. This solution allows you to quickly download multiple objects at lower costs.

Process and principles

Process

The following flowchart describes the process of downloading multiple objects from OSS at a time by using Function Compute:

image
  1. A user invokes a function and specifies the OSS bucket and objects to be compressed.

  2. Function Compute downloads objects from OSS and compress the objects into a ZIP file.

  3. Function Compute uploads the ZIP file to OSS.

  4. Function Compute returns the URL of the ZIP file in OSS.

  5. Download the ZIP file by using the URL that is returned in Step 4.

Principles

The following figure shows how Step 2 and Step 3 in the preceding flowchart are implemented.

image

This solution brings the following benefits:

  • The streaming download and upload method is used. This way, only a small amount of data is cached in the memory and the limited disk space of function running environments is not consumed.

  • The multipart upload feature of OSS is used to concurrently upload parts of the ZIP file in queues in multi-threaded mode.

Prerequisites

Procedure

  1. Run the following command to initialize a project:

    sudo s init start-zip-oss -d start-zip-oss 
  2. Run the following command to access the project and deploy the project:

    cd start-zip-oss && sudo s deploy - y

    After the deployment, an output that is similar to the following code snippet is returned. Records the value of system_url. The value is used when you invoke the function.

    
     Checking Service, Function, Triggers (2.44s)
     Creating Service, Function, Triggers (1.18s)
    
    ......
    fc-zip-oss-service:
      region:   cn-hangzhou
      service:
        name: zip-oss
      function:
        name:       zip-oss-func
        runtime:    python3.9
        handler:    main.main_handler
        memorySize: 3072
        timeout:    1800
      url:
        system_url:          https://zip-oss-func-zip-oss-****.cn-hangzhou.fcapp.run
        system_intranet_url: https://zip-oss-func-zip-oss-****.cn-hangzhou-vpc.fcapp.run
      triggers:
        -
          type: http
          name: http-test
  3. After the application is deployed, invoke the deployed function to download multiple files at a time.

    1. Create the event.json file in the start-zip-oss directory and specify the bucket name and the objects to be downloaded in the file.

      • Use source-dir to specify the directory where the objects to be downloaded are located.

        cat <<EOF > event.json
        {
          "bucket": "bucketname", 
          "source-dir": "filepath/"
        }
        EOF
      • Use source-files to specify a list of objects to download.

        cat <<EOF > event.json
        {
          "bucket": "bucketname", 
          "source-files": ["files1.txt","filepath/files2.txt"]
        }
        EOF
    2. Use the curl command to directly invoke the function.

      Create a folder in the start-zip-oss directory to store compressed files. In this example, tmp folder is used. The following code snippet shows an example of curl command:

      Note

      Replace the endpoint in the sample command to the value of system_url that is returned after the application is deployed.

      curl -v -L -o /tmp/my.zip -d @./event.json https://zip-oss-func-zip-oss-****.cn-hangzhou.fcapp.run

      After the invocation, you can view the downloaded compressed file in the start-zip-oss/tmp folder. Log on to the OSS console and view the downloaded compressed files in the output directory of the bucket.

Test data

Test

Number of objects

Total size before compression

Total size after compression

Execution duration

1

7

1.2 MB

1.16 MB

0.4s

2

57

1.06 GB

0.91 GB

63s

Description:

  • Test 1 shows that the storage space of objects are reduced if you use Function Compute to download multiple objects at a time.

  • Test 2 shows that a large number of objects can be downloaded in a short period of time if you use Function Compute to download multiple objects at a time.

More information