Use Function Compute to package and download multiple objects from OSS

Process and principles

Process

The following flowchart describes the process of downloading multiple objects from OSS at a time by using Function Compute:

1. A user invokes a function and specifies the OSS bucket and objects to be compressed.

2. Function Compute downloads objects from OSS and compress the objects into a ZIP file.

3. Function Compute uploads the ZIP file to OSS.

4. Function Compute returns the URL of the ZIP file in OSS.

5. Download the ZIP file by using the URL that is returned in Step 4.

Principles

The following figure shows how Step 2 and Step 3 in the preceding flowchart are implemented.

This solution brings the following benefits:

• The streaming download and upload method is used. This way, only a small amount of data is cached in the memory and the limited disk space of function running environments is not consumed.

• The multipart upload feature of OSS is used to concurrently upload parts of the ZIP file in queues in multi-threaded mode.

Procedure

1. Run the following command to initialize a project:

   sudo s init start-zip-oss -d start-zip-oss 

2. Run the following command to access the project and deploy the project:

   cd start-zip-oss && sudo s deploy - y

   After the deployment, an output that is similar to the following code snippet is returned. Records the value of system_url. The value is used when you invoke the function.

   
    Checking Service, Function, Triggers (2.44s)
    Creating Service, Function, Triggers (1.18s)
   
   ......
   fc-zip-oss-service:
     region:   cn-hangzhou
     service:
       name: zip-oss
     function:
       name:       zip-oss-func
       runtime:    python3.9
       handler:    main.main_handler
       memorySize: 3072
       timeout:    1800
     url:
       system_url:          https://zip-oss-func-zip-oss-****.cn-hangzhou.fcapp.run
       system_intranet_url: https://zip-oss-func-zip-oss-****.cn-hangzhou-vpc.fcapp.run
     triggers:
       -
         type: http
         name: http-test

3. After the application is deployed, invoke the deployed function to download multiple files at a time.

   1. Create the event.json file in the start-zip-oss directory and specify the bucket name and the objects to be downloaded in the file.

      • Use source-dir to specify the directory where the objects to be downloaded are located.

        cat <<EOF > event.json
        {
          "bucket": "bucketname", 
          "source-dir": "filepath/"
        }
        EOF

      • Use source-files to specify a list of objects to download.

        cat <<EOF > event.json
        {
          "bucket": "bucketname", 
          "source-files": ["files1.txt","filepath/files2.txt"]
        }
        EOF

   2. Use the curl command to directly invoke the function.

      Create a folder in the start-zip-oss directory to store compressed files. In this example, tmp folder is used. The following code snippet shows an example of curl command:

      Note

      Replace the endpoint in the sample command to the value of system_url that is returned after the application is deployed.

      curl -v -L -o /tmp/my.zip -d @./event.json https://zip-oss-func-zip-oss-****.cn-hangzhou.fcapp.run

      After the invocation, you can view the downloaded compressed file in the start-zip-oss/tmp folder. Log on to the OSS console and view the downloaded compressed files in the output directory of the bucket.

Test data

More information

• You can deploy a function in Function Compute to automatically decompress ZIP files is OSS. You can also configure automatic decompression in OSS. For more information, see Use Function Compute to automatically decompress files in OSS and ZIP package decompression.

• If you want to associate more than 10 triggers with a bucket, you can create EventBridge-based OSS triggers. For more information, see Configure an EventBridge-based OSS trigger.