use Serverless Devs to deploy FFmpeg-based applications to process audio and video - Function Compute

Serverless Devs allows you to deploy applications together with services such as Function Compute, Object Storage Service (OSS), and FFmpeg to process audio and video files. For example, you can use Serverless Devs to query audio and video information, add watermarks to audio and video files, and convert file formats.

Background

FFmpeg is an open source software project that consists of a set of programs that allow you to record, convert, and stream audio and video files. FFmpeg is licensed under the GNU Lesser General Public License (LGPL) or GNU General Public License (GPL). FFmpeg provides a comprehensive solution to record, convert, and stream audio and videos. FFmpeg uses libavcodec, an advanced audio and video codec library, to ensure high portability and quality in encoding and decoding scenarios. For more information, see FFmpeg.

This topic provides examples that cover the features when you process audio and video files. In this topic, Python language is used. The following table describes sample functions to implement specific features. You can modify the function code for each sample function to implement secondary development based on your business requirements.

Sample function	Description
GetMediaMeta	Queries the metadata of an audio or video file.
GetDuration	Queries the duration of an audio or video file.
GetSprites	Creates an image sprite for a video.
VideoWatermark	Adds text watermarks, image watermarks, or animated Graphics Interchange Format (GIF) watermarks to a video.
AudioConvert	Converts the format of an audio or video file.
VideoGif	Converts a video file to a GIF file.

Before you begin

Make sure that the following operations are complete:

Deploy an application by using Serverless Devs

Run the following command to initialize the project that you want to manage:
```
sudo s init ffmpeg-app -d ffmpeg-app
```
In the preceding command, -d specifies the name of the generated directory.
Run the following command to go to the directory of the project:
```
cd ffmpeg-app
```
(Optional) You can modify the sample code in the directory based on your business requirements to implement your business logic.

Run the following command to deploy the project:

sudo s deploy -y

Note

If you want to deploy a specific feature of the project, run the corresponding command. For example, if you want to deploy the GetMediaMeta feature to query the metadata information of audio or video files, run the following command:

sudo s GetMediaMeta deploy

If you want to deploy another feature, replace GetMediaMeta with another feature name.

Command output

[2021-11-25T17:35:56.524] [INFO ] [S-CLI] - Start ...
[2021-11-25T17:35:56.529] [INFO ] [S-CLI] - It is detected that your project has the following projects < AudioConvert,GetMediaMeta,GetDuration,VideoGif,GetSprites,VideoWatermark > to be execute
[2021-11-25T17:35:56.530] [INFO ] [S-CLI] - Start executing project AudioConvert
[2021-11-25T17:35:57.725] [INFO ] [FC-DEPLOY] - Using region: cn-qingdao
[2021-11-25T17:35:57.725] [INFO ] [FC-DEPLOY] - Using access alias: default
[2021-11-25T17:35:57.726] [INFO ] [FC-DEPLOY] - Using accessKeyID: LTAI4G4cwJkK4Rza6xd9****
[2021-11-25T17:35:57.726] [INFO ] [FC-DEPLOY] - Using accessKeySecret: eCc0GxSpzfq1DVspnqqd6nmYNN****
 Using fc deploy type: sdk, If you want to deploy with pulumi, you can [s cli fc-default set deploy-type pulumi] to switch.
 ......

There is auto config in the service: FcOssFFmpeg

......
AudioConvert:
  region:   cn-qingdao
  service:
    name: FcOssFFmpeg
  function:
    name:       AudioConvert
    runtime:    python3
    handler:    index.handler
    memorySize: 256
    timeout:    600
GetMediaMeta:
  region:   cn-qingdao
  service:
    name: FcOssFFmpeg
  function:
    name:       GetMediaMeta
    runtime:    python3
    handler:    index.handler
    memorySize: 1024
    timeout:    600
GetDuration:
  region:   cn-qingdao
  service:
    name: FcOssFFmpeg
  function:
    name:       GetDuration
    runtime:    python3
    handler:    index.handler
    memorySize: 256
    timeout:    600
VideoGif:
  region:   cn-qingdao
  service:
    name: FcOssFFmpeg
  function:
    name:       VideoGif
    runtime:    python3
    handler:    index.handler
    memorySize: 512
    timeout:    600
GetSprites:
  region:   cn-qingdao
  service:
    name: FcOssFFmpeg
  function:
    name:       GetSprites
    runtime:    python3
    handler:    index.handler
    memorySize: 512
    timeout:    600
VideoWatermark:
  region:   cn-qingdao
  service:
    name: FcOssFFmpeg
  function:
    name:       VideoWatermark
    runtime:    python3
    handler:    index.handler
    memorySize: 256
    timeout:    600

Call a function.

Details of invoking the GetMediaMeta function

To invoke the GetMediaMeta function to query the metadata of an audio or video file, run the following command. Replace the values of specific parameters based on your business requirements.

sudo s GetMediaMeta invoke -e '{"bucket_name": "test-bucket","object_key": "a.mp4"}'

Parameters:

bucket_name: the name of the OSS bucket.
object_key: the name of the object that you want to query the metadata.

Sample command output:

[2021-11-26T14:19:02.045] [INFO ] [S-CLI] - Start ...
========= FC invoke Logs begin =========
FunctionCompute python3 runtime inited.
FC Invoke Start RequestId: dda964e0-82b6-452a-b849-6b0b835f****
2021-11-26T06:19:04.688Z dda964e0-82b6-452a-b849-6b0b835f**** [INFO] current Function [handler] excute time is 0.23 seconds
FC Invoke End RequestId: dda964e0-82b6-452a-b849-6b0b835f****

Duration: 1238.78 ms, Billed Duration: 1239 ms, Memory Size: 1024 MB, Max Memory Used: 118.39 MB
========= FC invoke Logs end =========

FC Invoke Result:
{
    "format": {
        "bit_rate": "17024829",
        "duration": "110.037333",
        "filename": "http://test-bucket.oss-cn-qingdao-internal.aliyuncs.com/a.mp4......",
        "format_long_name": "QuickTime / MOV",
        "format_name": "mov,mp4,m4a,3gp,3g2,mj2",
        "nb_programs": 0,
        "nb_streams": 2,
        "probe_score": 100,
        "size": "234170850",
        "start_time": "0.000000",
        "tags": {
            "compatible_brands": "mp42mp41",
            "creation_time": "2020-09-05T06:03:49.000000Z",
            "major_brand": "mp42",
            "minor_version": "0"
        }
    },
    "streams": [
        {
            "avg_frame_rate": "25/1",
            "bit_rate": "16708594",
            "bits_per_raw_sample": "8",
            "chroma_location": "left",
            "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
            "codec_name": "h264",
            "codec_tag": "0x31637661",
            "codec_tag_string": "avc1",
            "codec_time_base": "1/50",
            "codec_type": "video",
            "coded_height": 1088,
            "coded_width": 1920,
            "color_primaries": "bt709",
            "color_range": "tv",
            "color_space": "bt709",
            "color_transfer": "bt709",
            "disposition": {
                "attached_pic": 0,
                "clean_effects": 0,
                "comment": 0,
                "default": 1,
                "dub": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "karaoke": 0,
                "lyrics": 0,
                "original": 0,
                "timed_thumbnails": 0,
                "visual_impaired": 0
            },
            "duration": "110.000000",
            "duration_ts": 2750000,
            "has_b_frames": 1,
            "height": 1080,
            "index": 0,
            "is_avc": "true",
            "level": 41,
            "nal_length_size": "4",
            "nb_frames": "2750",
            "pix_fmt": "yuv420p",
            "profile": "Main",
            "r_frame_rate": "25/1",
            "refs": 1,
            "start_pts": 0,
            "start_time": "0.000000",
            "tags": {
                "creation_time": "2020-09-05T06:03:49.000000Z",
                "encoder": "AVC Coding",
                "handler_name": "\u001fMainconcept Video Media Handler",
                "language": "eng"
            },
            "time_base": "1/25000",
            "width": 1920
        },
        {
            "avg_frame_rate": "0/0",
            "bit_rate": "317375",
            "bits_per_sample": 0,
            "channel_layout": "stereo",
            "channels": 2,
            "codec_long_name": "AAC (Advanced Audio Coding)",
            "codec_name": "aac",
            "codec_tag": "0x6134706d",
            "codec_tag_string": "mp4a",
            "codec_time_base": "1/48000",
            "codec_type": "audio",
            "disposition": {
                "attached_pic": 0,
                "clean_effects": 0,
                "comment": 0,
                "default": 1,
                "dub": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "karaoke": 0,
                "lyrics": 0,
                "original": 0,
                "timed_thumbnails": 0,
                "visual_impaired": 0
            },
            "duration": "110.000000",
            "duration_ts": 5280000,
            "index": 1,
            "max_bit_rate": "417750",
            "nb_frames": "5158",
            "profile": "LC",
            "r_frame_rate": "0/0",
            "sample_fmt": "fltp",
            "sample_rate": "48000",
            "start_pts": 0,
            "start_time": "0.000000",
            "tags": {
                "creation_time": "2020-09-05T06:03:49.000000Z",
                "handler_name": "#Mainconcept MP4 Sound Media Handler",
                "language": "eng"
            },
            "time_base": "1/48000"
        }
    ]
}
End of method: invoke

Details of invoking the GetDuration function

To invoke the GetDuration function to query the duration of an audio or video file, run the following command. Replace the values of specific parameters based on your business requirements.

sudo s GetDuration invoke -e '{"bucket_name": "bucket-name","object_key": "a.mp4"}'

Parameters:

bucket_name: the name of the OSS bucket.
object_key: the name of the object whose duration you want to query.

Sample command output:

[2021-11-26T14:21:48.877] [INFO ] [S-CLI] - Start ...
========= FC invoke Logs begin =========
FunctionCompute python3 runtime inited.
FC Invoke Start RequestId: 6bb9ecae-7f53-4efb-afea-7614ef87****
2021-11-26T06:21:50.273Z 6bb9ecae-7f53-4efb-afea-7614ef87**** [INFO] current Function [handler] excute time is 0.17 seconds
FC Invoke End RequestId: 6bb9ecae-7f53-4efb-afea-7614ef87****

Duration: 754.63 ms, Billed Duration: 755 ms, Memory Size: 256 MB, Max Memory Used: 61.21 MB
========= FC invoke Logs end =========

FC Invoke Result:
110.037333

End of method: invoke

Details of invoking the GetSprites function

To invoke the GetSprites function to creates a sprite of a video file, run the following command. Replace the values of specific parameters based on your business requirements.

sudo s GetSprites invoke -e '{"bucket_name": "test-bucket","object_key": "aclear.mp4", "output_dir" : "output/", "tile": "3*4"}'

Parameters:

bucket_name: the name of the OSS bucket.
object_key: the name of the object for which you want to use to create a sprite.
output_dir: the name of the OSS bucket in which you want to store the video file after you transcode the file.
tile: the rows and columns of the sprite.
(Optional) start: the start point of the video from which you want to create the sprite. Default value: 0.
(Optional) duration: the duration of the video clip for creating the sprite after you specify the start parameter. For example, if you set the start parameter to 10 and the duration parameter to 20, snapshots are captured during the 10th to 30th seconds of the video.
(Optional) itsoffset: the latency of the video stream that is displayed. Default value: 0. This parameter must be used together with the start and interval parameters. Examples:
- If you set the start parameter to 0, the interval parameter to 10, and the itsoffset parameter to 0, snapshots are captured at the 5th, 15th, and 25th seconds of the video.
- If you set the start parameter to 0, the interval parameter to 10, and the itsoffset parameter to 1, snapshots are captured at the 4th, 14th, and 24th seconds of the video.
- If you set the start parameter to 0, the interval parameter to 10, and the itsoffset parameter to -1, snapshots are captured at the 6th, 16th, and 26th seconds of the video.
- If you set the start parameter to 0, the interval parameter to 10, and the itsoffset parameter to 4.999, snapshots are captured at the 0th, 10th, and 20th seconds of the video.
Note
If you set the itsoffset parameter to 5, the snapshot captured at the 0th second is lost. We recommend that you set this parameter to 4.999.
(Optional) scale: the size of the captured snapshot. By default, the aspect ratio of the captured snapshot is -1:-1.
(Optional) interval: the interval at which snapshots are captured from the video. Unit: seconds. Default value: 1.
(Optional) padding: the distance between snapshots. Default value: 0.
(Optional) color: the background color of the sprite. By default, the background color is black.
(Optional) dst_type: the format of the sprite. Default value: JPG. Valid values: JPG and PNG.

Sample command output:

[2021-11-26T16:07:42.585] [INFO ] [S-CLI] - Start ...
========= FC invoke Logs begin =========
FunctionCompute python3 runtime inited.
FC Invoke Start RequestId: 1b427831-e10f-4c2b-b780-9b504c29aa67
2021-11-26T08:07:44.684Z 1b427831-e10f-4c2b-b780-9b504c29aa67 [INFO] b'{"bucket_name": "test-bucket","object_key": "a.mp4", "output_dir" : "output/", "dst_type":".wav"}'
2021-11-26T08:07:51.642Z 1b427831-e10f-4c2b-b780-9b504c29aa67 [INFO] Uploaded /tmp/transcoded_a.wav to output/transcoded_a.wav
2021-11-26T08:07:51.642Z 1b427831-e10f-4c2b-b780-9b504c29aa67 [INFO] current Function [handler] excute time is 6.96 seconds
FC Invoke End RequestId: 1b427831-e10f-4c2b-b780-9b504c29aa67

Duration: 7876.26 ms, Billed Duration: 7877 ms, Memory Size: 3072 MB, Max Memory Used: 119.06 MB
========= FC invoke Logs end =========

FC Invoke Result:
ok


End of method: invoke

Details of invoking the VideoWatermark function

To invoke the VideoWatermark function to add a watermark to a video file, run the following command. Replace the values of specific parameters based on your business requirements.

sudo s VideoWatermark invoke -e '{"bucket_name": "test-bucket","object_key": "a.mp4", "output_dir" : "output/", "vf_args" : "drawtext=fontfile=/usr/share/fonts/truetype/wqy/wqy-zenhei.ttc:text='hello Function Compute':x=100:y=50:fontsize=24:fontcolor=red"}'

Parameters:

bucket_name: the name of the OSS bucket.
object_key: the name of the video file to which you want to add watermarks.
output_dir: the name of the OSS bucket in which you want to store the video file after you transcode the file.
vf_args: the text watermark or image watermark that you want to add.
filter_complex_args: the animated GIF watermark that you want to add. By default, if you configure this parameter and the vf_args parameter at the same time, the value of the vf_args parameter becomes invalid.

Sample command output:

[2021-11-26T15:20:24.396] [INFO ] [S-CLI] - Start ...
========= FC invoke Logs begin =========
static-master/target/lib ......
......
FC Invoke End RequestId: 31ecddfa-4e41-44bb-9489-00708b07****

Duration: 1302.08 ms, Billed Duration: 1303 ms, Memory Size: 256 MB, Max Memory Used: 256.00 MB
========= FC invoke Logs end =========

FC Invoke Result:
ok


End of method: invoke

Details of invoking the AudioConvert function

To invoke the AudioConvert function to convert the format of an audio or a video file, run the following command. Replace the values of specific parameters based on your business requirements.

sudo s AudioConvert invoke -e '{"bucket_name": "test-bucket","object_key": "a.mp4", "output_dir" : "output/", "dst_type":".wav", "ac":"1", "ar":"4000"}'

Parameters:

bucket_name: the name of the OSS bucket.
object_key: the name of the audio file whose format you want to convert.
output_dir: the name of the OSS bucket in which you want to store the video file after you transcode the file.
dst_type: the converted format.
(Optional) ac: the number of sound channels.
(Optional) ar: the audio sampling rate.

Sample command output:

[2021-11-26T16:04:16.293] [INFO ] [S-CLI] - Start ...
========= FC invoke Logs begin =========
......
2021-11-26T08:04:18.520Z 2fc578cd-8787-4681-ab21-a3f6b4ab1e2a [ERROR] returncode:1
......
Duration: 1156.09 ms, Billed Duration: 1157 ms, Memory Size: 256 MB, Max Memory Used: 88.23 MB
========= FC invoke Logs end =========

FC Invoke Result:
ok


End of method: invoke

Details of invoking the VideoGif function

To invoke the VideoGif function to generate an animated GIF of a video file, run the following command. Replace the values of specific parameters based on your business requirements.

sudo s VideoGif invoke -e '{"bucket_name": "test-bucket","object_key": "a.mp4", "output_dir" : "output/", "vframes": "5", "start": "0",  "duration": "2"}'

Parameters:

bucket_name: the name of the OSS bucket.
object_key: the name of the video file that you want to convert to a GIF file.
output_dir: the name of the OSS bucket that is used to store the video file after you transcode the file.
(Optional) vframes: the duration of the video clip that you want to create a GIF file after you specify the start parameter.
(Optional) start: the start point of the video from which you want to convert. Default value: 0.
(Optional) duration: the duration of the video that you want to create a GIF file after you specify the start parameter.

Note

If you configure the duration and vframes parameters at the same time, the value of the duration parameter prevails. By default, if none of the two parameters are configured, the entire video is converted to a GIF file.

Sample command output:

[2021-11-26T15:27:26.647] [INFO ] [S-CLI] - Start ...
========= FC invoke Logs begin =========
FunctionCompute python3 runtime inited.
FC Invoke Start RequestId: a49fc8b4-ee8f-4e8b-9923-b8b41ced47cb
2021-11-26T07:27:28.279Z a49fc8b4-ee8f-4e8b-9923-b8b41ced47cb [INFO] b'{"bucket_name": "test-bucket","object_key": "a.mp4", "output_dir" : "output/", "vframes": "5", "start": "0",  "duration": "2"}'
2021-11-26T07:27:28.280Z a49fc8b4-ee8f-4e8b-9923-b8b41ced47cb [INFO] cmd = ffmpeg -y -ss 0 -t 2 -accurate_seek -i ......
2021-11-26T07:27:30.150Z a49fc8b4-ee8f-4e8b-9923-b8b41ced47cb [INFO] Uploaded /tmp/a.gif to output/a.gif
2021-11-26T07:27:30.151Z a49fc8b4-ee8f-4e8b-9923-b8b41ced47cb [INFO] current Function [handler] excute time is 1.87 seconds
FC Invoke End RequestId: a49fc8b4-ee8f-4e8b-9923-b8b41ced47cb

Duration: 2495.95 ms, Billed Duration: 2496 ms, Memory Size: 512 MB, Max Memory Used: 85.32 MB
========= FC invoke Logs end =========

FC Invoke Result:
ok

End of method: invoke

References

To accelerate the transcoding of large videos or perform complex operations, you can use the Serverless Workflow orchestration function to build a video processing system. For more information, see Build an elastic and highly available audio and video processing system based on the serverless architecture.
If you want to process audio and video data in real time or near real time, you can deploy applications on GPU-accelerated instances. For more information, see Best practices for audio and video processing.