SemanticQuery - Intelligent Media Management - Alibaba Cloud Documentation Center

Queries metadata in a dataset by inputting natural language.

Operation description

Precautions

Before you call this operation, make sure that you are familiar with the billing of Intelligent Media Management (IMM).**** Each time you call this operation, you are charged for semantic understanding and query fees.
Before you call this operation, make sure that the file that you want to use is indexed into the dataset that you use. To index a file into a dataset, you can call one of the following operations: CreateBinding , IndexFileMeta , and BatchIndexFileMeta .
The response provided in this example is for reference only. The categories and content of metadata vary based on configurations of workflow templates. If you have questions, search for and join the DingTalk group numbered 21714099.

Usage limits

Each time you call this operation, up to 1,000 metadata files are returned.
Pagination is not supported.
The natural language processing capability may not always produce completely accurate results.

Usage methods

You can query files within a dataset by using natural language keywords. Key information supported for understanding includes labels (Labels.LabelName), time (ProduceTime), and location (Address.AddressLine). For example, if you use 2023 Hangzhou scenery as the query criterion, the operation intelligently breaks the query criterion down into the following sub-criteria, and returns the files that meet all the sub-criteria:

ProduceTime: 00:00 on January 1, 2023 to 00:00 on December 31, 2023.
Address.AddressLine: Hangzhou
Labels.LabelName: scenery.

When you call this operation, you can configure a workflow template that includes the ImageEmbeddingExtraction operator. This allows the operation to return image content when the query you input matches the image content, thereby achieving intelligent image retrieval.``

Debugging

You can run this interface directly in OpenAPI Explorer, saving you the trouble of calculating signatures. After running successfully, OpenAPI Explorer can automatically generate SDK code samples.

Debug

Authorization information

The following table shows the authorization information corresponding to the API. The authorization information can be used in the Action policy element to grant a RAM user or RAM role the permissions to call this API operation. Description:

Operation: the value that you can use in the Action element to specify the operation on a resource.
Access level: the access level of each operation. The levels are read, write, and list.
Resource type: the type of the resource on which you can authorize the RAM user or the RAM role to perform the operation. Take note of the following items:
- The required resource types are displayed in bold characters.
- If the permissions cannot be granted at the resource level, All Resources is used in the Resource type column of the operation.
Condition Key: the condition key that is defined by the cloud service.
Associated operation: other operations that the RAM user or the RAM role must have permissions to perform to complete the operation. To complete the operation, the RAM user or the RAM role must have the permissions to perform the associated operations.

Operation	Access level	Resource type	Condition key	Associated operation
imm:SemanticQuery	list	*Dataset `acs:imm:{#regionId}:{#accountId}:project/{#ProjectName}/dataset/{#DatasetName}`	none	none

Request parameters

Parameter	Type	Required	Description	Example
NextToken	string	No	This parameter is no longer available.	MTIzNDU2Nzg6aW1tdGVzdDpleGFtcGxlYnVja2V0OmRhdGFzZXQwMDE6b3NzOi8vZXhhbXBsZWJ1Y2tldC9zYW1wbGVvYmplY3QxLmpwZw==
MaxResults	integer	No	The maximum number of entries to return. Valid values: 1 to 1000.	10
ProjectName	string	Yes	The name of the project.	immtest
DatasetName	string	Yes	The name of the dataset.	immDatatest
Query	string	Yes	The content of the query that you input.
WithFields	array	No	The fields that you want to include in the response. Including only necessary metadata fields can help reduce the size of the response. If you do not specify this parameter or set the value to null, all existing metadata fields are returned.
	string	No	The field that you want to include in the response. Including only necessary metadata fields can help reduce the size of the response. If you do not specify this parameter or set the value to null, all existing metadata fields are returned.	["URI", "Filename"]
MediaTypes	array	No	The types of the media that you want to query. Default value: ["image"]
	string	No	The type of the medium that you want to query. Set the value to image.	["image"]

Response parameters

Parameter	Type	Description	Example
	object	The response parameters.
RequestId	string	The request ID.	2C5C1E0F-D8B8-4DA0-8127-EC32C771****
Files	array	The files.
Files	File	The file.

Examples

Sample success responses

JSONformat

{
  "RequestId": "2C5C1E0F-D8B8-4DA0-8127-EC32C771****",
  "Files": [
    {
      "OwnerId": "102321002467****",
      "ProjectName": "test-project",
      "DatasetName": "test-dataset",
      "ObjectType": "file",
      "ObjectId": "75d5de2c50754e3dadd5c35dbca5f9949369e37eb342a73821f690c94c36c7f7",
      "UpdateTime": "2021-06-29T14:50:13.011643661+08:00",
      "CreateTime": "2021-06-29T14:50:13.011643661+08:00",
      "URI": "oss://examplebucket/sampleobject.jpg",
      "OSSURI": "oss://examplebucket/sampleobject.jpg",
      "Filename": "sampleobject.jpg",
      "MediaType": "image",
      "ContentType": "image/jpeg",
      "Size": 1000,
      "FileHash": "1d9c280a7c4f67f7ef873e28449dbe17",
      "FileModifiedTime": "2021-06-29T14:50:13.011643661+08:00",
      "FileCreateTime": "2021-06-29T14:50:13.011643661+08:00",
      "FileAccessTime": "2021-06-29T14:50:13.011643661+08:00",
      "ProduceTime": "2021-06-29T14:50:13.011643661+08:00",
      "LatLong": "30.134390,120.074997",
      "Timezone": "",
      "Addresses": [
        {
          "Language": "zh-hans",
          "AddressLine": "",
          "Country": "",
          "Province": "",
          "City": "",
          "District": "",
          "Township": ""
        }
      ],
      "TravelClusterId": "-",
      "Orientation": 0,
      "Figures": [
        {
          "FigureId": "868a9e74-cde5-4c7a-9013-28bb984****",
          "FigureConfidence": 1,
          "FigureClusterId": "Cluster-dbe72fec-b84c-4ab6-885b-3678e64****",
          "FigureClusterConfidence": 1,
          "FigureType": "face",
          "Age": 29,
          "AgeSD": 5,
          "Gender": "female",
          "GenderConfidence": 1,
          "Emotion": "happiness",
          "EmotionConfidence": 0.70784568786621,
          "FaceQuality": 0.960875928401947,
          "Boundary": {
            "Width": 200,
            "Height": 300,
            "Left": 10,
            "Top": 30,
            "Polygon": [
              {
                "X": 10,
                "Y": 10
              }
            ]
          },
          "Mouth": "close",
          "MouthConfidence": 1,
          "Beard": "none",
          "BeardConfidence": 1,
          "Hat": "none",
          "HatConfidence": 1,
          "Mask": "none",
          "MaskConfidence": 1,
          "Glasses": "none",
          "GlassesConfidence": 0.8,
          "Sharpness": 0.7,
          "Attractive": 0.96,
          "HeadPose": {
            "Pitch": 18.385589599609375,
            "Roll": 4.204030513763428,
            "Yaw": 2.4945924282073975
          }
        }
      ],
      "FigureCount": 10,
      "Labels": [
        {
          "Language": "zh-Hans",
          "LabelName": "",
          "LabelLevel": 1,
          "LabelConfidence": 0.9891784601980591,
          "ParentLabelName": "",
          "CentricScore": 0.7319999933242798
        }
      ],
      "Title": "test",
      "ImageWidth": 270,
      "ImageHeight": 500,
      "EXIF": "{\"Compression\":{\"value\":\"6\"},\"DateTime\":{\"value\":\"2020:08:19 17:11:11\"}}",
      "ImageScore": {
        "OverallQualityScore": 0.736
      },
      "CroppingSuggestions": [
        {
          "AspectRatio": "2:3",
          "Confidence": 0.7079545259475708,
          "Boundary": {
            "Width": 200,
            "Height": 300,
            "Left": 10,
            "Top": 30,
            "Polygon": [
              {
                "X": 10,
                "Y": 10
              }
            ]
          }
        }
      ],
      "OCRContents": [
        {
          "Language": "zh-hans",
          "Contents": "",
          "Confidence": 0.8254936695098877,
          "Boundary": {
            "Width": 200,
            "Height": 300,
            "Left": 10,
            "Top": 30,
            "Polygon": [
              {
                "X": 10,
                "Y": 10
              }
            ]
          }
        }
      ],
      "VideoWidth": 1080,
      "VideoHeight": 1920,
      "VideoStreams": [
        {
          "Index": 0,
          "Language": "en",
          "CodecName": "h264",
          "CodecLongName": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
          "Profile": "High",
          "CodecTimeBase": "1373627/83160000",
          "CodecTagString": "avc1",
          "CodecTag": "0x31637661",
          "Width": 1280,
          "Height": 720,
          "HasBFrames": 2,
          "SampleAspectRatio": "1:1",
          "DisplayAspectRatio": "16:9",
          "PixelFormat": "yuv420p",
          "Level": 40,
          "FrameRate": "25/1",
          "AverageFrameRate": "25/1",
          "TimeBase": "1/12800",
          "StartTime": 0,
          "Duration": 22.88,
          "Bitrate": 5407765,
          "FrameCount": 572,
          "Rotate": "90",
          "BitDepth": 8,
          "ColorSpace": "bt709",
          "ColorRange": "tv",
          "ColorTransfer": "bt709",
          "ColorPrimaries": "bt709"
        }
      ],
      "Subtitles": [
        {
          "Index": 2,
          "Language": "en",
          "CodecName": "mov_text",
          "CodecLongName": "MOV text",
          "CodecTagString": "tx3g",
          "CodecTag": "0x67337874",
          "StartTime": 0,
          "Duration": 71.378,
          "Bitrate": 74,
          "Content": "",
          "Width": 600,
          "Height": 30
        }
      ],
      "AudioStreams": [
        {
          "Index": 1,
          "Language": "en",
          "CodecName": "aac",
          "CodecLongName": "AAC (Advanced Audio Coding)",
          "CodecTimeBase": "1/44100",
          "CodecTagString": "mp4a",
          "CodecTag": "0x6134706d",
          "TimeBase": "1/48000",
          "StartTime": 0.0235,
          "Duration": 3.690667,
          "Bitrate": 320087,
          "FrameCount": 173,
          "Lyric": "test",
          "SampleFormat": "fltp",
          "SampleRate": 48000,
          "Channels": 2,
          "ChannelLayout": "stereo"
        }
      ],
      "Artist": "Jane",
      "AlbumArtist": "Jane",
      "AudioCovers": [
        {
          "ImageWidth": 500,
          "ImageHeight": 820,
          "EXIF": "{\"FileSize\":{\"value\":\"29304\"},\"Format\":{\"value\":\"jpg\"}}",
          "ImageScore": {
            "OverallQualityScore": 0.736
          },
          "CroppingSuggestions": [
            {
              "AspectRatio": "2:3",
              "Confidence": 0.7079545259475708
            }
          ],
          "OCRContents": [
            {
              "Language": "zh-hans",
              "Contents": "",
              "Confidence": 0.8254936695098877
            }
          ]
        }
      ],
      "Composer": "Jane",
      "Performer": "Jane",
      "Language": "eng",
      "Album": "FirstAlbum",
      "PageCount": 5,
      "ETag": "\"1D9C280A7C4F67F7EF873E28449****\"",
      "CacheControl": "no-cache",
      "ContentDisposition": "attachment; filename =test.jpg",
      "ContentEncoding": "UTF-8",
      "ContentLanguage": "zh-CN",
      "AccessControlAllowOrigin": "https://aliyundoc.com",
      "AccessControlRequestMethod": "PUT",
      "ServerSideEncryptionCustomerAlgorithm": "SM4",
      "ServerSideEncryption": "AES256",
      "ServerSideDataEncryption": "SM4",
      "ServerSideEncryptionKeyId": "9468da86-3509-4f8d-a61e-6eab1eac****",
      "OSSStorageClass": "Standard",
      "OSSCRC64": "559890638950338001",
      "ObjectACL": "default",
      "ContentMd5": "HZwoCnxPZ/fvhz4oRJ2+Fw==",
      "OSSUserMeta": {
        "key": "val"
      },
      "OSSTaggingCount": 2,
      "OSSTagging": {
        "key": "val"
      },
      "OSSExpiration": "2120-01-01T12:00:00.000Z",
      "OSSVersionId": "CAEQNhiBgMDJgZCA0BYiIDc4MGZjZGI2OTBjOTRmNTE5NmU5NmFhZjhjYmY0****",
      "OSSDeleteMarker": "CAEQMhiBgIDXiaaB0BYiIGQzYmRkZGUxMTM1ZDRjOTZhNjk4YjRjMTAyZjhl****",
      "OSSObjectType": "Normal",
      "CustomId": "\t\nmember-image-id-0001",
      "CustomLabels": {
        "MemberName": "Tim",
        "Enabled": "True",
        "ItemCount": "10"
      },
      "StreamCount": 1,
      "ProgramCount": 1,
      "FormatName": "mov",
      "FormatLongName": "QuickTime / MOV",
      "StartTime": 0,
      "Bitrate": 13091201,
      "Duration": 15.263,
      "SemanticSimilarity": 0.855123,
      "SemanticTypes": [
        "content"
      ],
      "Elements": [
        {
          "ElementContents": [
            {
              "Type": "text",
              "Content": "",
              "URL": "http://aliyun.com",
              "TimeRange": [
                0
              ]
            }
          ],
          "ObjectId": "",
          "ElementType": "",
          "SemanticSimilarity": 0,
          "ElementRelations": [
            {
              "Type": "",
              "ObjectId": ""
            }
          ]
        }
      ],
      "SceneElements": [
        {
          "TimeRange": [
            0
          ],
          "FrameTimes": [
            0
          ]
        }
      ],
      "OCRTexts": "Alibaba Cloud IMM\n",
      "FileStatus": "",
      "Reason": "[InternalError] The request has been failed due to some unknown error. status: 500, requestId: CC5ACFBD-BB7A-496D-A9D6-****",
      "SmartClusters": [
        {
          "SmartClusterId": "",
          "Similarity": 0
        }
      ],
      "ObjectStatus": "Indexed",
      "SequenceNumber": 3,
      "ClusterStatus": {
        "key": {
          "key": [
            ""
          ]
        }
      },
      "ClusterModifications": {
        "key": {
          "key": [
            ""
          ]
        }
      }
    }
  ]
}

Error codes

For a list of error codes, visit the Service error codes.

Change history

Change time	Summary of changes	Operation
2024-09-05	The request parameters of the API has changed	View Change Details
2024-09-04	The request parameters of the API has changed	View Change Details
2023-11-20	The request parameters of the API has changed. The response structure of the API has changed	View Change Details
2023-09-04	The response structure of the API has changed	View Change Details