Intelligent Media Management:Face clustering

Scenarios

Person albums in cloud drives

Face clustering groups photos by face in a cloud drive and generates albums of individual persons.

Home surveillance

Face clustering records faces of family members. The face of a stranger cannot be grouped into any existing face clusters and triggers an alert. This helps you identify and handle dangerous persons and events at the earliest opportunity and ensure the safety of your family members.

Customer management in New Retail

Face clustering can be used to deduplicate photos of customers and obtain accurate customer traffic, which can be used to analyze customer purchase preferences for precise marketing.

Prerequisites

• The metadata of images to be grouped is indexed into the corresponding dataset. You can automatically index metadata by calling the CreateBinding to bind a bucket to a dataset, or manually index metadata by calling the IndexFileMeta or BatchIndexFileMeta operation.CreateDataset

• At least three face images of the same person that meet the following requirements are contained in the dataset:

• The face area of an image must be greater than 75 × 75 pixels.

• The absolute value of each of the three elements in the HeadPose parameter must be less than 30.

• The value of the FaceQuality parameter must be greater than 0.8.

Note

• You can call the GetFileMeta operation to query the head pose and face quality.

• After a face cluster is created, faces that do not meet the preceding requirements may also be added to the same cluster. For FAQ about face clustering, see FAQ about image management.

Create a face clustering task

Call the CreateFigureClusteringTask operation to create a face clustering task to cluster faces of different persons in images by person. The following example shows how to perform face clustering on images in the test-dataset dataset.

Sample request

{
    "ProjectName": "test-project",
    "DatasetName": "test-dataset"
}

Sample response

{
    "TaskId": "CreateFigureClusteringTask-ba5784b8-f61e-485d-8ea0-****",
    "RequestId": "42F4F8FD-006D-0EF0-8F2A-****",
    "EventId": "140-1L5dh6eSUErqdxV1ZvJ****"
}

# -*- coding: utf-8 -*-

import os
from alibabacloud_imm20200930.client import Client as imm20200930Client
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_imm20200930 import models as imm_20200930_models
from alibabacloud_tea_util import models as util_models
from alibabacloud_tea_util.client import Client as UtilClient


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client(
        access_key_id: str,
        access_key_secret: str,
    ) -> imm20200930Client:
        """
        Use your AccessKey ID and AccessKey secret to initialize the client. 
        @param access_key_id:
        @param access_key_secret:
        @return: Client
        @throws Exception
        """
        config = open_api_models.Config(
            access_key_id=access_key_id,
            access_key_secret=access_key_secret
        )
        # Specify the endpoint. 
        config.endpoint = f'imm.cn-beijing.aliyuncs.com'
        return imm20200930Client(config)

    @staticmethod
    def main() -> None:
        # The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. 
        # For security reasons, we recommend that you do not embed your AccessKey pair in your project code. 
        # In this example, the AccessKey pair is obtained from the environment variables to implement identity verification for API access. For information about how to configure environment variables, visit https://www.alibabacloud.com/help/en/imm/developer-reference/configure-environment-variables. 
        imm_access_key_id = os.getenv("AccessKeyId")
        imm_access_key_secret = os.getenv("AccessKeySecret")
        # Initialize the client. 
        client = Sample.create_client(imm_access_key_id, imm_access_key_secret)
        # Construct a request. 
        create_figure_clustering_task_request = imm_20200930_models.CreateFigureClusteringTaskRequest(
            # Specify the name of the IMM project. 
            project_name='test-project',
            # Specify the name of the dataset.
            dataset_name='test-dataset'
        )
        runtime = util_models.RuntimeOptions()
        try:
            # Print the response.
            response = client.create_figure_clustering_task_with_options(
                create_figure_clustering_task_request, runtime)
            print(response.body.to_map())
        except Exception as error:
            # Print the error message if necessary. 
            UtilClient.assert_as_string(error.message)
            print(error)


if __name__ == '__main__':
    Sample.main()

Query face clusters

After a face clustering task is created, you can call the QueryFigureClusters operation to query the information about clusters, such as the number of clusters and the number of images in each cluster. The following example shows how to query the information about face clusters in the test-dataset dataset.

Sample request

{
    "ProjectName": "test-project",
    "DatasetName": "test-dataset"
}

Sample response

{
    "FigureClusters": [
        {
            "AverageAge": 27.125,
            "Cover": {
                "Addresses": [],
                "AudioCovers": [],
                "AudioStreams": [],
                "CroppingSuggestions": [],
                "Figures": [
                    {
                        "Attractive": 0.9980000257492065,
                        "Beard": "none",
                        "BeardConfidence": 0.9959999918937683,
                        "Boundary": {
                            "Height": 270,
                            "Left": 573,
                            "Top": 104,
                            "Width": 202
                        },
                        "FaceQuality": 1.0,
                        "FigureId": "d7365ab8-1378-4bec-83cb-eccad8d11e0b",
                        "FigureType": "face",
                        "Glasses": "none",
                        "GlassesConfidence": 0.9990000128746033,
                        "Hat": "none",
                        "HatConfidence": 1.0,
                        "HeadPose": {
                            "Pitch": -0.7369999885559082,
                            "Roll": 2.5399999618530273,
                            "Yaw": 9.138999938964844
                        },
                        "Mask": "none",
                        "MaskConfidence": 0.7269999980926514,
                        "Mouth": "open",
                        "MouthConfidence": 0.9959999918937683,
                        "Sharpness": 1.0
                    }
                ],
                "ImageHeight": 683,
                "ImageWidth": 1024,
                "Labels": [],
                "OCRContents": [],
                "ObjectId": "170ffdeb36cec846f4214c78a0f3a0d4b7e37d0305370216ae780f7b8c72f871",
                "Subtitles": [],
                "URI": "oss://bucket1/photos/2.jpg",
                "VideoStreams": []
            },
            "CreateTime": "2022-07-12T16:41:19.336825716+08:00",
            "DatasetName": "dataset1",
            "FaceCount": 16,
            "Gender": "female",
            "ImageCount": 16,
            "MaxAge": 30.0,
            "MinAge": 23.0,
            "ObjectId": "Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6",
            "ObjectType": "figure-cluster",
            "OwnerId": "*****",
            "ProjectName": "test-project",
            "UpdateTime": "2022-09-19T17:08:59.374781532+08:00",
            "VideoCount": 0
        },
        {
            "AverageAge": 24.200000762939453,
            "Cover": {
                "Addresses": [],
                "AudioCovers": [],
                "AudioStreams": [],
                "CroppingSuggestions": [],
                "Figures": [
                    {
                        "Attractive": 0.9990000128746033,
                        "Beard": "none",
                        "BeardConfidence": 0.9990000128746033,
                        "Boundary": {
                            "Height": 266,
                            "Left": 301,
                            "Top": 218,
                            "Width": 196
                        },
                        "FaceQuality": 0.8859999775886536,
                        "FigureId": "f58bbdce-f3d1-4674-be6b-43d4b47c08e1",
                        "FigureType": "face",
                        "Glasses": "none",
                        "GlassesConfidence": 1.0,
                        "Hat": "none",
                        "HatConfidence": 1.0,
                        "HeadPose": {
                            "Pitch": 13.963000297546387,
                            "Roll": -12.21399974822998,
                            "Yaw": -6.2210001945495605
                        },
                        "Mask": "none",
                        "MaskConfidence": 0.7490000128746033,
                        "Mouth": "open",
                        "MouthConfidence": 0.9940000176429749,
                        "Sharpness": 1.0
                    }
                ],
                "ImageHeight": 1024,
                "ImageWidth": 683,
                "Labels": [],
                "OCRContents": [],
                "ObjectId": "b9c80e51aa95072413e2a0a6e5262644bc3cba14a4754f54f3fa9850c4d244f1",
                "Subtitles": [],
                "URI": "oss://bucket1/photos/11.jpg",
                "VideoStreams": []
            },
            "CreateTime": "2022-09-19T17:08:59.374932448+08:00",
            "DatasetName": "test-dataset",
            "FaceCount": 5,
            "Gender": "female",
            "ImageCount": 5,
            "MaxAge": 26.0,
            "MinAge": 22.0,
            "ObjectId": "Cluster-856be781-bf5a-46d7-8494-8d7c44f5e282",
            "ObjectType": "figure-cluster",
            "OwnerId": "*****",
            "ProjectName": "test-project",
            "UpdateTime": "2022-09-19T17:08:59.374932448+08:00",
            "VideoCount": 0
        }
    ],
    "NextToken": "",
    "TotalCount": 2,
    "RequestId": "42B3DD92-FE0D-09B7-B582-*****"
}

# -*- coding: utf-8 -*-

import os
from alibabacloud_imm20200930.client import Client as imm20200930Client
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_imm20200930 import models as imm_20200930_models
from alibabacloud_tea_util import models as util_models
from alibabacloud_tea_util.client import Client as UtilClient


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client(
        access_key_id: str,
        access_key_secret: str,
    ) -> imm20200930Client:
        """
        Use your AccessKey ID and AccessKey secret to initialize the client. 
        @param access_key_id:
        @param access_key_secret:
        @return: Client
        @throws Exception
        """
        config = open_api_models.Config(
            access_key_id=access_key_id,
            access_key_secret=access_key_secret
        )
        # Specify the endpoint. 
        config.endpoint = f'imm.cn-beijing.aliyuncs.com'
        return imm20200930Client(config)

    @staticmethod
    def main() -> None:
        # The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. 
        # For security reasons, we recommend that you do not embed your AccessKey pair in your project code. 
        # In this example, the AccessKey pair is obtained from the environment variables to implement identity verification for API access. For information about how to configure environment variables, visit https://www.alibabacloud.com/help/en/imm/developer-reference/configure-environment-variables. 
        imm_access_key_id = os.getenv("AccessKeyId")
        imm_access_key_secret = os.getenv("AccessKeySecret")
        # Initialize the client. 
        client = Sample.create_client(imm_access_key_id, imm_access_key_secret)
        # Construct a request. 
        query_figure_clusters_request = imm_20200930_models.QueryFigureClustersRequest(
            # Specify the name of the IMM project. 
            project_name='test-project',
            # Specify the dataset name. 
            dataset_name='test-dataset'
        )
        runtime = util_models.RuntimeOptions()
        try:
            # Print the response of the API operation. 
            response = client.query_figure_clusters_with_options(query_figure_clusters_request, runtime)
            print(response.body.to_map())
        except Exception as error:
            # Print the error message if necessary. 
            UtilClient.assert_as_string(error.message)
            print(error)


if __name__ == '__main__':
    Sample.main()

Query images in a face cluster

You can call the SimpleQuery operation to query images in a specific cluster. The following example shows how to query images in the Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6 cluster that belongs to the test-dataset dataset:

Sample request

{
    "ProjectName": "test-project",
    "DatasetName": "test-dataset",
    "Query": "{\"Field\": \"Figures.FigureClusterId\", \"Operation\": \"eq\", \"Value\": \"Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6\"}",
    "MaxResults": 100
}

Sample responses

{
    "Aggregations": [],
    "Files": [
        {
            "Addresses": [],
            "AudioCovers": [],
            "AudioStreams": [],
            "ContentMd5": "ViAbCBHAZgNU4zvs5****==",
            "ContentType": "image/jpeg",
            "CreateTime": "2022-07-12T15:57:47.792615815+08:00",
            "CroppingSuggestions": [],
            "DatasetName": "test-dataset",
            "ETag": "\"56201B0811C0660354E33BECE4C****\"",
            "EXIF": "****",
            "Figures": [
                {
                    "FaceQuality": 1.0,
                    "FigureClusterId": "Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6",
                    "FigureConfidence": 1.0,
                    "FigureId": "cd9139bf-f339-4ec2-b5fd-****",
                    "FigureType": "face",
                    "Glasses": "none",
                    "GlassesConfidence": 0.9990000128746033,
                    "Hat": "none",
                    "HatConfidence": 1.0,
                    "HeadPose": {
                        "Pitch": -0.8999999761581421,
                        "Roll": 1.1660000085830688,
                        "Yaw": 7.932000160217285
                    },
                    "Mask": "none",
                    "MaskConfidence": 0.6830000281333923,
                    "Mouth": "close",
                    "MouthConfidence": 0.7879999876022339,
                    "Sharpness": 1.0,
                    ...
                }
            ],
            "FileHash": "\"56201B0811C0660354E33BECE****\"",
            "FileModifiedTime": "2022-07-12T15:56:41+08:00",
            "Filename": "3.jpg",
            "ImageHeight": 1024,
            "ImageScore": {
                "OverallQualityScore": 0.7490000128746033
            },
            "ImageWidth": 683,
            "Labels": [
                {
                    "CentricScore": 0.8349999785423279,
                    "LabelConfidence": 1.0,
                    "LabelLevel": 2,
                    "LabelName": "\u7167\u7247\u62cd\u6444",
                    "Language": "zh-Hans",
                    "ParentLabelName": "\u827a\u672f\u54c1"
                },
                ...
            ],
            "MediaType": "image",
            "OCRContents": [],
            "OSSCRC64": "3400224321778591044",
            "OSSObjectType": "Normal",
            "OSSStorageClass": "Standard",
            "OSSTaggingCount": 0,
            "ObjectACL": "default",
            "ObjectId": "d132a61122c659f6fc1b42ecee1662aff358c7f1720027bead225****",
            "ObjectType": "file",
            "Orientation": 1,
            "OwnerId": "****",
            "ProduceTime": "2014-02-21T00:03:36+08:00",
            "ProjectName": "test-project",
            "Size": 187674,
            "Subtitles": [],
            "URI": "oss://bucket1/1.jpg",
            "UpdateTime": "2022-07-12T16:41:19.336736388+08:00",
            "VideoStreams": []
        },
        ...
    ],
    "NextToken": "",
    "RequestId": "84E4D242-8D15-0312-B976-****"
}

# -*- coding: utf-8 -*-

import os
from alibabacloud_imm20200930.client import Client as imm20200930Client
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_imm20200930 import models as imm_20200930_models
from alibabacloud_tea_util import models as util_models
from alibabacloud_tea_util.client import Client as UtilClient


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client(
        access_key_id: str,
        access_key_secret: str,
    ) -> imm20200930Client:
        """
        Use your AccessKey ID and AccessKey secret to initialize the client. 
        @param access_key_id:
        @param access_key_secret:
        @return: Client
        @throws Exception
        """
        config = open_api_models.Config(
            access_key_id=access_key_id,
            access_key_secret=access_key_secret
        )
        # Specify the endpoint. 
        config.endpoint = f'imm.cn-beijing.aliyuncs.com'
        return imm20200930Client(config)

    @staticmethod
    def main() -> None:
        # The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. 
        # For security reasons, we recommend that you do not embed your AccessKey pair in your project code. 
        # In this example, the AccessKey pair is obtained from the environment variables to implement identity verification for API access. For information about how to configure environment variables, visit https://www.alibabacloud.com/help/en/imm/developer-reference/configure-environment-variables. 
        imm_access_key_id = os.getenv("AccessKeyId")
        imm_access_key_secret = os.getenv("AccessKeySecret")
        # Initialize the client. 
        client = Sample.create_client(imm_access_key_id, imm_access_key_secret)
        # Construct a request. 
        request = imm_20200930_models.SimpleQueryRequest()
        params = {
            # Set the query conditions. 
            "Query": {"Field": "Figures.FigureClusterId", "Operation": "eq", "Value": "Cluster-7bdbcedb-bd79-42e7-a1e2-b29a48532bd6"},
            # Specify the name of the IMM project. 
            "ProjectName": "test-project",
            # Specify the dataset name. 
            "DatasetName": "test-dataset",
            # Specify that up to 100 query results can be returned. 
            "MaxResults": 100
        }
        request.from_map(params)
        runtime = util_models.RuntimeOptions()
        try:
            # Print the response of the API operation. 
            response = client.simple_query_with_options(request, runtime)
            print(response.body.to_map())
        except Exception as error:
            # Print the error message if necessary. 
            UtilClient.assert_as_string(error.message)
            print(error)


if __name__ == '__main__':
    Sample.main()