OpenSearch:Best practices for vectorized image retrieval

Solution architecture

There are three methods for users to upload images to build an image search engine:

• OSS+MaxCompute+OpenSearch Retrieval Engine Edition: Users initially upload images to OSS and store business table data along with corresponding image addresses (OSS paths, e.g., /image/1.jpg) in MaxCompute.

• MaxCompute+OpenSearch Retrieval Engine Edition: Users store Base64-encoded images and their table data in MaxCompute.

• API+OpenSearch Retrieval Engine Edition: Users push Base64-encoded images and their table data to the OpenSearch Retrieval Engine Edition instance via the data push interface provided by OpenSearch Retrieval Engine Edition.

This document focuses on building an image search engine using OSS+MaxCompute+OpenSearch Retrieval Engine Edition.

Prerequisites

1. Create AK and SK

Upon first logging into the Alibaba Cloud console, you are prompted to create an access key to continue using the service.

• Creating and using application dependencies necessitates access key parameters, which must not be empty under the primary account.

• Once you have created access key parameters for the primary account, you can also generate RAM user access keys for use by RAM users, who are assigned appropriate access privileges. For more information, see RAM user creation and authorization.

2. Create Object Storage Service (OSS)

1. Activate the OSS service

2. Create a bucket through the console

3. Upload files using the console

For this document, 1000 images were uploaded to OSS:

The images are located at: /test/images/

Purchase an OpenSearch Retrieval Engine Edition instance

For details on purchasing an OpenSearch Retrieval Engine Edition instance, see Purchase an OpenSearch Retrieval Engine Edition instance.

Configure tables

For new instances, the status on the product page is "Pending Configuration," and an empty cluster with the same number and specifications of query and data nodes as purchased will be automatically deployed. You must configure Table Basic Information>Data Synchronization>Index Schema>Index Rebuild for the cluster to enable normal search operations.

1. Configuring table basic information involves customizing the Table Name, setting the Number of Shards, and specifying the Number of Data Update Resources:

2. For data synchronization, configure the full data source (current options include MaxCompute data source, API push data source, and Object Storage Service (OSS)). Here, MaxCompute data source is used as an example: Click "Add Data Source," select "MaxCompute" as the data source type, input project, accesskeyID, accesskeySecret, Table, partition key, and optionally enable "Automatic Index Rebuild":

3. Upon successful configuration of the data source, proceed to set up the Index Schema:

4.1. With the MaxCompute data source established, the data source fields are automatically mapped:

1. Field settings: The end-to-end image search solution converts images into vectors through the system and then retrieves data through vectors. Therefore, three fields need to be configured here, as shown in the figure above (names can be customized):

   1. Primary key field: The type can be STRING or integer, and the primary key needs to be selected.

   2. vector_source_image: Stores the image path in OSS, in this document: /test/images/10031.png, with the following advanced configuration: The Pro configuration is as follows:

{
  "content_type": "oss",
  "oss_endpoint": "oss-cn-hangzhou-internal.aliyuncs.com",
  "oss_bucket": "oss bucket name, in this document test-image-vector",
  "crop": "true", //Image to vector, pass the value when using OSS image to vector, and the value must be "true"
  "oss_use_slr": "true",//Use service-linked role (slr) to access OSS-related APIs, must be a string type "true"
  "uid": ""//Alibaba Cloud account uid
}

vector: The field that stores the vector after the image is converted to a vector. This field needs to be set to FLOAT type and multi-value selected:

The Pro configuration is as follows:

{
  "vector_model": "clip",
  "vector_modal": "image",
  "vector_source_field": "vector_source_image"  
}

4.2. Index settings: Essential indexes to configure are the primary key index and the vector index.

4.3. Edit vector index fields:

4.4. Edit advanced configuration of vector index: In the advanced configuration of the vector index, the dimension must be set to 512 dimensions, which is fixed and cannot be modified. Other parameters should typically remain at their default settings.

Example schema:

"fields": [
    {
      "field_name": "id",
      "field_type": "INT64",
      "compress_type": "equal"
    },
    {
      "user_defined_param":
      {
        "oss_endpoint": "oss-cn-hangzhou-internal.aliyuncs.com",
        "oss_bucket": "/opensearch",
        "crop": "true",
        "content_type": "oss",
        "oss_use_slr": "true",
        "uid": "xxx"
      },
      "field_name": "source_image",
      "field_type": "STRING",
      "compress_type": "uniq"
    },
    {
      "field_name": "cate_id",
      "field_type": "INT64",
      "compress_type": "equal"
    },
    {
      "user_defined_param": {
        "vector_model": "clip",
        "vector_modal": "image",
        "vector_source_field": "vector_source_image"
      },
      "field_name": "vector",
      "field_type": "FLOAT",
      "multi_value": true
    }
  ]

5. Once configuration is complete, click Confirm to create.

6. You can monitor the creation progress in Function Extension>Change History. Once complete, you can conduct query tests:

Effect test

Syntax introduction

query=image_index:'text content to be searched&modal=text&n=10&search_params={}'&&kvpairs=formula:proxima_score(vector)&&sort=+RANK

• modal specifies the search mode. Set modal to text for text-based searches and to image for image-based searches.

• n defines the number of top results to return from the vector search.

Search by text

Execute a HA query on the query test page.

vector:'motorcycle helmet&modal=text&n=10&search_params={}'&&kvpairs=formula:proxima_score(vector)&&sort=+RANK

Inspect the 2042.png image in OSS.

Search by image

Due to the extensive length of encoded images, direct searches on the query test page in the console are not currently supported. Instead, users can perform searches using the SDK.

Example:

vector:'Base64 encoded image&modal=image&n=10&search_params={}'&&kvpairs=formula:proxima_score(vector)&&sort=+RANK

Retrieve data in SDK

Add the following dependency:

pip install alibabacloud-ha3engine

Search demonstration:

# -*- coding: utf-8 -*-

from alibabacloud_ha3engine import models, client
from alibabacloud_tea_util import models as util_models
from Tea.exceptions import TeaException, RetryError
def search():
    Config = models.Config(
        endpoint="Refer to the API domain name under API entry on the instance details page",
        instance_id="",
        protocol="http",
        access_user_name="Username set when purchasing the instance",
        access_pass_word="Password set when purchasing the instance"
    )

    # If the user request time is long, you can use this configuration to increase the request waiting time. Unit ms
    # This parameter can be used in the search_with_options method
    runtime = util_models.RuntimeOptions(
        connect_timeout=5000,
        read_timeout=10000,
        autoretry=False,
        ignore_ssl=False,
        max_idle_conns=50
    )

    # Initialize Ha3Engine Client
    ha3EngineClient = client.Client(Config)

    optionsHeaders = {}

    try:
        # Example 1: Directly use the ha query string for search.
        # =====================================================
        query_str = "config=hit:4,format:json,fetch_summary_type:pk,qrs_chain:search&&query=image_index:'text content to be searched&modal=text&n=10&search_params={}'&&cluster=general"
        haSearchQuery = models.SearchQuery(query=query_str)
        haSearchRequestModel = models.SearchRequestModel(optionsHeaders, haSearchQuery)
        hastrSearchResponseModel = ha3EngineClient.search(haSearchRequestModel)
        print(hastrSearchResponseModel)
    except TeaException as e:
        print(f"send request with TeaException : {e}")
    except RetryError as e:
        print(f"send request with Connection Exception  : {e}")

Precautions

● For stringent requirements on vector retrieval time, consider the mmap strategy example.

● The field for storing the image path or Base64-encoded image should be set as STRING type. ● The vector index must be configured as CUSTOMIZED type. ● This scenario supports HA syntax and RESTFUL API, but not SQL.