Promo Center

50% off for new user

Direct Mail-46% off

Learn More

Best practices of using vector indexes

Updated at: 2024-08-27 07:20

This topic describes how to configure a vector index in an existing OpenSearch Retrieval Engine Edition instance and use an SDK to retrieve data by using the vector index.

Purchase an OpenSearch Retrieval Engine Edition instance

For more information, see Purchase an OpenSearch Retrieval Engine Edition instance.

Configure the instance

On the details page of the purchased instance, you can view that the instance is in Pending Configuration state and a cluster that contains no data sources is automatically deployed for the instance. The numbers and specifications of Query Result Searcher (QRS) workers and Searcher workers in the cluster are those you specify when you purchase the instance. You must configure a data source and an index schema and rebuild indexes for the cluster before you can use the search service.

image

  1. Configure the data source. You can configure a MaxCompute data source or an API data source. In this example, a MaxCompute data source is used. To configure a MaxCompute data source for the cluster, perform the following operations: In the Configure Data Source step, click Add Data Source. In the Add Data Source panel, select MaxCompute as the data source type. Configure the Project, AccessKey, AccessKey Secret, Table, and Partition Key parameters. You can specify whether to turn on Automatic Reindexing based on your business requirements. For more information, see the "Configure automatic reindexing" section of the MaxCompute data source topic.

image

After the verification is passed, click OK to add the data source.

image

  1. After the data source is configured, click Next to configure the index schema.

image

2.1. Click Create Index Table.

image

2.2. Configure the index table.

image

  • Index Table: Enter a custom name.

  • Data Source: Select the data source that you configure in Step 1.

  • Data Shards: Enter a value that is no more than the number of Searcher workers that you purchase.

2.3. Configure fields. You must configure at least two fields: a primary key field and a vector field. The vector field must be of the multi-value FLOAT data type.

image

If you want to configure a vector index with categories, you can add a category field between the primary key field and the vector field. The category field must be a single-value or multi-value field of the INTEGER data type.

image

Specify whether to compress attribute fields and field data:

  • Attribute fields: By default, attribute fields are not compressed. If file_compressor is selected for an attribute field, the attribute field is compressed.

  • Field data: By default, field data is not compressed. For multi-value fields or fields of the STRING type, uniq is selected by default. For single-value fields, equal is selected by default.

Note
  • When you configure a vector index, you must configure the fields in the order of the primary key field, category field, and vector field. The category field is optional. The preceding figure provides an example.

  • If you compress attribute fields, we recommend that you modify the index loading method to reduce the impact on performance. To modify the index loading method, perform the following operations: On the instance details page of an instance, click Deployment Management. On the page that appears, click the Searcher worker that you want to manage. In the Searcher Worker Configurations panel, click the Online Table Configurations tab.

3. Configure the indexes. You must set the type of the primary key index to PRIMARYKEY64 and the type of the vector index to CUSTOMIZED.

image

Specify whether to compress index fields:

  • By default, index fields are not compressed. If file_compressor is selected for an index field, the index field is compressed.

Note
  • The primary key index cannot be compressed.

  • If you compress index fields, we recommend that you modify the index loading method to reduce the impact on performance. To modify the index loading method, perform the following operations: On the instance details page of an instance, click Deployment Management. On the page that appears, click the Searcher worker that you want to manage. In the Searcher Worker Configurations panel, click the Online Table Configurations tab.

3.1. Specify the fields contained in the vector index.

image

Note
  • The primary key field and vector field are required. The category field is optional and can be left empty.

  • You can select only the three fixed fields and cannot add new fields.

3.2. Configure advanced settings. You must configure the parameters for the vector index. The following figure shows an example. For more information, see Vector indexes.

image

The following figure shows more parameters.

image.png

The following sample code provides an example on how to configure the build_index_params parameter:

{
  "proxima.qc.builder.quantizer_class": "Int8QuantizerConverter",
  "proxima.qc.builder.quantize_by_centroid": true,
  "proxima.qc.builder.optimizer_class": "BruteForceBuilder",
  "proxima.qc.builder.thread_count": 10,
  "proxima.qc.builder.optimizer_params": {
    "proxima.linear.builder.column_major_order": true
  },
  "proxima.qc.builder.store_original_features": false,
  "proxima.qc.builder.train_sample_count": 3000000,
  "proxima.qc.builder.train_sample_ratio": 0.5
}

The following sample code provides an example on how to configure the search_index_params parameter:

{
  "proxima.qc.searcher.scan_ratio": 0.01
}

  1. After the configuration is complete, click Save Version. In the dialog box that appears, enter the description and click Publish. The description is optional.

image

After the index is published, click Next to rebuild the index.

image

  1. Rebuild the index. Configure the parameters based on your index rebuilding requirements and click Next.

  • API data source

image

  • MaxCompute data source

image

  1. In the left-side pane, choose O&M Center > Change History and click Data Source Changes. On the Data Source Changes tab, you can view the progress of reindexing. After the reindexing is complete, you can perform a query test.

image

  1. Perform tests on the query test page.

image

Syntax for queries

For more information, see Vector-based retrieval syntax.

Use an SDK to perform vector-based retrieval

Add the following dependency:

pip install alibabacloud-ha3engine

Search demo:

from alibabacloud_ha3engine import models, client
from alibabacloud_tea_util import models as util_models
from Tea.exceptions import TeaException, RetryError

def search():
    Config = models.Config(
        endpoint="",
        instance_id="",
        protocol="http",
        access_user_name="",
        access_pass_word=""
    )

    # If the request takes an extended period of time to complete, you can configure this parameter to increase the wait time for the request. Unit: millisecond.
    # This parameter can be used in the search_with_options method.
    runtime = util_models.RuntimeOptions(
        connect_timeout=5000,
        read_timeout=10000,
        autoretry=False,
        ignore_ssl=False,
        max_idle_conns=50
    )
    # Initialize the OpenSearch Retrieval Engine Edition V3.0 client.
    ha3EngineClient = client.Client(Config)
    optionsHeaders = {}
    try:
        # Example 3: Use an SQL query string in OpenSearch Retrieval Engine Edition to query data.
        # =====================================================
        sql_str = '''query=select proxima_score('index_name') as score,id from table_name where MATCHINDEX('index_name', ?) order by score asc limit 5&&kvpair=timeout:1000,iquan.plan.cache.enable:true;urlencode_data:false;iquan.plan.prepare.level:jni.post.optimize;dynamic_params:[["0.892704,0.783731&n=10"]]'''
        sqlsearchQuery = models.SearchQuery(sql=sql_str)
        sqlSearchRequestModel = models.SearchRequestModel(optionsHeaders, sqlsearchQuery)
        sqlstrSearchResponseModel = ha3EngineClient.search(sqlSearchRequestModel)
        print(sqlstrSearchResponseModel)

    except TeaException as e:
        print(f"send request with TeaException : {e}")
    except RetryError as e:
        print(f"send request with Connection Exception  : {e}")


if(__name__  == "__main__"):
    search()

Usage notes

  • If you want to minimize the time required for vector-based retrieval, we recommend that you lock vector indexes into the memory. For more information, see Lock policy for vector indexes.

  • On this page (1, O)
  • Purchase an OpenSearch Retrieval Engine Edition instance
  • Configure the instance
  • Syntax for queries
  • Use an SDK to perform vector-based retrieval
  • Usage notes
Feedback
phone Contact Us

Chat now with Alibaba Cloud Customer Service to assist you in finding the right products and services to meet your needs.

alicare alicarealicarealicare