Prerequisites
An Alibaba Cloud account is created and the real-name verification is complete.
When you create an Alibaba Cloud account and log on to the console for the first time, you are prompted to create an AccessKey pair before you perform subsequent operations.
You must create an AccessKey pair for your Alibaba Cloud account because the AccessKey pair is required to create and use an OpenSearch application.
After you create an AccessKey pair for your Alibaba Cloud account, you can create an AccessKey pair for a RAM user. This way, you can access applications as the RAM user. For more information about how to grant permissions to RAM users, see RAM authorization. Note: The RAM user must be granted the AliyunSearchEngineFullAccess or AliyunSearchEngineReadOnlyAccess permission to access OpenSearch Vector Search Edition instances.
3. A virtual private cloud (VPC) is created. For more information, see What is a VPC?
If you want to access OpenSearch Vector Search Edition instances as a RAM user, you must grant the AliyunSearchEngineFullAccess or AliyunSearchEngineReadOnlyAccess permission to the RAM user by using your Alibaba Cloud account.
Purchase an instance
Log on to the OpenSearch console. In the upper-left corner, switch to OpenSearch Vector Search Edition.
In the left-side navigation pane, click Instance Management. On the Instance Management page, click Create Instance.
On the buy page, select Vector Search Edition as the service edition and configure the Region and Zone, Query Node Quantity, Query Node Type, Data Node Quantity, Data Node Type, Total Storage Space of Single Searcher Worker, VPC, and vSwitch parameters for the instance. Then, specify a username and password and click Buy Now. The username and password are used for permission verification in queries and are not your Alibaba Cloud account and password.
Specify the numbers and specifications of Query Result Searcher (QRS) workers and Searcher workers that you want to purchase based on your business requirements. After you specify the specifications, the actual fee is automatically displayed on the buy page.
The VPC and vSwitch that you specify must be consistent with those configured for the Elastic Compute Service (ECS) instance that you use to access the OpenSearch Vector Search Edition instance. Otherwise, the error {'errors':{'code':'403','message':'Forbidden'}} is returned when you access the OpenSearch Vector Search Edition instance.
A free quota of storage space is provided for each Searcher worker. You can increase the quota in increments of 50 GB. If the total storage space exceeds the free quota, you are charged for the excess storage space.
On the Confirm Order page, check the configurations and the service agreement and click Activate Now.
After you purchase the instance, click Console. On the Instance Management page, you can view the purchased instance.
By default, the name of the instance is automatically set. To modify the name of the instance, click Manage in the Actions column to go to the details page of the instance.
Click the Modify icon, modify the instance name as prompted, and then click Confirm.
Configure a cluster
On the details page of the purchased instance, you can view that the instance is in the Pending Configuration state and an empty cluster is automatically deployed for the instance. The numbers and specifications of QRS workers and Searcher workers in the cluster are those you specify when you purchase the instance. You must configure a data source and an index schema and rebuild indexes for the cluster before you can use the search service. 
Configure the data source. Data from a MaxCompute data source or an API data source is supported. In this example, a MaxCompute data source is used. To configure a MaxCompute data source for the cluster, perform the following operations: In the Configure Data Source step, click Add Data Source. In the Add Data Source panel, select MaxCompute as Data Source Type. Configure the Project, AccessKey ID, AccessKey Secret, Table, and Partition Key parameters. Select Yes or No for Automatic Reindexing based on your business requirements.
After the verification is passed, click OK to add the data source.
After the data source is configured, click Next to configure the index schema.
2.1. Click Create Index Table.
2.2. Configure an index table. In the Select Template section, select Common Template.
Index Table: Enter a custom name.
Data Source: Select the data source that you configure in Step 1.
Data Shards: Enter a value based on the number of Searcher workers that you purchase.
2.3. Configure fields. You must specify at least two fields: a primary key field and a vector field. The vector field must be of the multi-value FLOAT data type.
If you want to configure a vector index with categories, you can add a category field between the primary key field and the vector field. The category field must be a single-value field or multi-value field of the INTEGER data type.
Specify whether to compress attribute fields and field data:
Attribute fields: By default, attribute fields are not compressed. If file_compressor is selected for an attribute field, the attribute field is compressed.
Field data: By default, field data is not compressed. For multi-value fields or fields of the STRING type, uniq is selected by default. For single-value fields, equal is selected by default.
When you configure a vector index, you must specify the fields in the order of the primary key field, category field, and vector field. The category field is optional. The preceding figure provides an example.
If you compress attribute fields, we recommend that you modify the index loading method to reduce the impact on performance. To modify the index loading method, perform the following operations: On the instance details page of an instance, click Deployment Management. On the page that appears, click the Searcher worker that you want to manage. In the Searcher Worker Configurations panel, click the Online Table Configurations tab.
Configure the indexes. You must set the type of the primary key index to PRIMARYKEY64 and set the type of the vector index to CUSTOMIZED.
Specify whether to compress index fields:
By default, index fields are not compressed. If file_compressor is selected for an index field, the index field is compressed.
The primary key index cannot be compressed.
If you compress index fields, we recommend that you modify the index loading method to reduce the impact on performance. To modify the index loading method, perform the following operations: On the instance details page of an instance, click Deployment Management. On the page that appears, click the Searcher worker that you want to manage. In the Searcher Worker Configurations panel, click the Online Table Configurations tab.
3.1. Specify the fields contained in the vector index.
The primary key field and the vector field are required. The category field is optional and can be left empty.
You can select only the three fixed fields and cannot add new fields.
3.2. Configure advanced settings. You must configure the parameters for the vector index. The following figure provides an example. For more information, see Vector indexes.
The following figure shows more parameters.
The following sample code provides an example on how to configure the build_index_params parameter:
{
"proxima.qc.builder.quantizer_class": "Int8QuantizerConverter",
"proxima.qc.builder.quantize_by_centroid": true,
"proxima.qc.builder.optimizer_class": "BruteForceBuilder",
"proxima.qc.builder.thread_count": 10,
"proxima.qc.builder.optimizer_params": {
"proxima.linear.builder.column_major_order": true
},
"proxima.qc.builder.store_original_features": false,
"proxima.qc.builder.train_sample_count": 3000000,
"proxima.qc.builder.train_sample_ratio": 0.5
}The following sample code provides an example on how to configure the search_index_params parameter:
{
"proxima.qc.searcher.scan_ratio": 0.01
}The system automatically configures the parameters for a vector index. If you have no special requirements, you can click OK to complete the configuration.
After the configuration is complete, click Save Version. In the dialog box that appears, enter the description and click Publish. The description is optional.
After the index is published, click Next to rebuild the index.
Rebuild the index. Configure the parameters based on your index rebuilding requirements and click Next.
API data source
MaxCompute data source
In the left-side navigation pane, choose O&M Center > Change History and click the Data Source Changes tab. On the Data Source Changes tab, you can view the progress of reindexing. After reindexing is complete, you can perform a query test.
Perform tests on the query test page.
Syntax for queries
Syntax for a regular query
HA3 syntax
query=index_name:'0.1,0.2,0.98,0.6;0.3,0.4,0.98,0.6...'
Note: The index_name parameter specifies the name of your vector index. Specify the vectors that you want to query after the colon (:).SQL syntax
query=select proxima_score('index_name') as score,id from table_name where MATCHINDEX('index_name', ?) order by score asc limit 5&&kvpair=timeout:1000,iquan.plan.cache.enable:true;urlencode_data:false;iquan.plan.prepare.level:jni.post.optimize;dynamic_params:[["0.892704,0.783731"]]
Note: The index_name parameter specifies the name of your vector index. The dynamic_params parameter in the kvpair clause specifies the vectors that you want to query.Syntax for a query for top n vectors
HA3 syntax
query=index_name:'0.1,0.2,0.98,0.6;0.3,0.4,0.98,0.6&n=10'
Note: The index_name parameter specifies the name of your vector index. Specify the vectors that you want to query after the colon (:) and before the ampersand (&). The n parameter specifies the top n vectors that can be returned.
SQL syntax
query=select proxima_score('index_name') as score,id from table_name where MATCHINDEX('index_name', ?) order by score asc limit 5&&kvpair=timeout:1000,iquan.plan.cache.enable:true;urlencode_data:false;iquan.plan.prepare.level:jni.post.optimize;dynamic_params:[["0.892704,0.783731&n=10"]]
Note: The index_name parameter specifies the name of your vector index. The dynamic_params parameter in the kvpair clause specifies the vectors that you want to query. The n parameter specifies the top n vectors that can be returned.
Syntax for a query that includes a specific threshold value
HA3 syntax
query=index_name:'0.1,0.2,0.98,0.6;0.3,0.4,0.98,0.6&n=10&sf=0.8'
Note: The index_name parameter specifies the name of your vector index. Specify the vectors that you want to query after the colon (:) and before the first ampersand (&). The sf parameter specifies the threshold value based on which the system filters documents.
SQL syntax
query=select proxima_score('index_name') as score,id from table_name where MATCHINDEX('index_name', ?) order by score asc limit 5&&kvpair=timeout:1000,iquan.plan.cache.enable:true;urlencode_data:false;iquan.plan.prepare.level:jni.post.optimize;dynamic_params:[["0.892704,0.783731&n=10&sf=0.8"]]
Note: The index_name parameter specifies the name of your vector index. The dynamic_params parameter in the kvpair clause specifies the vectors that you want to query. The sf parameter specifies the threshold value based on which the system filters documents.
Syntax for a query with retrieval parameters
HA3 syntax
query=index_name:'0.1,0.2,0.98,0.6;0.3,0.4,0.98,0.6&n=10&sf=0.8&search_params={"proxima.qc.searcher.scan_ratio":0.001,"proxima.general.searcher.scan_count":10000}'
Note: The search_params parameter specifies the parameters that you want to configure for vector retrieval. The value must be in the JSON format. For more information about the proxima.qc.searcher.scan_ratio parameter, see the "Parameter description" section in the "Vector indexes" topic. The proxima.general.searcher.scan_count parameter is equivalent to the min_scan_doc_cnt parameter.
Note: The order of the n, sf, and search_params parameters cannot be changed.SQL syntax
query=select proxima_score('index_name') as score,id from table_name where MATCHINDEX('index_name', ?) order by score asc limit 5&&kvpair=timeout:1000,iquan.plan.cache.enable:true;urlencode_data:false;iquan.plan.prepare.level:jni.post.optimize;dynamic_params:[["0.892704,0.783731&n=10&sf=0.8&search_params={"proxima.qc.searcher.scan_ratio":0.001,"proxima.general.searcher.scan_count":10000}"]]
Note: The index_name parameter specifies the name of your vector index. The dynamic_params parameter in the kvpair clause specifies the vectors that you want to query. The search_params parameter specifies the parameters that you want to configure for vector retrieval. The value must be in the JSON format. For more information about the proxima.qc.searcher.scan_ratio parameter, see the "Parameter description" section in the "Vector indexes" topic. The proxima.general.searcher.scan_count parameter is equivalent to the min_scan_doc_cnt parameter.
Note: The order of the n, sf, and search_params parameters cannot be changed.If a category field is configured in a vector index, you must include the category field in the query syntax. Otherwise, the data cannot be retrieved. Example:
query=select proxima_score('vector_index_name') as score, cate_id from table_name where MATCHINDEX('vector_index_name', ?) &&kvpair=timeout:1000,iquan.plan.cache.enable:true;urlencode_data:false;iquan.plan.prepare.level:jni.post.optimize;trace:INFO;formatType:json;dynamic_params:[["168#0.01747940666973591,-0.03125246614217758,-0.03254068270325661..."]]In the dynamic_params parameter, 168 indicates the value of the category field. Separate the value of the category field and the content after the value with a number sign (#).
Use an SDK to perform vector-based retrieval
Add the following dependency:
pip install alibabacloud-ha3engineSearch demo:
from alibabacloud_ha3engine import models, client
from alibabacloud_tea_util import models as util_models
from Tea.exceptions import TeaException, RetryError
def search():
Config = models.Config(
endpoint="",
instance_id="",
protocol="http",
access_user_name="",
access_pass_word=""
)
# If the request takes an extended period of time to complete, you can configure this parameter to increase the wait time for a request. Unit: milliseconds.
# This parameter can be used in the search_with_options method.
runtime = util_models.RuntimeOptions(
connect_timeout=5000,
read_timeout=10000,
autoretry=False,
ignore_ssl=False,
max_idle_conns=50
)
# Initialize the OpenSearch Vector Search Edition client.
ha3EngineClient = client.Client(Config)
optionsHeaders = {}
try:
# Example 3: Use an SQL query string in OpenSearch Vector Search Edition to query data.
# =====================================================
sql_str = '''query=select proxima_score('index_name') as score,id from table_name where MATCHINDEX('index_name', ?) order by score asc limit 5&&kvpair=timeout:1000,iquan.plan.cache.enable:true;urlencode_data:false;iquan.plan.prepare.level:jni.post.optimize;dynamic_params:[["0.892704,0.783731&n=10"]]'''
sqlsearchQuery = models.SearchQuery(sql=sql_str)
sqlSearchRequestModel = models.SearchRequestModel(optionsHeaders, sqlsearchQuery)
sqlstrSearchResponseModel = ha3EngineClient.search(sqlSearchRequestModel)
print(sqlstrSearchResponseModel)
except TeaException as e:
print(f"send request with TeaException : {e}")
except RetryError as e:
print(f"send request with Connection Exception : {e}")
if(__name__ == "__main__"):
search()