Background information
In retrieval-augmented generation (RAG) and semantic search scenarios, text inverted indexes and dense vectors are combined to efficiently perform text retrieval and similarity search. This way, the fast retrieval capability of inverted indexes is combined with the semantic representation capability of dense vectors.
Purchase an OpenSearch Vector Search Edition instance
For more information, see Purchase an OpenSearch Vector Search Edition instance.
Configure an instance
On the details page of the purchased instance, the instance is in the Pending Configuration state. The system automatically deploys an instance that contains no data. The number and specifications of Query Result Searcher (QRS) workers and Searcher workers are the same as those you purchase. Before you can use the instance for search, perform the following steps: configure a table, add a data source, configure fields, configure an index schema, and then perform reindexing for the instance.
1. Configure the basic information of a table
In the left-side pane on the instance details page, click Table Management. On the Table Management page, click Add Table. In the Basic Table Information step of the Create wizard, configure the Table Name, Data Shards, Number of Resources for Data Updates, and Scenario Template parameters.
Parameters:
Table Name: the name of the table. You can enter a custom table name.
Data Shards: the number of data shards contained in the table. Enter a positive integer in the range of 1 to 256. You can perform sharding to accelerate the full indexing and improve the performance of a single query. If you create multiple index tables in an existing OpenSearch instance, make sure that the index tables contain the same number of shards. Alternatively, make sure that at least one index table contains one shard and other index tables contain the same number of shards.
Number of Resources for Data Updates: the number of resources used for data updates. By default, a free quota of two resources for data updates is provided for each index. Each resource consists of 4 CPU cores and 8 GB of memory. You are charged for resources that exceed the free quota. For more information, see the "Data update resource fee" section of the Billing overview of OpenSearch Vector Search Edition topic.
2. Add a data source
In the Data Synchronization step, add a data source. You can add an Object Storage Service (OSS) data source, a MaxCompute data source, an API data source, or a Data Lake Formation (DLF) data source. In this example, OSS + API is selected for Full Data Source. Configure the OSS Path, Bucket, and Data Format parameters. Then, click Check to check the data source information. After the check is passed, click Next.
For more information about MaxCompute data sources, see Create a table for a MaxCompute data source.
For more information about API data sources, see Create a table for an API data source.
For more information about OSS data sources, see Create a table for an OSS data source.
For more information about DLF data sources, see Create a table for a DLF data source.
3. Configure fields
In the Field Configuration step, configure the fields. If you select Hybrid Search by Dense and Sparse Vectors in the Basic Table Information step, you must define id as the primary key field, vector as the dense vector field, and title as the text inverted index field.
Take note of the following information:
The primary key field and vector field are required. For the primary key field, you must set the Type parameter to INT or STRING and select the check box in the Primary Key column.
The vector field is required. You can specify multiple vectors as the vector fields. You must define the vector field as a multi-value field of the FLOAT type.
You must set the Type parameter to TEXT for the text inverted index field. An inverted index with the same name as the field will be automatically created.
4. Configure the index schema
Parameters:
Vector indexes
Index Name: This parameter is required. You can enter a custom index name.
Hybrid Search: After you enable Hybrid Search, you must configure the sparse vector subscript and sparse vector value fields.
Vector Dimension: the dimensions of the dense vector.
Distance Type: You can select InnerProduct (inner product distance, the higher the score, the more relevant the vector) or SquareEuclidean (Euclidean distance, the lower the score, the more relevant the vector).
Real-time Indexing: You can set this parameter to true or false.
Vector Index Algorithm: The Liner, HNSW, QGraph, and QC algorithms are supported. For more information about the supported algorithms, see Introduction to vectors.
You must separately configure parameters for the advanced configurations of the vector index. For more information, see Common configurations of vector indexes.
Other indexes
For fields whose Type is set to TEXT, a regular index will be automatically created.
Global index configuration
You can configure an automatic cleanup policy for expired documents.
5. Confirm the creation
In the Confirm step, click Confirm.
6. View the change history
In the left-side pane on the instance details page, click Change History. On the page that appears, you can view all finite-state machines (FSMs) related to the processes of creating a table, creating indexes, and performing reindexing for full data. After the search engine is built, you can perform query tests in the instance.