Overview
OpenSearch Vector Search Edition is a large-scale distributed search engine that is developed by Alibaba Group. OpenSearch Vector Search Edition provides search services for the entire Alibaba Group, including Taobao, Tmall, Cainiao, Youku, and other e-commerce platforms that are provided for customers in regions outside the Chinese mainland. OpenSearch Vector Search Edition is also a base engine of Alibaba Cloud OpenSearch. After years of development, OpenSearch Vector Search Edition has met the business requirements for high availability, high timeliness, and cost-effectiveness. OpenSearch Vector Search Edition also provides an automated O&M system on which you can build a custom search service based on your business features.
Architecture
OpenSearch Vector Search Edition consists of the following three main components: a management system, an online search system, and data sources. The online search system loads indexes and provides vector search services. Data sources are configured to import full data and synchronize incremental data in real time. The management system provides automated O&M services that allow you to create and manage clusters with ease.
System architecture
Online search system
For users, the online search system is used based on the table dimension. You can separately query the data of each vector table by calling an API operation. You can also separately manage each table, such as configuring fields, configuring vector indexes, configuring a data source, and controlling the real-time data concurrency of each table.
The backend architecture of the online search system is a distributed architecture that consists of Query Result Searcher (QRS) workers and Searcher workers.
QRS workers parse, verify, or rewrite the query requests that you initiate, forward the parsed requests to Searcher workers for execution, and then collect and merge the results returned by Searcher workers. Then, QRS workers return the processed query results to you. QRS workers are compute nodes that do not load user data, and therefore do not occupy a large amount of memory. The memory usage of QRS workers is high only when a large number of documents are returned or a large number of statistical entries are generated. If the processing capacity of QRS workers reaches a bottleneck, you can increase the number of QRS workers or upgrade the specifications of QRS workers.
Searcher workers load the index data, retrieve documents based on queries, and filter, collect, and sort the documents. The indexes on Searcher workers can be sharded. You can split indexes into multiple shards and hash the fields in the shards to values in the range of [0,65535]. You can specify the number of shards when you create indexes. If a cluster contains a large amount of data or requires high query performance, you can perform sharding to improve the performance of a single query. If you want to improve the query performance of a cluster, for example, increase the queries per second (QPS) from 1,000 to 10,000, you can increase the number of replicas. In this case, you must split data on all Searcher workers into the specified number of shards. The hash values of the shards must constitute the complete range of [0,65535].
The resources of Searcher workers, such as CPU, memory, and disks, are shared among tables.
The preceding figure shows an example. Vector Table 1 contains four shards and Vector Table 2 contains one shard. Vector Table 1 evenly distributes data to each shard based on the generated indexes. Vector Table 2 stores full vector data on each Searcher worker as a broadcast table.
Data sources
You can configure a data source to build an index for the raw data by performing reindexing (full indexing), and import the index data after reindexing into a table in the online search system for data retrieval. A data source corresponds to a table. OpenSearch Vector Search Edition supports the following types of data sources: MaxCompute, Search, Advertisement, and Recommendation Offline (SARO), and API. The raw data of an API data source is empty. If you use an API data source, you must push all data to the corresponding table in the online search system by calling an API operation.
Management system
The management system is an O&M platform for OpenSearch Vector Search Edition instances. This platform helps you significantly reduce O&M costs.
Data synchronization process
Full indexing
OpenSearch Vector Search Edition supports indexes of multiple versions. Each index version is built based on a copy of raw data. By default, the raw data of an API data source is empty. Each time reindexing is triggered, full indexing is performed. Full indexing is not a longtime job. The process ends when data is processed and a full index is generated. Then, the full index is applied to online clusters for data retrieval.
Indexes of multiple versions ensure data integrity despite multiple data changes. When the index schema or the data structure changes, the index of a new version is isolated from the indexes of earlier versions. In this case, if an error occurs on data changes, you can roll back the data to an earlier version.
Several steps are required to build full indexes, such as data processing, index building, and index merging. You can set the concurrency of index processing in each step to accelerate full indexing.
Real-time incremental indexing
After the full index is built, each index version has a longtime incremental indexing process. The incremental data is synchronized by using data update nodes.
You can push incremental data from MaxCompute, Object Storage Service (OSS), and API data sources to tables by calling API operations. Data update nodes consume the incremental data. Then, Searcher workers build indexes in real time for data retrieval.
Incremental indexing is a longtime job. Each full indexing of a table corresponds to an incremental indexing process. You can adjust the number of data update nodes to improve the capability of processing real-time data.
Benefits of OpenSearch Vector Search Edition
Stability
The underlying layer of OpenSearch Vector Search Edition is developed by using the C++ programming language. After more than ten years of development, OpenSearch Vector Search Edition provides stable search services for various core business systems. OpenSearch Vector Search Edition is suitable for core search scenarios that require high stability.
Efficiency
OpenSearch Vector Search Edition is a distributed search engine that allows you to retrieve large amounts of data. OpenSearch Vector Search Edition supports real-time data updates. Data updates can take effect within seconds. Therefore, OpenSearch Vector Search Edition is suitable for query and search scenarios that are time-sensitive.
Cost-effectiveness
OpenSearch Vector Search Edition supports multiple policies for index compression and multi-value index loading tests and can meet query requirements in a cost-effective way.