Tablestore:Vector search

Scenarios

Vector search is suitable for scenarios such as recommendation systems, image and video retrieval, and natural language processing and semantic search.

• Retrieval-augmented generation (RAG)

  RAG is an AI framework that combines retrieval capabilities with those of Large Language Models (LLMs). It uses retrieval to improve the accuracy of LLM outputs, especially for private or professional data. RAG is widely used in knowledge base scenarios.

• Recommendation system

  On platforms such as e-commerce, social media, and streaming media, you can encode user behaviors, preferences, and content features into vectors. Vector search can then quickly find products, articles, or videos that match user interests. This process enables personalized recommendations and improves user satisfaction and retention rates.

• Similarity detection (images, videos, and voice)

  In fields such as image, video, voice, voiceprint, and facial recognition, you can convert unstructured data into vector representations. Vector search can then be used to quickly find the most similar targets. For example, on an e-commerce platform, after a user uploads an image, the system can quickly find product images with similar styles, colors, or patterns.

• Natural language processing and semantic search

  In the natural language processing (NLP) field, you can convert text into vector representations, such as Word2Vec or Bidirectional Encoder Representations from Transformers (BERT) embeddings. Vector search can then be used to understand the semantics of a query and find the most semantically relevant documents, news, or Q&A pairs. This improves search result relevance and user experience.

• Knowledge graph and AI chat

  Knowledge graph nodes and relationships can be represented as vectors. Vector search can accelerate entity linking, relationship inference, and the response speed of AI chat systems. This allows the system to more accurately understand and answer complex questions.

Core advantages

Function overview

A KnnVectorQuery finds the most similar data items in a large-scale dataset by performing an approximate nearest neighbor query on numerical vectors.

Vector search inherits all the features of a search index. It is a ready-to-use, pay-as-you-go service that requires no system deployment. It supports stream-based index building, allowing data to be queried in near-real-time after it is written to a table. It also supports high-throughput additions, updates, and deletions. The query performance is comparable to systems that use the HNSW algorithm.

When you use the KnnVectorQuery feature to query data, you must specify the query vector, the vector field to search, and the number of nearest neighbors (TopK) to retrieve. This process retrieves the TopK vectors from the specified vector field that are most similar to your query vector. You can also combine this query with other non-vector search features to filter the results.

Notes

When you use vector search, note the following:

• Limits apply to the number of vector field types, dimensions, and other properties. For more information, see Search index limits.

• The search index is partitioned on the server-side. Each partition returns its own TopK nearest neighbors, and the results are then aggregated at the client node. Therefore, if you use a token to paginate through all data, the total number of rows returned depends on the number of server-side partitions.

• Currently, the vector search feature is available in the following regions: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Zhangjiakou), China (Ulanqab), China (Shenzhen), China (Guangzhou), China (Chengdu), China (Hong Kong), Japan (Tokyo), Singapore, Malaysia (Kuala Lumpur), Indonesia (Jakarta), Philippines (Manila), Thailand (Bangkok), Germany (Frankfurt), UK (London), US (Virginia), US (Silicon Valley), SAU (Riyadh - Partner Region).

Procedure

1. Generate vectors.

   Use open source models to convert data in Tablestore into vectors and store them.

2. Write vector data to Tablestore.

3. When you create a search index, configure the vector field.

   Configure the vector field's type, dimensions, and distance measure algorithm.

4. Use vector search to query data.

Billing

During the public preview, you are not charged for billable items specific to the KNN vector query feature. You are charged for other billable items based on existing billing rules.

When you use a search index to query data, you are charged for the read throughput consumed. For more information, see Billable items of search indexes.

Appendix 1: Use with BoolQuery

You can combine KnnVectorQuery and BoolQuery in different ways to achieve different results. This section describes two common use cases. This example assumes a scenario where a filter matches a small amount of data.

Assume a table contains 100 million images. A user has 50,000 images in total, but only 50 of them were added in the last 7 days. The user wants to use search by image to find the 10 most similar images from those added in the last 7 days. The following table shows how using BoolQuery inside the KnnVectorQuery filter can meet this requirement.

Appendix 2: Vector normalization example

The following code shows an example of vector normalization:

  public static float[] l2normalize(float[] v, boolean throwOnZero) {
    double squareSum = 0.0f;
    int dim = v.length;
    for (float x : v) {
      squareSum += x * x;
    }
    if (squareSum == 0) {
      if (throwOnZero) {
        throw new IllegalArgumentException("can't normalize a zero-length vector");
      } else {
        return v;
      }
    }
    double length = Math.sqrt(squareSum);
    for (int i = 0; i < dim; i++) {
      v[i] /= length;
    }
    return v;
  }