AnalyticDB:Create a document collection

Sample code

def create_document_collection(account,
                               account_password,
                               namespace,
                               collection,
                               metadata: str = None,
                               full_text_retrieval_fields: str = None,
                               parser: str = None,
                               embedding_model: str = None,
                               metrics: str = None,
                               hnsw_m: int = None,
                               pq_enable: int = None,
                               external_storage: int = None,):
    request = gpdb_20160503_models.CreateDocumentCollectionRequest(
        region_id=ADBPG_INSTANCE_REGION,
        dbinstance_id=ADBPG_INSTANCE_ID,
        manager_account=account,
        manager_account_password=account_password,
        namespace=namespace,
        collection=collection,
        metadata=metadata,
        full_text_retrieval_fields=full_text_retrieval_fields,
        parser=parser,
        embedding_model=embedding_model,
        metrics=metrics,
        hnsw_m=hnsw_m,
        pq_enable=pq_enable,
        external_storage=external_storage
    )
    response = get_client().create_document_collection(request)
    print(f"create_document_collection response code: {response.status_code}, body:{response.body}")


if __name__ == '__main__':
    metadata = '{"title":"text", "page":"int"}'
    full_text_retrieval_fields = "title"
    embedding_model = "m3e-small"
    create_document_collection("testacc", "Test1234", "ns1", "dc1", 
                               metadata=metadata, full_text_retrieval_fields=full_text_retrieval_fields, 
                               embedding_model=embedding_model)


# output: body:
# {
#    "Message":"success",
#    "RequestId":"7BC35B66-5F49-1E79-A153-8D26576C4A3E",
#    "Status":"success"
# }

You can invoke the create_document_collection() function to create a document collection in an AnalyticDB for PostgreSQL instance. The parameters for create_document_collection are described below:

• account: the privileged account of the AnalyticDB for PostgreSQL instance.

• account_password: the password of the privileged account.

• namespace: the name of the namespace in which you want to create the document collection.

• collection: the name of the document collection that you want to create.

• metadata: the metadata of the custom map structure which contains multiple fields represented by key-value pairs. The key specifies the field name and the value specifies the field type.

• full_text_retrieval_fields: the custom full-text search fields separated by commas (,). The fields must be the keys of metadata.

• parser: the tokenizer. This parameter is a full-text search parameter. Default value: zh_cn. For more information, see Full-text search.

• embedding_model: the embedding model. For more information, see Embedding models.

• metrics: the distance or similarity metric algorithm. This parameter is a vector index parameter. For more information, see Vector indexes.

• hnsw_m: the maximum number of neighbors for the Hierarchical Navigable Small World (HNSW) algorithm. This parameter is a vector index parameter. Valid values: 1 to 1000. For more information, see Vector indexes.

• pq_enable: specifies whether to enable the product quantization (PQ) feature for index acceleration and dimensionality reduction. This parameter is a vector index parameter. For more information, see Vector indexes.

• external_storage: specifies whether to use the memory mapping technology to create HNSW indexes. This parameter is a vector index parameter. For more information, see Vector indexes.

  Important

  Only AnalyticDB for PostgreSQL V6.0 supports the external_storage parameter.

View database changes

After the code invocation is successful, you can log on to the DMS console to view the schema of the created table named ns1.dc1.

Full-text search

To improve search accuracy, AnalyticDB for PostgreSQL supports full-text search in addition to vector similarity search. Full-text search can be used together with vector similarity search to implement two-way retrieval.

Embedding models

AnalyticDB for PostgreSQL supports the following embedding models:

Vector indexes

The following table describes the parameters for vector indexes: