Faiss-Server - - Alibaba Cloud Documentation Center

Supported formats of vector files

File name extensions are used to identify file formats. The following file formats are supported:

.fvecs

.svm (Libsvm file)

Example of a Libsvm file:

21187279 1:0.2663046344870018 2:0.36652042181588923 3:1.0686633708024278 4:0.48038935355720136 5:0.25852895390543884 6:0.3548201486996048 7:0.5256999599627583 8:0.6950687558969181 9:0.678305894615746 10:0.22776047265501018
21264640 1:0.3939700179792862 2:0.2754414668803289 3:0.9439534329739017 4:0.40100161008614665 5:0.4995636031244379 6:0.013824724791385053 7:0.5587276328436032 8:0.5785080421720411 9:0.2784678162956546 10:0.5387681136371216

The content in the file must be in the format of tagid 1:xx 2:xx 3:xx. tagid can be any character string. In most cases, tagid is an item ID. The digit on the left of each colon (:) is a vector dimension number. In most cases, the digit starts from 1. xx indicates the value of a dimension.

.index (index file generated by Faiss)
If you directly load a Libsvm file that contains a large amount of data, it requires a long period of time to build an index. You can first generate an index file based on the Libsvm file in offline mode. Then, use Faiss-Server to load the index file. This way, you can build an index in a short period of time.

Note When you name a file, make sure that the file format and file name extension match. In some scenarios, when an Object Storage Service (OSS) file is generated, OSS adds random characters to the file name extension. In this case, you must change the file name extension to .index before you load the file.

Load a local vector file

Sample file: article_word2vec.svm

Sample code used to load the file:

docker run -it -p  9000:9000  -v /home/bruceding.jing:/index datascience-registry.cn-beijing.cr.aliyuncs.com/tools/faiss-server:1.0.0 --index_source=/index/article_word2vec.svm --index_params=Flat

The output shows that the gRPC server is started and listens on port 9000.

The following table describes the parameters in the preceding sample code. You can configure the parameters based on your business requirements.


Parameter	Description
-p	The `-p` parameter of Docker is used for port mapping. In this example, port 9000 is used. It is the default listening port of Faiss-Server.
/home/bruceding.jing	The path to which the Docker image address is mapped. Configure this parameter based on your business requirements.
datascience-registry.cn-beijing.cr.aliyuncs.com/tools/faiss-server:1.0.0	The Docker image address provided by Faiss-Server.
--index_source	The path of the vector file.
--index_params	The mode in which a Faiss index is built. In this example, the index is built in `Flat` mode. Faiss-Server uses the index_factory function of Faiss to build indexes. For more information about the `--index_params` parameter, see Faiss indexes.

Load a vector file in OSS

In most cases, a vector file is generated in OSS. Faiss-Server allows you to build an index based on a vector file in OSS.

docker run -it -p 9000:9000 datascience-registry.cn-beijing.cr.aliyuncs.com/tools/faiss-server:1.0.0 --index_params=IVF256,Flat --search_params=nprobe=128 --accessid=xxx --accesskey=xxx --endpoint=oss-cn-beijing.aliyuncs.com --bucket_name=faiss-server --object_name=demo_data/article_word2vec.svm

The following table describes the parameters in the preceding sample code.


Parameter	Description
--index_params	`IVF256,Flat` indicates that a clustered index is built, and the indexed data is divided into 256 pieces.
--search_params	In this example, Faiss-Server searches for the first 128 clusters whose centroids are closest to the vector. For more information about this parameter, see Index IO, cloning and hyper parameter tuning.
--bucket_name	The name of the OSS bucket.
--object_name	The path of the vector file in OSS.
--endpoint	The endpoint of OSS. In this example, a public endpoint is specified. If a VPC is used, you must specify the internal endpoint of OSS to minimize traffic fees and accelerate the data read speed.

Note To read data from OSS, you must specify the AccessKey ID and AccessKey secret of your Alibaba Cloud account.

You can run the following command to view the parameters supported by Faiss-Server:

docker run datascience-registry.cn-beijing.cr.aliyuncs.com/tools/faiss-server:1.0.0 --help

Load vector files in OSS based on the version file

Faiss-Server is an online service. It may require a long period of time to directly load a vector file to Faiss-Server. To save time, you can generate an index file based on the vector file and load the index file to Faiss-Server.

In some scenarios, vector files need to be loaded on a regular basis.

A vector file may have multiple versions, and its latest version is automatically recorded in a version file. Faiss-Server checks the content of the version file in an asynchronous manner. If the version file is updated, Faiss-Server automatically loads the latest index and idxmap files.

Procedure:

1. Use a vector recall algorithm to generate a vector file for an item and store the file in an OSS bucket.
2 and 3. Faiss-Server reads the vector file from OSS and builds an index.
The version file contains the locations of the index and idxmap files.
- The index file is the generated Faiss index file.
- The idxmap file is a mapping table that maps tag IDs (tagid) to specific vector locations in an index.
4. Faiss-Server is always in the started state and listens to the version file.

The following procedure describes how to build an index file in offline mode:

Generate an index file.
Run the following command to build an index. After the index is built, the process exits. The gRPC service is not involved in this operation. You do not need to configure the -p parameter of Docker in the command.
```
docker run -it datascience-registry.cn-beijing.cr.aliyuncs.com/tools/faiss-server:1.0.0 --index_params=IVF256,Flat --search_params=nprobe=128 --accessid=xxx --accesskey=xxxx --endpoint=oss-cn-beijing.aliyuncs.com --bucket_name=faiss-server --object_name=demo_data/article_word2vec.svm --version_name=demo_data/versions --action=build_index
```
Parameters in the command:
- --version_name: specifies the location of the version file in OSS. If the version file does not exist, the system automatically generates a version file.
- --action: Set the value to build_index.
After the code is successfully executed, three files are generated.
If the Libsvm vector file is stored on your computer, run the following command to upload the generated files into OSS:
```
docker run -it -v /home/bruceding.jing:/index  datascience-registry.cn-beijing.cr.aliyuncs.com/tools/faiss-server:1.0.0 --index_params=IVF256,Flat --search_params=nprobe=128 --accessid=xxx --accesskey=xxx --endpoint=oss-cn-beijing.aliyuncs.com --bucket_name=faiss-server --index_source=/index/article_word2vec.svm --version_name=demo_data/versions --action=build_index
```
Note The -v parameter in the command is used to transfer the Libsvm file to Docker.
Start Faiss-Server.
Open a new terminal window and run the following command to start Faiss-Server:
```
docker run -it -p 9000:9000 datascience-registry.cn-beijing.cr.aliyuncs.com/tools/faiss-server:1.0.0 --index_params=IVF256,Flat --search_params=nprobe=128 --accessid=xxxx --accesskey=xxxx --endpoint=oss-cn-beijing.aliyuncs.com --bucket_name=faiss-server --version_name=demo_data/versions --check_version=true
```
Parameters in the command:
- --index_params and --search_params: Set these parameters to the same values as those specified when you build the index.
- --version_name: specifies the location of the version file in OSS. If the version file does not exist, the system automatically generates a version file.
- --check_version: If you set this parameter to true, the version check feature is enabled. Only after you enable the version check feature, Faiss-Server automatically loads the latest index when the version file is updated.
After Faiss-Server is started, go back to the previous terminal window and build a new index. If information similar to the following output is displayed, the new index is loaded.
```
2020-05-01 17:09:37,478 INFO src/faiss_index.cc:197 start to load
2020-05-01 17:09:37,567 INFO src/oss_util.cc:30 GetObjectToFile success62
2020-05-01 17:09:37,568 INFO src/faiss_index.cc:219 localPath=/tmp/20200501170924.index
2020-05-01 17:09:37,703 INFO src/oss_util.cc:30 GetObjectToFile success535963
2020-05-01 17:09:37,815 INFO src/oss_util.cc:30 GetObjectToFile success98159
2020-05-01 17:09:37,817 INFO src/faiss_index.cc:245 load index success
```

Append a vector file.

After an entire index file is loaded, you can run the following command to append another Libsvm file to Faiss-Server:

docker run -it -p 9000:9000 datascience-registry.cn-beijing.cr.aliyuncs.com/tools/faiss-server:1.0.0 --index_params=IVF256,Flat --search_params=nprobe=128 --accessid=xxx --accesskey=xxxx --endpoint=oss-cn-beijing.aliyuncs.com --bucket_name=faiss-server --version_name=demo_data/versions --check_version=true --append_dir=demo_data/append

--append_dir: specifies the directory where the Libsvm file that you want to append is stored.

If you have placed the Libsvm file in this directory, information similar to the following output is displayed:

2020-05-01 17:24:53,161 INFO src/oss_util.cc:30 GetObjectToFile success2428471
2020-05-01 17:24:53,357 INFO src/faiss_index.cc:383 append index, file:article_word2vec.svm, size=10907, idx size=10907

Test Faiss-Server

After you start Faiss-Server, use one of the following tools to test the vector similarity search feature:

faiss-client-go.tar.gz and faiss-client-java.tar.gz

./faiss-client-go --host "11.158.**.**:9000" --vector "1:0.8254435895816826 2:0.3546776922906237 3:0.14982954432756015 4:0.4796270382425792 5:0.5478646494350313 6:0.3771617042371068 7:0.5107318036421319 8:0.7005006312566686 9:0.7030542773195687 10:0.692132791047145" --k 20

The following table describes the parameters in the preceding sample code.


Parameter	Description
--host	Specifies the IP address and port number of Faiss-Server.
--vector	Specifies the requested vector data that is in the same format as the Libsvm file. The dimensions must also be the same as the dimensions in the Libsvm file. In this example, 10-dimensional data is used.
--k	Specifies the number of the search results to be returned. The default value is 10. In this example, this parameter is set to 20.

The following output is returned: Output

Note tagid list: the tag IDs in the Libsvm file. In most cases, tag IDs are item IDs.

Deploy Faiss-Server in EMR

Create an E-MapReduce (EMR) Data Science cluster. For more information, see Overview.
By default, the Faiss-Server service is supported.

Note The datascience-registry.cn-beijing.cr.aliyuncs.com/tools/faiss-server:1.0.0 image is integrated into EMR. The image name is faiss-server:1.0.0.

Create a Shell job to load a vector file and start the gRPC service.

sudo docker run -d --restart unless-stopped -p 9001:9000 faiss-server:1.0.0 --index_params=IVF256,Flat --search_params=nprobe=128 --accessid=xxxx --accesskey=xxx --endpoint=oss-cn-huhehaote-internal.aliyuncs.com --bucket_name=faiss-test --object_name=article_word2vec.svm