OpenSearch:Drop-down suggestions

The drop-down suggestion feature is a basic feature of OpenSearch. This feature provides recommended search queries for you to select when you enter a search query. You may find the expected search query before you enter the full content of the search query. This helps you obtain the expected search results as soon as possible.

OpenSearch intelligently extracts search queries from documents to implement the drop-down suggestion feature. When you enter a search query in Chinese, OpenSearch provides suggestions based on the first few characters in Chinese phrases, full pinyin spelling, pinyin acronyms, Chinese characters and pinyin, first few characters after analysis, and Chinese homophones.

For example, the search query 连衣裙长款 is recommended when you enter the following content:

• First few characters in Chinese phrases: 连 and 连衣

• First few letters in full pinyin spelling: l, li, lian, lianyi, and lianyiqun

• First few letters in pinyin acronyms: l, ly, and lyq

• Chinese characters and pinyin: 连yi, and 连衣qun

• First few characters after analysis: 长款, 长款连衣, and 连衣长

• Chinese homophones: 连衣群 and 联谊群

You can also use intervention entries to intervene in drop-down suggestions. OpenSearch provides multiple metrics for drop-down suggestions. For more information, see Drop-down suggestion report.

Data source

Documents from an application serve as the data source of drop-down suggestions. You can select a maximum of three fields from an application as the data source for each drop-down suggestion model. To provide drop-down suggestions, OpenSearch selects millions of documents from an application and processes the values of the selected fields in these documents based on specified rules to generate recommended search queries. Then, OpenSearch displays only a number of recommended search queries in drop-down suggestions based on specified rules.

Rules for generating recommended search queries

OpenSearch generates recommended search queries from historical search queries within the last N days based on specified conditions. For example, a condition can be the term weights, number of retrieved documents, or number of search times of a historical search query, or whether search results are returned for the historical search query within the last day. By default, historical search queries in the last seven days are used to generate recommended search queries. To generate recommended search queries, OpenSearch extracts terms from historical search queries or uses original historical search queries:

• Extract terms from historical search queries: OpenSearch uses an analyzer that is trained by the natural language processing (NLP) team of Alibaba Cloud to analyze search queries. This analyzer is developed based on a large amount of natural language data. After analysis, OpenSearch combines terms that have actual meanings to generate recommended search queries. This ensures that the required documents can be retrieved based on the recommended search queries.

• Use original historical search queries: OpenSearch selects original historical search queries without changes as recommended search queries in drop-down suggestions. If a historical search query contains more than 30 characters, OpenSearch truncates the search query by keeping only the first 30 characters and uses the truncated search query as the recommended search query. This method applies to application fields that require no analysis, such as the store name, username, and song name. This method also applies to custom search queries that you want to use as recommended search queries without changes. We recommend that you select short, concise, and explicit fields.

Note:
Recommended search queries are extracted from search requests that contain the raw_query parameter in the last N days (seven days by default).

Manual intervention

You can intervene in drop-down suggestions in the following ways:

• Intervene in recommended search queries by using whitelists and blacklists.

• Filter documents from the data source. After you configure filter conditions, OpenSearch generates recommended search queries based only on application documents that meet the conditions.

The following figure shows how to configure filter conditions in the OpenSearch console.

Intervene in recommended search queries

Blacklists: If a recommended search query contains a keyword in a blacklist, the recommended search query does not appear in drop-down suggestions. If a recommended search query is not applicable to your business, you can add related keywords to a blacklist. This way, all recommended search queries that contain the keywords do not appear in drop-down suggestions.

Whitelists: If a search query in a whitelist meets the specified recommendation conditions, the search query ranks top in drop-down suggestions. If a recommended search query that is applicable to your business ranks low or is not included in drop-down suggestions, you can add the search query to a whitelist. This improves the quality of drop-down suggestions. **For more information about how to configure whitelists and blacklists, see Blacklists and whitelists.

• To optimize drop-down suggestions, we recommend that you associate a drop-down suggestion request with a search request. This helps increase the number of recommended search queries and the click-through rate. For more information, see the "Associate a drop-down suggestion request with a search request" section of this topic.

• Select concise fields that are related to their document topics.

• Extract terms from historical search queries or use original historical search queries without changes based on your requirements.

• In the response to a search request based on drop-down suggestions, the suggestions parameter contains the recommended search queries. The errors parameter indicates whether an error occurs. The suggestions parameter may contain data even if the errors parameter is returned. If recommended search queries are returned in the suggestions parameter, these search queries appear in drop-down suggestions.

1. Log on to the OpenSearch console. In the left-side navigation pane, choose Search Algorithm Center > Search Guidance. On the Drop-down Suggestions page, click Create.

2. On the Create Drop-down Suggestions Model page, enter a model name, select training fields, select an extraction method for each field, turn on Historical Search Queries as needed, enter filter conditions as needed, and then click Submit.

3. On the Drop-down Suggestions page, find the created drop-down suggestion model and click Train in the Actions column to train the model.

4. Wait until the training is complete. The training will last for 20 to 30 minutes.

5. Test the drop-down suggestion model. The following figures show the recommended search queries when you select Retain Original Value or Extract from Original Value as the extraction method.

Retain Original Value

Extract from Original Value

6. Obtain recommended search queries in the OpenSearch console, as described in the demo in the following part of this topic. For more information about relevant API operations, see Query drop-down suggestions.

Drop-down Suggestions page

In the OpenSearch console, choose Search Algorithm Center > Search Guidance in the left-side navigation pane. Then, the Drop-down Suggestions page appears, as shown in the following figure.

On the Drop-down Suggestions page, you can view the name, creation time, status, and latest version status of each drop-down suggestion model. The latest version status includes Pending Training, Training, Trained and Passed, and Abnormal Data. In the Actions column, you can view the details of a model, train a model, and delete a model.

Details page of a drop-down suggestion model

Basic Information: In this section, you can view the creation time, status, latest training time, and latest version status of the drop-down suggestion model. If the model is in the Abnormal Data state, you can view the exception report on this page.

Configuration Information: In this section, you can view the training fields, filter conditions, whitelists, and blacklists configured for the drop-down suggestion model, and whether the scheduled training and historical search query features are enabled.

Data Verification: In this section, you can view the data integrity and the integrity level of the model.

Training History: In this section, you can view the training records of the model.

Core metrics data

OpenSearch displays the key metrics of drop-down suggestion models in a table and a line chart. You can filter the metrics by time range.

Note: For more information about the metrics, see Drop-down suggestion report.

API request:

GET v3/openapi/suggestions/{suggestion_name}/actions/search?hit=10&query={your_query}&re_search=homonym&user_id=xxx

Maven dependencies for the SDK for Java:

<dependency>
    <groupId>com.aliyun.opensearch</groupId>
    <artifactId>aliyun-sdk-opensearch</artifactId>
    <version>4.0.0</version>
</dependency>

For more information, see Release notes.

Demo code:

package com.example.opensearch;

import com.aliyun.opensearch.OpenSearchClient;
import com.aliyun.opensearch.SuggestionClient;
import com.aliyun.opensearch.sdk.generated.OpenSearch;
import com.aliyun.opensearch.sdk.generated.commons.OpenSearchClientException;
import com.aliyun.opensearch.sdk.generated.commons.OpenSearchException;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import java.nio.charset.Charset;

public class SuggestDemo {
    static private final String accesskey = "AccessKey ID of your Alibaba Cloud account";
    static private final String secret = "AccessKey secret of your Alibaba Cloud account";
    static private final String host = "Endpoint of OpenSearch API in the region of the application for which you want to call the drop-down suggestion model";
    OpenSearch openSearch;
    OpenSearchClient openSearchClient;
    static private final byte hits = 8; // The maximum number of recommended search queries in drop-down suggestions.
    static private final String suggestionName = "Name of the drop-down suggestion model"; // The name of the drop-down suggestion model.

    @Before
    public void setUp() {
        // Create an OpenSearch object.
        openSearch = new OpenSearch(accesskey, secret, host);
        openSearchClient = new OpenSearchClient(openSearch);
    }

    @Test
    public void TestEnv() {
        // Obtain the file encoding format and default encoding format.
        System.out.println(String.format("file.encoding: %s", System.getProperty("file.encoding")));
        System.out.println(String.format("defaultCharset: %s", Charset.defaultCharset().name()));

        // Create a drop-down suggestion object.
        SuggestionClient suggestionClient = new SuggestionClient(suggestionName, openSearchClient);
        String query = "Search query";
try {
            SuggestParams suggestParams = new SuggestParams();
            suggestParams.setQuery(query); // The search query.
            suggestParams.setHits(10); // The maximum number of recommended search queries in drop-down suggestions.
            suggestParams.setUserId("12345678"); // The user ID.
            // By default, OpenSearch retrieves documents by replacing terms in search queries with Chinese homophones. You can configure this feature by setting the re_search parameter in the request.
            // If you set this parameter to ReSearch.findByValue(1), this feature is disabled. If you set this parameter to ReSearch.findByValue(0) or do not specify this parameter, this feature is disabled.
            suggestParams.setReSearch(ReSearch.findByValue(1));
            SearchResult result = suggestionClient.execute(suggestParams); 
            System.out.println(result); // Display the retrieval results.
        } catch (OpenSearchException e) {
            e.printStackTrace();
        } catch (OpenSearchClientException e) {
            e.printStackTrace();
        }
    }

    @After
    public void clean() {
        openSearch.clear();
    }
}

For more information about how to use the SDK for Java to implement drop-down suggestions, see Demo code for calling the drop-down suggestion model.

Sample retrieved data:

{
  "request_id": "159851481919726888064081",
  "searchtime": 0.006246,
  "suggestions": [
    {
      "suggestion": "skirt fashion"
    },
    {
      "suggestion": "skirt shorty dress"
    },
    {
      "suggestion": "skirt dotted dress"
    },
    {
      "suggestion": "skirt young"
    },
    {
      "suggestion": "skirt dotted"
    },
    {
      "suggestion": "skirt shorty"
    },
    {
      "suggestion": "skirt shorty dotted"
    }
  ]
}

Note: The request ID that is returned with the retrieved data can be used to associate search requests.

The association of a drop-down suggestion request and a search request has the following benefits:

1. You can obtain the data of metrics to evaluate how much the drop-down suggestion feature optimizes search results. For example, you can obtain the data of metrics, such as the page views (PVs) and click-through rate of search results that are obtained based on drop-down suggestions, and the ratio of search engine results pages (SERPs) that are returned with few or zero results. For more information, see Drop-down suggestion report.

2. After you associate a drop-down suggestion request with a search request, you can obtain the data of metrics such as the click data. You can optimize the relevant drop-down suggestion model based on such data to improve the search performance.

   Association method:

   If a search request of an OpenSearch application is based on drop-down suggestions, you must include from_request_id={from_request_id} in the request. The from_request_id variable specifies the source of the current search query. If the search query is recommended by the drop-down suggestion, top search, or hint feature, set the from_request_id parameter to the ID of the feature request. After you associate the feature request with the current search request, you can obtain the metrics of the feature to evaluate and improve the search performance. For more information about the from_request_id parameter, see Initiate search requests.

Example:

The request ID that is returned by a drop-down suggestion operation is 159851481919726888064081. The following code provides an example on how to associate the request ID with a search request:

SearchParams searchParams = new SearchParams(config);
searchParams.setQuery("title:'skirt shorty'"); // The search query is recommended by the drop-down suggestion feature.

// Add the from_request_id parameter.
Map<String, String> customParam =new HashMap<>();
customParam.put("from_request_id","159851481919726888064081");
searchParams.setCustomParam(customParam);

// Run the query and return the results in the form of a SearchResult object.
SearchResult execute = searcherClient.execute(searchParams);
// Convert the SearchResult object into a string and return the string.
String result = execute.getResult();
System.out.println(result);