OpenSearch:Intervention dictionaries for spelling correction

An intervention dictionary is a collection of custom intervention entries. These entries are used to correct spelling errors that OpenSearch cannot identify in search queries. After you create an intervention dictionary for spelling correction, you can select the dictionary when you create or modify a query analysis rule. This way, you can intervene in the spelling correction of search queries.

OpenSearch is designed with a built-in dictionary to implement automatic spelling correction. You can perform the following steps to intervene in spelling correction:

1. Create an intervention dictionary for spelling correction. To create an intervention dictionary, log on to the OpenSearch console. In the left-side navigation pane, choose Search Algorithm Center > Retrieval Configuration. On the Basic Configuration page, click Dictionary Management in the left-side pane. On the Dictionary Management page, click Create in the upper-right corner. In the Create Query Analysis Dictionary panel, enter a name for the intervention dictionary, select a dictionary type, and then click Save. After the intervention dictionary is created, it appears in the dictionary list.

2. Add and manage intervention entries in the intervention dictionary. Find the created dictionary in the dictionary list and click Manage Entries in the Actions column to go to the Manage Entries page. On this page, add and manage intervention entries as needed. You can intervene in synonym configuration in the following ways:

   1. Add a correcting query and enable it for the spelling correction of a search query. After you perform this operation, OpenSearch corrects the search query and retrieves documents based on the correcting query.

   2. Add a correcting query and block it for the spelling correction of a search query. After you perform this operation, OpenSearch no longer corrects the search query or retrieves documents based on the correcting query.

3. Use the intervention dictionary. After you add intervention entries to the intervention dictionary, you can associate the intervention dictionary with a query analysis rule of an application as needed.

4. Test and publish the intervention dictionary. After the intervention dictionary is associated with a query analysis rule, we recommend that you perform a search test before you apply the rule to online environments. This ensures expected search performance.

1. The intervention entry that is added earlier takes precedence.

For example, a custom search query is specified as china constructing back, and the following intervention entries are added: china constructing -> china construction and construction back -> construction bank. In this case, the predefined search queries in the two intervention entries can both be matched by consecutive semantic terms after analysis. However, an overlap exists between the predefined search queries in the two intervention entries and only the intervention entry that is added earlier takes effect. Therefore, the result of spelling correction is china construction back.

2. For intervention entries whose predefined search queries share the same beginning, the intervention entry that has a longer predefined search query takes precedence. The length of a predefined search query indicates the number of semantic terms after analysis.

For example, a custom search query is specified as opansearching is excellent, and the following intervention entries are added: opan -> open and opansearching -> opensearch. In this case, the result of spelling correction is opensearch is excellent.

3. Multiple intervention entries can take effect on a single custom search query in inclusive match mode.

Usage notes

• A spelling correction operation with human interventions has high credibility.

• OpenSearch uses the built-in dictionary for spelling correction together with the intervention entries that you add. For a custom search query, after its spelling is corrected based on intervention entries, OpenSearch retains the changes and processes the unchanged part by using the built-in dictionary.

Sample intervention entry:

mobile phone changer -> mobile phone charger

Sample custom search query:

which mobile phone changer brad is good

Result of spelling correction: The custom search query is first changed to which mobile phone charger brad is good after spelling correction based on the intervention entry mobile phone changer -> mobile phone charger. Then, OpenSearch retains this change and processes the unchanged part of the search query by using the built-in dictionary for spelling correction. The final result is which mobile phone charger brand is good.

Scenario: You have created query analysis rules with spelling correction capabilities for the OpenSearch application of your e-commerce shopping guide service. After you apply these rules to the online application, invalid spelling correction results exist. Therefore, intervention in spelling correction is implemented to resolve this issue.

Invalid spelling correction results: A customer enters the search query "manhatan" and only a few results that contain the word "manhattan" are retrieved. Most results that contain the word "manhatan" fail to be retrieved.

Issue description: The search query "manhatan" entered by the customer has a spelling error, and the built-in dictionary for spelling correction fails to identify the spelling error. The correct spelling is "manhattan".

Solution: Create an intervention dictionary for spelling correction and add the intervention entry "manhatan -> manhattan" for the search query "manhatan". Then, associate the intervention dictionary with a query analysis rule of the online application.

Procedure:

1. Log on to the OpenSearch console. In the left-side navigation pane, choose Search Algorithm Center > Retrieval Configuration. On the Basic Configuration page, click Dictionary Management in the left-side pane. On the Dictionary Management page, click Create in the upper-right corner to create an intervention dictionary.

In the Create Query Analysis Dictionary panel, specify a name for the intervention dictionary and set the Dictionary Type parameter to Spelling Correction.

2. Find the created intervention dictionary in the dictionary list and click Manage Entries in the Actions column. On the page that appears, click Add Intervention Entry. In the Add Intervention Entry panel, enter manhatan in the Search Query field and manhattan in the Corrected Word field, set the Intervention Type parameter to Add, and then click Save.

3. Go to the Query Analysis Rule Configuration page and click Create in the upper-right corner. In the Create Rule panel, associate the created intervention dictionary for spelling correction with the query analysis rule.

4. Perform a search test. OpenSearch returns a large number of results that contain the search query "manhattan" after spelling correction. The search performance is as expected.

• You cannot change the name and type of an intervention dictionary after it is created.

• You must specify different search queries for different intervention entries.

• An intervention dictionary can be associated with multiple query analysis rules.

• OpenSearch uses the built-in dictionaries together with the intervention entries that you add. The built-in dictionary is automatically selected for intervention.

• The intervention dictionary for spelling correction takes effect only when one or more (up to five) consecutive and complete semantic terms in a query match the query in the intervention entry. This is called the inclusive match mode. The inclusive match mode is not equivalent to the substring match mode of strings.

• If an intervention dictionary is associated with a query analysis rule, you cannot delete the dictionary regardless of whether the rule is applied to an online or offline application. You must first disassociate the dictionary from the rule.

• You can create a maximum of 20 intervention dictionaries for spelling correction.

• You can specify only one search query and one correcting query for an intervention entry.

• You can add a maximum of 1,000 intervention entries to an intervention dictionary for spelling correction.

• An intervention entry takes effect when one of the terms in the custom search query matches the predefined search query in the entry. For example, you create an intervention entry "changer -> charger". If you specify a custom search query "mobile phone changer", OpenSearch corrects the spelling based on the intervention entry "changer -> charger". In this case, OpenSearch does not need to use an intervention entry for the entire search query "mobile phone changer".

• OpenSearch normalizes the content of intervention entries. All uppercase letters are converted into lowercase letters, and all full-width characters are converted into half-width characters.