OpenSearch:A/B tests

To help you use and debug various algorithms in OpenSearch, OpenSearch provides the A/B test feature. Before you use a feature in your online business, you can run an A/B test to test the feature by routing a specific proportion of queries to the specified test group. This helps prevent negative impacts on your online business in scenarios in which you use a feature that is not tested. The A/B test feature allows you to test the effects of query analysis, rough sort, and fine sort.

Step 1: Configure an A/B test

To obtain more comprehensive test metrics, we recommend that you enable the feature of collecting click behavioral data before you configure an A/B test. If you use the A/B test feature for the first time, you must perform the following four steps to configure an A/B test:

1. Go to the Create Test Group page.

Log on to the OpenSearch console. In the left-side navigation pane, choose Feature Extensions > A/B Test. On the A/B Test page, click Create.

2. Create a test group.

You can create a test group to test the effects of query analysis, rough sort, fine sort, and category prediction. Test Group Name: the name of the test group to be created. The name is for display purpose only. You can specify the name as needed. The name of a test group cannot exceed 30 characters in length.

3. Create a test.

After you specify the name of the test group and click Next, you can create a test by clicking Create Test in the Configure Test step. You can create up to 20 tests in a test group. However, only a maximum of 10 online tests can take effect at the same time.

3.1 Specify the name of the test to be created. You can customize the test name. The name of a test cannot exceed 30 characters in length. 3.2 Configure the test.

You can specify the query analysis, rough sort, and fine sort that you want to test. You can select only existing query analysis rules, rough sorts, and fine sorts.

When you specify the query analysis for a test in the Query Analysis Types and Rules column:

• If you select Custom from the drop-down list, you must select a query analysis rule that you create for the application.

• If you select Use Default Online Configuration from the drop-down list, the default online configuration is used for the test.

The preceding logic also applies when you specify the rough sort and fine sort for a test in the Sort Policy Types and Policies column.

If you select a sort policy that is being configured but unintentionally modify the sort policy, the effects of the A/B test may be affected. To resolve the issue, you are allowed to select only the sort policies that are published. In addition, after you select a sort policy, you are not allowed to modify it.

3.3 Specify the test traffic. The minimum traffic proportion that you can configure for a test is 1%. In the same scenario, the sum of the traffic proportions that you configure for all online tests in a test group cannot exceed 100%.

4. Complete the creation of the test group.

After you create and configure the test for the test group, click Next. The "The test group is created" message appears in the Complete step. Click Complete to return to the A/B Test page.The following figure shows that a created test group is in the Pending state.

5. Start testing.

After you create a test group, you can click Start Testing in the Actions column of the test group to start testing. After testing is started, the status of the test group becomes Testing.

6. Use the A/B test for online search queries.

After the A/B test feature is enabled and the test group is created and configured, if you want to use the A/B test for an online search query, you must specify the abtest parameter in the query. The abtest parameter consists of two sub-parameters: scene_tag and flow_divider.

The following code is a URL that is used for a search query:

/v3/openapi/apps/160029126/search?query=query=default:'Shenzhen'&&config=start:0,hit:10,format:fulljson&abtest=scene_tag:test_1,flow_divider:123456

• scene_tag: the name of the test group. Originally, scene_tag specifies the tag of the test scene. You can set the value of this parameter to the name of a test group that you create in the OpenSearch console. The preceding figure shows the steps. This way, the test traffic that you configure for the tests in the test group is sent to the corresponding tests.

• flow_divider: You must specify this parameter. The backend system performs the hash operation on the value of this parameter to allocate query traffic to different tests in the test group. The backend system allocates the traffic based on the proportions that you configure for each test in the OpenSearch console. We recommend that you set the flow_divider parameter to the ID of a user. You can also set the parameter to the ID or IP address of the user device.

Notes:

• When you access OpenSearch by using OpenSearch SDKs, if the values of the scene_tag and flow_divider parameters do not contain spaces or punctuations, you can directly call the corresponding methods without encoding the values of the scene_tag and flow_divider parameters. If the values of the scene_tag and flow_divider parameters contain spaces or punctuations, you must first encode the values of the scene_tag and flow_divider parameters. For more information, see the "Practical examples" section in this topic.

• If you access OpenSearch by calling API operations, the values of the scene_tag and flow_divider parameters must be URL-encoded. The value of the abtest parameter is passed to OpenSearch in the following format: abtest=urlencode(scene_tag:urlencode(\$scene),flow_divider:urlencode(\$value)). In the format, urlencode is a function that is used to encode URLs.

• For more information, see FAQ about A/B testing.

Step 2: Manage test groups and tests

Manage test groups.

After a test group is created, it is displayed in the test group list on the A/B Test page.You can perform the following basic operations on test groups that you create:

1. Start testing.

You can start testing for a test group whose status is Pending or Stopped.

2. Stop testing.

You can stop testing for a test group whose status is Testing.

3. Delete a test group.

You can delete a test group on the A/B Test page.

Manage tests.

After a test group is created, it is displayed in the test group list on the A/B Test page.To view the details of a test group, you can click Details in the Actions column of the test group. On the Test Group Details page, the Test List section lists the name, query analysis types and rules, sort policy types and policies, and traffic proportion of each test in the test group, and the operations that you can perform on the tests.You can perform the following basic operations on tests that you create:

1. Edit a test: You can edit the name, configuration, and traffic of a test.

2. Delete a test: If you delete a test, the system no longer retains its configurations. The test that you delete no longer takes effect online.

3. Configure a whitelist:

OpenSearch provides the whitelist feature that allows you to add a specific value of the flow_divider parameter to a specific test. This way, you can view the search effects of the test.

Step 3: View the details of an A/B test group

Behavioral data status

After you create a test group, you can click Details of the test group in the Actions column on the A/B Test page. On the Test Group Details page, you can view the behavioral data status. The behavioral data may be in the following states:

• Not activated: No behavioral data is uploaded for the application. For more information, see Data collection V2.0.

• Activated with no data: The feature of collecting behavioral data is enabled for the application. However, no behavioral data is received by the application.

• Abnormal (Unavailable): The behavioral data is found to be untrustworthy based on the quality check.

Test group status

After you create a test group, you can click Details in the Actions column of the test group on the A/B Test page. On the Test Group Details page, you can view the status of the test group. The test group may be in the following states:

• Pending: Testing may be started or stopped for the test group.

• Testing: Testing is started for the test group. The number of days that elapsed since the day when testing is started is displayed.

• Stopped: Testing is stopped for the test group. The accumulated test time is calculated from the time when testing is started for the first time to the time when testing is finally stopped. Only the number of days on which testing is actually running is accumulated.

Data statistics

To view the data statistics of a test group, find the test group that you want to view on the A/B Test page, and click Details in the Actions column of the test group. In the Data Statistics section of the Test Group Details page, you can view the data report of the A/B test. Alternatively, you can choose Report Statistics > A/B Test in the left-side navigation pane to go to the A/B Test page. In the OpenSearch console, you can view the data statistics of a test group one day after testing is started for the test group. After the OpenSearch console is upgraded, the core metrics comparison page and specific metric data tables are merged. You can select the metrics that you want to view from the Metric Type drop-down list. Instead of the original data form, the core metrics data is displayed in a line chart. By default, the metrics data of the last day is displayed. In the Data Statistics section of the Test Group Details page, you can select multiple tests whose data you want to compare from the Test drop-down list. Available core metrics include page views (PVs), user views (UVs), zero-result rate, average PVs per user, exposure times, search queries, and average search queries per user.

• Note: If you start an A/B test on a day, you can view the report data that is generated for the A/B test after 08:00 the next day. Even if you stop the A/B test on the day, you can view the report data that is generated before the A/B test is stopped the next day.

For example, you want to use OpenSearch to handle the following types of traffic of queries for an e-commerce product:

• Type 1: the traffic of the search requests that are sent from users based on the product keyword. Example:

query=config=format:fulljson&&query=default:'infant formula'&&sort=price

• Type 2: the call traffic that is sent from other internal services. Example:

query=config=format:fulljson&&query=cat_id:'1'|'2'|'3'&&sort=timestamp

For the traffic of type 1, you want to run an A/B test based on the member IDs of users to compare the effects of several sort expressions, category prediction models, or query analysis rules. To meet the requirement, you can perform the following steps:

1. Create a test group and required tests by using the A/B test feature in the OpenSearch console. In the Basic Information step on the Create Test Group page, set the Test Group Name parameter to user_search.

2. Set the abtest parameter in the search query. The Test Group Name parameter is set to user_search in the OpenSearch console. Therefore, you must set the scene_tag parameter to user_search and the flow_divider parameter to xxxx, which is the value of the member ID, in the search query.

2.1Use SDKs. The following code shows how to use the SDK for Java to configure the search query. The use of the SDK for PHP is similar to that of the SDK for Java.aliyun-sdk-opensearch-3.4.1 (Java) and opensearch-sdk-php-3.3.0 (PHP)

2.2Use APIs.

i. Construct the query: query=config=format:fulljson&&query=default:'infant formula'&&sort=-price&abtest=scene_tag:user_search,flow_divider:%e5%bc%a0%e4%b8%89.

ii. Perform URL encoding on the value of each parameter, such as query, sort, and abtest, in a request.

query=config%3dformat%3afulljson&&query%3ddefault%3a%27%e5%ae%9d%e5%ae%9d%e5%a5%b6%e7%b2%8

3. After you complete the preceding configurations, you can run the A/B test by routing the traffic of search queries from users based on the member ID to the test group.

UI

To go to the A/B test report page, perform the following steps: In the left-side navigation pane in the OpenSearch console, choose Feature Extensions > A/B Test. Find the test that you want to view and click Statistical report in the Actions column.

You can also choose Report Statistics > A/B test report in the left-side navigation pane to go to the A/B test report page.

Note: For more information about metrics that you can view on the A/B test report page, see A/B test report.