Overview
OpenSearch provides five categories of statistical metrics for A/B tests: core metrics, traffic metrics, behavioral metrics, transaction metrics, and user analysis metrics.
Core metrics
Metric | Definition | Calculation method | Dependency | Description |
Search PV | The number of successful search requests. PV is short for page view. Search requests include paging requests. | Counts each search request for which a success response is received. | Calculates the number of successful search requests on the current day. | |
Search UV | The number of users who visited search results pages. UV is short for unique visitor. | Counts each user only once regardless of how many times the user visited search results pages. | Depends on the user_id field that is specified in search requests when users send requests by calling an API operation or using an OpenSearch SDK. | Calculates the number of users who sent search requests on the current day. |
Page turning rate | The ratio of paging requests to all search requests. | Divides the number of paging requests by Search PV. | Calculates the ratio of paging requests to all search requests to assess the effects of search retrieval and search results sorting. | |
No result rate | The ratio of search requests whose results contain no items to all search requests. | Divides the number of search requests whose results contain no items by Search PV. | Calculates the ratio of search requests whose results contain no items to all search requests to assess the effect and content diversity of search retrieval. | |
PV-CTR | The click-through rate (CTR) of search results. | Divides IPV by Search PV. | Depends on the data collection feature and the uploaded behavioral data about user clicks. | Calculates the CTR of search results to assess the effects of search retrieval and search results sorting. |
UV-CTR | The CTR of users. | Divides IPV_UV by Search UV. | Depends on the data collection feature and the uploaded behavioral data about user clicks. | Calculates the CTR of users to assess the effects of search retrieval and search results sorting. |
TOP5 PV-CTR | The CTR of the top five items in search results. These items are the first five items on the first page of search results. | Divides the number of clicks on the top five items in search results by Search PV. | Depends on the data collection feature and the uploaded behavioral data about user clicks. | Calculates the CTR of the top five items in search results to assess the effect of search results sorting. |
Search PV per capita | The average number of successful search requests per user. | Divides Search PV by Search UV. | Depends on the user_id field that is specified in search requests when users send requests by calling an API operation or using an OpenSearch SDK. | Calculates the average number of successful search requests per user. More search requests indicate that users are interested in searching but often need to search for many times to obtain the expected results. |
Next day retention rate | The ratio of users who sent search requests for two consecutive days to users who sent search requests on the previous day. | Divides the intersection of Search UV (t-1) and Search UV (t) by Search UV (t-1). | Depends on the user_id field that is specified in search requests when users send requests by calling an API operation or using an OpenSearch SDK. | Indicates user stickiness. |
Traffic metrics
Metric | Definition | Calculation method | Dependency | Description |
Search PV | The number of successful search requests. Search requests include paging requests. | Counts each search request for which a success response is received. | Calculates the number of successful search requests on the current day. | |
Search UV | The number of users who visited search results pages. | Counts each user only once regardless of how many times the user visited search results pages. | Depends on the user_id field that is specified in search requests when users send requests by calling an API operation or using an OpenSearch SDK. | Calculates the number of users who sent search requests on the current day. |
Search PV per capita | The average number of successful search requests per user. | Divides Search PV by Search UV. | Depends on the user_id field that is specified in search requests when users send requests by calling an API operation or using an OpenSearch SDK. | Calculates the average number of successful search requests per user. More search requests indicate that users are interested in searching but often need to search for many times to obtain the expected results. |
Item impressions | The number of item exposures. Items refer to commodities or content on search results pages. If an item on the current page appears on the new page after a paging request or loading process, the second appearance of the item is counted as a new item exposure. | If the data collection feature is enabled and data about item exposures is uploaded, each item exposure is counted for this metric. Otherwise, the total number of items in search results is counted for this metric. | Depends on the data collection feature and the uploaded behavioral data about item exposures for higher accuracy. If the data collection feature is disabled or no behavioral data about item exposures is uploaded, the accuracy is low. | Calculates the number of item exposures in search results on the current day. |
Query number | The total number of queries from all users after deduplication by user. For each user, the same query that the user performed multiple times is counted only once. | Aggregates the number of deduplicated queries from each user. The same query that is performed by two users is counted twice. | Depends on the user_id field that is specified in search requests when users send requests by calling an API operation or using an OpenSearch SDK. If a search request contains the raw_query field, the raw query is used. Otherwise, the value for the default index is used as the query. If no default index is specified, the query is empty. | Calculates the total number of queries from all users after deduplication by user. Deduplication prevents repeated counting of paging requests and repeated searches. |
Number of queries per capita | The average number of deduplicated queries per user. | Divides Query number by Search UV. | Depends on the user_id field that is specified in search requests when users send requests by calling an API operation or using an OpenSearch SDK. | Calculates the average number of deduplicated queries per user. Deduplication prevents repeated counting of paging requests and repeated searches. |
Number of independent queries | The number of independent queries from all users. | Counts the number of queries from all users after deduplication. The same query that is performed by multiple users is counted only once. | Depends on the raw_query field that is specified in search requests when users send requests by calling an API operation or using an OpenSearch SDK. This dependency is optional. If a search request contains the raw_query field, the raw query is used. Otherwise, the value for the default index is used as the query. If no default index is specified, the query is empty. | Indicates the diversity of queries. |
Page turning rate | The ratio of paging requests to all search requests. | Divides the number of paging requests by Search PV. | Calculates the ratio of paging requests to all search requests to assess the effects of search retrieval and search results sorting. | |
No result rate | The ratio of search requests whose results contain no items to all search requests. | Divides the number of search requests whose results contain no items by Search PV. | Calculates the ratio of search requests whose results contain no items to all search requests to assess the effect and content diversity of search retrieval. | |
Low result rate | The ratio of search requests whose results contain no more than five items to all search requests. | Divides the number of search requests whose results contain no more than five items by Search PV. | Calculates the ratio of search requests whose results contain less than five items to all search requests to assess the effect and content diversity of search retrieval. | |
Request error times | The number of times when a search request is sent but an error response is received due to reasons such as throttling. | Counts each request for which an error message is received. | Indicates whether you need to adjust the application. For example, you may need to scale out the application. | |
Request error rate | The ratio of failed search requests to all search requests. The requests failed due to reasons such as throttling. | Divides Request error times by the sum of Search PV and Request error times. | Indicates whether you need to adjust the application. For example, you may need to scale out the application. |
Behavioral metrics
Metric | Definition | Calculation method | Dependency | Description |
IPV | The number of clicks on items in search results. | Counts each click on an item in search results. | Depends on the data collection feature and the uploaded behavioral data about user clicks. | Calculates the number of clicks on items in search results. |
IPV_UV | The number of users who clicked items on search results pages. | Counts each user only once regardless of how many times the user clicked items in search results. | Depends on the data collection feature and the uploaded behavioral data about user clicks. The user_id field must be specified in the data that is uploaded. | Calculates the number of users who clicked items in search results. |
IPV per capita | The average number of clicks on items in search results per user. | Divides IPV by Search UV. | Depends on the data collection feature and the uploaded behavioral data about user clicks. The user_id field must be specified in the data that is uploaded. | Calculates the average number of clicks on items in search results per user. |
PV-CTR | The CTR of search results. | Divides IPV by Search PV. | Depends on the data collection feature and the uploaded behavioral data about user clicks. | Calculates the CTR of search results to assess the effects of search retrieval and search results sorting. |
UV-CTR | The CTR of users. | Divides IPV_UV by Search UV. | Depends on the data collection feature and the uploaded behavioral data about user clicks. | Calculates the CTR of users to assess the effects of search retrieval and search results sorting. |
Item-CTR | The CTR of items. Items refer to commodities or content on search results pages. | Divides IPV by Item impressions. | Depends on the data collection feature and the uploaded behavioral data about user clicks. | Calculates the CTR of search results to assess the effects of search retrieval and search results sorting. |
Percentage of clicked PV | The ratio of search requests whose results are clicked by users to all search requests. | Divides the number of search requests whose results are clicked by users by Search PV. | Depends on the data collection feature and the uploaded behavioral data about user clicks. | Calculates the CTR of search results to assess the effects of search retrieval and search results sorting. |
TOP3 PV-CTR | The CTR of the top three items in search results. These items are the first three items on the first page of search results. | Divides the number of clicks on the top three items in search results by Search PV. | Depends on the data collection feature and the uploaded behavioral data about user clicks. | Calculates the CTR of the top three items in search results to assess the effect of search results sorting. |
TOP5 PV-CTR | The CTR of the top five items in search results. These items are the first five items on the first page of search results. | Divides the number of clicks on the top five items in search results by Search PV. | Depends on the data collection feature and the uploaded behavioral data about user clicks. | Calculates the CTR of the top five items in search results to assess the effect of search results sorting. |
TOP10 PV-CTR | The CTR of the top ten items in search results. These items are the first ten items on the first page of search results. | Divides the number of clicks on the top ten items in search results by Search PV. | Depends on the data collection feature and the uploaded behavioral data about user clicks. | Calculates the CTR of the top ten items in search results to assess the effect of search results sorting. |
Number of collections | The number of times when users add items from search results to favorites. | Counts each time when a user adds an item from search results to favorites. | Depends on the data collection feature and the uploaded behavioral data about adding items to favorites. The value of the bhv_type field must be collect in the data that is uploaded. | Assesses the effects of search retrieval, search results sorting, and search results on guiding users to add items to favorites. |
Favorite conversion rate | The ratio of times when users add items from search results to favorites to all search requests. | Divides Number of collections by Search PV. | Depends on the data collection feature and the uploaded behavioral data about adding items to favorites. The value of the bhv_type field must be collect in the data that is uploaded. | Assesses the effects of search retrieval, search results sorting, and search results on guiding users to add items to favorites. |
Additional purchases | The number of times when users add items from search results to the shopping cart. | Counts each time when a user adds an item from search results to the shopping cart. | Depends on the data collection feature and the uploaded behavioral data about adding items to the shopping cart, bookshelf, or playlist. The value of the bhv_type field must be cart in the data that is uploaded. | Assess the effects of search retrieval, search results sorting, and search results on guiding users to add items to the shopping cart. |
Add-on conversion rate | The ratio of times when users add items from search results to the shopping cart to all search requests. | Divides Additional purchases by Search PV. | Depends on the data collection feature and the uploaded behavioral data about adding items to the shopping cart, bookshelf, or playlist. The value of the bhv_type field must be cart in the data that is uploaded. | Assess the effects of search retrieval, search results sorting, and search results on guiding users to add items to the shopping cart. |
Likes | The number of times when users like items in search results. | Counts each time when a user likes an item in search results. | Depends on the data collection feature and the uploaded behavioral data about liking items. The value of the bhv_type field must be like in the data that is uploaded. | Assesses the effects of search retrieval, search results sorting, and search results on guiding users to like items in search results. |
Like conversion rate | The ratio of times when users like items in search results to all search requests. | Divides Likes by Search PV. | Depends on the data collection feature and the uploaded behavioral data about liking items. The value of the bhv_type field must be like in the data that is uploaded. | Assesses the effects of search retrieval, search results sorting, and search results on guiding users to like items in search results. |
Number of comments | The number of times when users post comments on items in search results. | Counts each time when a user posts comments on an item in search results. | Depends on the data collection feature and the uploaded behavioral data about posting comments. The value of the bhv_type field must be comment in the data that is uploaded. | Assesses the effects of search retrieval, search results sorting, and search results on guiding users to post comments. |
Comment conversion rate | The ratio of times when users post comments on items in search results to all search requests. | Divides Number of comments by Search PV. | Depends on the data collection feature and the uploaded behavioral data about posting comments. The value of the bhv_type field must be comment in the data that is uploaded. | Assesses the effects of search retrieval, search results sorting, and search results on guiding users to post comments. |
Transaction metrics
Metric | Definition | Calculation method | Dependency | Description |
Search guide GMV | The gross merchandise volume (GMV) of purchases that users make based on search results. | Aggregates the transaction amounts. | Depends on the data collection feature and the uploaded behavioral data about user purchases. The value of the bhv_type field must be buy in the data that is uploaded. | Assesses the effects of search retrieval, search results sorting, and search results on guiding users to make purchases. |
Search guide deal UV | The number of users who made purchases based on search results. | Counts each user only once regardless of how many purchases the user made based on search results. | Depends on the data collection feature and the uploaded behavioral data about user purchases. The value of the bhv_type field must be buy in the data that is uploaded. The user_id field must be specified in the data that is uploaded. | Assesses the effects of search retrieval, search results sorting, and search results on guiding users to make purchases. |
Search guide customer unit price | The average transaction amount per user among users who made purchases based on search results. | Divides Search guide GMV by Search guide deal UV. | Depends on the data collection feature and the uploaded behavioral data about user purchases. The value of the bhv_type field must be buy in the data that is uploaded. | Assesses the effects of search retrieval, search results sorting, and search results on guiding users to make purchases. |
Order conversion rate | The ratio of users who made purchases based on search results to all users who visited search results pages. | Divides Search guide deal UV by Search UV. | Depends on the data collection feature and the uploaded behavioral data about user purchases. The value of the bhv_type field must be buy in the data that is uploaded. | Assesses the effects of search retrieval, search results sorting, and search results on guiding users to make purchases. |
Click-through rate | The ratio of users who made purchases after they clicked items in search results to all users who clicked items in search results. | Divides Search guide deal UV by IPV_UV. | Depends on the data collection feature and the uploaded behavioral data about user purchases. The value of the bhv_type field must be buy in the data that is uploaded. | Assesses the effects of search retrieval, search results sorting, and search results on guiding users to make purchases. |
User analysis metrics
Metric | Definition | Calculation method | Dependency | Description |
Next day retention rate | The ratio of users who sent search requests for two consecutive days to users who sent search requests on the previous day. | Divides the intersection of Search UV (t-1) and Search UV (t) by Search UV (t-1). | Depends on the user_id field that is specified in search requests when users send requests by calling an API operation or using an OpenSearch SDK. | Indicates user stickiness. |
Lost user rate | The ratio of users who did not click items in search results to all users who visited search results pages. | Divides the difference between UV and IPV_UV by Search UV. | Depends on the data collection feature and the uploaded behavioral data about user clicks. The user_id field must be specified in the data that is uploaded. | Assesses the effects of search retrieval and search results sorting. |
Usage notes
When the A/B test report displays metrics for a day, the report displays the metrics for the day before the current day by default. You can set the date picker control to another day as needed. After you upload behavioral data, the A/B test report based on the uploaded data is available the next day.
The A/B test report provides line charts to display the trends of metrics in the last seven days. You can set the date picker control to another time period as needed. You can view the line chart of a metric by clicking the corresponding icon.
To view a line chart, perform the following steps: In the left-side navigation pane, choose Report Statistics > A/B Test. Select an application from the Application Name drop-down list, an experiment group, and then an experiment. Click a metric category tab. Then, set the date picker control to a time period as needed.
