All Products
Search
Document Center

Tablestore:Aggregation

Last Updated:Oct 17, 2024

You can perform aggregation operations to obtain the minimum value, maximum value, sum, average value, count and distinct count of rows, and percentile statistics. You can also perform aggregation operations to group results by field value, range, geographical location, filter, histogram, or date histogram, and perform nested queries. You can perform multiple aggregation operations for complex queries.

Procedure

The following figure shows the complete aggregation procedure.

fig_agg_pro

The server queries the data that meets the query conditions and performs aggregation on the data based on the request. Therefore, a request that requires aggregation is more complex than a request that does not require aggregation.

Background information

The following table describes the aggregation methods.

Method

Description

Minimum value

The aggregation method that can be used to return the minimum value of a field. This method can be used in a similar manner as the SQL MIN function.

Maximum value

The aggregation method that can be used to return the maximum value of a field. This method can be used in a similar manner as the SQL MAX function.

Sum

The aggregation method that can be used to return the sum of all values for a numeric field. This method can be used in a similar manner as the SQL SUM function.

Average value

The aggregation method that can be used to return the average of all values for a numeric field. This method can be used in a similar manner as the SQL AVG function.

Count

The aggregation method that can be used to return the total number of values for a field or the total number of rows in a search index. This method can be used in a similar manner as the SQL COUNT function.

Distinct count

The aggregation method that can be used to return the number of distinct values for a field. This method can be used in a similar manner as the SQL COUNT(DISTINCT) function.

Percentile statistics

A percentile value indicates the relative position of a value in a dataset. For example, when you collect statistics about the response time of each request during routine O&M of your system, you must analyze the response time distribution by using percentiles such as p25, p50, p90, and p99.

Group by field value

The aggregation method that can be used to group query results based on field values. The values that are the same are grouped together. Identical values are grouped together. The identical value for each group and the number of identical values in each group are returned.

Note

If the group contains an excessively large number of values, the calculated number may be different from the actual number.

Group by multiple fields

The aggregation method that can be used to group query results based on multiple fields. You can use tokens to perform paging.

Group by range

The aggregation method that can be used to group query results based on the value ranges of a field. Field values that are within a specified range are grouped together. The number of values in each range is returned.

Group by geographical location

The aggregation method that can be used to group query results based on distances from geographical locations to a central point. Query results in distances that are within a specified range are grouped together. The number of items in each range is returned.

Group by filter

The aggregation method that can be used to filter the query results and group them based on each filter to obtain the number of matching results. Results are returned in the order in which the filters are specified.

Query by histogram

The aggregation method can be used to group query results based on specific data intervals. Field values that are within the same range are grouped together. The value range of each group and the number of values in each group are returned.

Query by date histogram

The aggregation method that can be used to group query results based on specific date intervals. Field values that are within the same range are grouped together. The value range of each group and the number of values in each group are returned.

Query the rows that are obtained from the results of an aggregation operation in each group

After you group query results, you can query the rows in each group. This method can be used in a similar manner as ANY_VALUE(field) in MySQL.

Nesting

GroupBy supports nesting. You can perform sub-aggregation operations by using GroupBy.

Multiple aggregations

You can perform multiple aggregation operations.

Note

If you perform multiple complex aggregation operations at the same time, a long period of time may be required.

API operation

You can call the Search operation to use the aggregation features.

Prerequisites

You can use the Tablestore console, CLI, or SDKs to perform the aggregation operations. Before you perform the aggregation operations, make sure that the following preparations are made:

Important

If you use the Tablestore console or CLI, you may not be able to access all aggregation features. The actual aggregation features that are available to you shall prevail.

Use the Tablestore console

  1. Go to the Indexes tab.

    1. Log on to the Tablestore console.

    2. In the top navigation bar, select a resource group and a region.

    3. On the Overview page, click the name of the instance that you want to manage or click Manage Instance in the Actions column of the instance.

    4. On the Tables tab of the Instance Details tab, click the name of the data table or click Indexes in the Actions column of the data table.

  2. On the Indexes tab, find the search index that you want to use to query data and click Manage Data in the Actions column.

  3. In the Search dialog box, specify query conditions.

    1. By default, the system returns all attribute columns. To return specific attribute columns, turn off All Columns and specify the attribute columns that you want to return. Separate multiple attribute columns with commas (,).

      Note

      By default, the system returns all primary key columns of the data table.

    2. Select the And, Or, or Not logical operator based on your business requirements.

      If you select the And logical operator, data that meets the query conditions is returned. If you select the Or operator and specify a single query condition, data that meets the query condition is returned. If you select the Or logical operator and specify multiple query conditions, data that meets one of the query conditions is returned. If you select the Not logical operator, data that does not meet the query conditions is returned.

    3. Select an index field and click Add. Then, configure the Query Type and Value parameters.

      You can repeat this step to add query conditions for multiple index fields.

    4. By default, the sorting feature is disabled. If you want to sort the query results based on specific fields, turn on Sort and specify the fields based on which you want to sort the query results and the sorting order.

    5. By default, Collect Statistics is turned off. If you want to collect statistics for a specific field, turn on Collect Statistics and configure the following parameters based on your business requirements: Field Name, Statistics Type, Item, and Default.

      After you turn on Collect Statistics, you can add multiple fields at the same time. For the Statistics Type parameter, you can set the value to Minimum, Maximum, Sum, Average, Count, or Distinct count. The Default parameter takes effect when the index field does not exist in any row.

  4. Click OK.

    Data that meets the query conditions and statistical results are displayed on the Indexes tab.

Use the Tablestore CLI

You can run the search command in the Tablestore CLI to initiate aggregation requests. If you use the Tablestore CLI, the following aggregation operations are supported: obtain the minimum value, obtain the maximum value, obtain the sum, obtain the average value, and count the number of rows. For more information, see Search index.

  1. Run the following search command to query and analyze data in your data table and return all indexed columns:

    search -n search_index --return_all_indexed
  2. Enter the query conditions as prompted.

    The following sample code provides an example on how to query the rows in which the value of the gid column is less than 10 or exactly matches 77, and average the values of the gid column:

    {
        "Offset": -1,
        "Limit": 10,
        "Collapse": null,
        "Sort": null,
        "GetTotalCount": true,
        "Token": null,
        "Query": {
            "Name": "BoolQuery",
            "Query": {
                "MinimumShouldMatch": null,
                "MustQueries": null,
                "MustNotQueries": null,
                "FilterQueries": null,
                "ShouldQueries": [{
                    "Name": "RangeQuery",
                    "Query": {
                        "FieldName": "gid",
                        "From": null,
                        "To": 10,
                        "IncludeLower": false,
                        "IncludeUpper": false
                    }
                }, {
                    "Name": "TermQuery",
                    "Query": {
                        "FieldName": "gid",
                        "Term": 77
                    }
                }]
            }
        },
        "Aggregations": [{
            "Name": "avg",
            "Aggregation": {
                "AggName": "agg1",
                "Field": "gid",
                "MissingValue": null
            }
        }]
    }

Use Tablestore SDKs

The aggregation operations are supported by the following Tablestore SDKs: Tablestore SDK for Java, Tablestore SDK for Go, Tablestore SDK for Python, Tablestore SDK for Node.js, Tablestore SDK for .NET, and Tablestore SDK for PHP. In this example, Tablestore SDK for Java is used to describe how to perform aggregation operations.

Minimum value

The aggregation method that can be used to return the minimum value of a field. This method can be used in a similar manner as the SQL MIN function.

  • Parameters

    Parameter

    Description

    aggregationName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    fieldName

    The name of the field that is used to perform the aggregation operation. Only the LONG, DOUBLE, and DATE types are supported.

    missing

    The default value for the field on which the aggregation operation is performed, which is applied to rows in which the field value is empty.

    • If you do not specify a value for missing, the row is ignored.

    • If you specify a value for missing, the value of this parameter is used as the field value of the row.

  • Example

    /**
     * The price of each product is listed in the product table. Query the minimum price of the products that are produced in Zhejiang. 
     * Equivalent SQL statement: SELECT min(column_price) FROM product where place_of_production = "Zhejiang Province". 
     */
    public void min(SyncClient client) {
        // Use builder to create a query statement. 
        {
            SearchRequest searchRequest = SearchRequest.newBuilder()
                    .tableName("<TABLE_NAME>")
                    .indexName("<SEARCH_INDEX_NAME>")
                    .searchQuery(
                            SearchQuery.newBuilder()
                                    .query(QueryBuilders.term("place_of_production", "Zhejiang Province"))
                                    .limit(0) // If you want to obtain only the aggregation results instead of specific data, you can set limit to 0 to improve query performance. 
                                    .addAggregation(AggregationBuilders.min("min_agg_1", "column_price").missing(100))
                                    .build())
                    .build();
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            System.out.println(resp.getAggregationResults().getAsMinAggregationResult("min_agg_1").getValue());
        }
    
        // Create a query statement without using builder. 
        {
            SearchRequest searchRequest = new SearchRequest();
            searchRequest.setTableName("<TABLE_NAME>");
            searchRequest.setIndexName("<SEARCH_INDEX_NAME>");
    
            SearchQuery searchQuery = new SearchQuery();
            TermQuery query = new TermQuery();
            query.setTerm(ColumnValue.fromString("Zhejiang Province"));
            query.setFieldName("place_of_production");
            // In the following comment, builder is used to create a query statement. The method that uses builder to create a query statement has the same effect as the method that uses TermQuery to create a query statement. 
            // Query query2 = QueryBuilders.term("place_of_production", "Zhejiang Province").build();
    
            searchQuery.setQuery(query);
            searchQuery.setLimit(0);
    
            MinAggregation aggregation = new MinAggregation();
            aggregation.setAggName("min_agg_1");
            aggregation.setFieldName("column_price");
            aggregation.setMissing(ColumnValue.fromLong(100));
            // In the following comment, builder is used to create an aggregation statement. The method that uses builder to create an aggregation statement has the same effect as the method that uses TermQuery to create an aggregation statement. 
            // MinAggregation aggregation2 = AggregationBuilders.min("min_agg_1", "column_price").missing(100).build();
            List<Aggregation> aggregationList = new ArrayList<Aggregation>();
            aggregationList.add(aggregation);
            searchQuery.setAggregationList(aggregationList);
    
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            System.out.println(resp.getAggregationResults().getAsMinAggregationResult("min_agg_1").getValue());
        }
    }

Maximum value

The aggregation method that can be used to return the maximum value of a field. This method can be used in a similar manner as the SQL MAX function.

  • Parameters

    Parameter

    Description

    aggregationName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    fieldName

    The name of the field that is used to perform the aggregation operation. Only the LONG, DOUBLE, and DATE types are supported.

    missing

    The default value for the field on which the aggregation operation is performed, which is applied to rows in which the field value is empty.

    • If you do not specify a value for missing, the row is ignored.

    • If you specify a value for missing, the value of this parameter is used as the field value of the row.

  • Example

    /**
     * The price of each product is listed in the product table. Query the maximum price of the products that are produced in Zhejiang. 
     * Equivalent SQL statement: SELECT max(column_price) FROM product where place_of_production = "Zhejiang Province". 
     */
    public void max(SyncClient client) {
        // Use builder to create a query statement. 
        {
            SearchRequest searchRequest = SearchRequest.newBuilder()
                    .tableName("<TABLE_NAME>")
                    .indexName("<SEARCH_INDEX_NAME>")
                    .searchQuery(
                            SearchQuery.newBuilder()
                                    .query(QueryBuilders.term("place_of_production", "Zhejiang Province"))
                                    .limit(0) // If you want to obtain only the aggregation results instead of specific data, you can set limit to 0 to improve query performance. 
                                    .addAggregation(AggregationBuilders.max("max_agg_1", "column_price").missing(0))
                                    .build())
                    .build();
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            System.out.println(resp.getAggregationResults().getAsMaxAggregationResult("max_agg_1").getValue());
        }
    
        // Create a query statement without using builder. 
        {
            SearchRequest searchRequest = new SearchRequest();
            searchRequest.setTableName("<TABLE_NAME>");
            searchRequest.setIndexName("<SEARCH_INDEX_NAME>");
    
            SearchQuery searchQuery = new SearchQuery();
            TermQuery query = new TermQuery();
            query.setTerm(ColumnValue.fromString("Zhejiang Province"));
            query.setFieldName("place_of_production");
            // In the following comment, builder is used to create a query statement. The method that uses builder to create a query statement has the same effect as the method that uses TermQuery to create a query statement. 
            // Query query2 = QueryBuilders.term("place_of_production", "Zhejiang Province").build();
    
            searchQuery.setQuery(query);
            searchQuery.setLimit(0);
    
            MaxAggregation aggregation = new MaxAggregation();
            aggregation.setAggName("max_agg_1");
            aggregation.setFieldName("column_price");
            aggregation.setMissing(ColumnValue.fromLong(100));
            // In the following comment, builder is used to create an aggregation statement. The method that uses builder to create an aggregation statement has the same effect as the method that uses TermQuery to create an aggregation statement. 
            // MaxAggregation aggregation2 = AggregationBuilders.max("max_agg_1", "column_price").missing(100).build();
            List<Aggregation> aggregationList = new ArrayList<Aggregation>();
            aggregationList.add(aggregation);
            searchQuery.setAggregationList(aggregationList);
    
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            System.out.println(resp.getAggregationResults().getAsMaxAggregationResult("max_agg_1").getValue());
        }
    }

Sum

The aggregation method that can be used to return the sum of all values for a numeric field. This method can be used in a similar manner as the SQL SUM function.

  • Parameters

    Parameter

    Description

    aggregationName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    fieldName

    The name of the field that is used to perform the aggregation operation. Only the LONG and DOUBLE types are supported.

    missing

    The default value for the field on which the aggregation operation is performed, which is applied to rows in which the field value is empty.

    • If you do not specify a value for missing, the row is ignored.

    • If you specify a value for missing, the value of this parameter is used as the field value of the row.

  • Example

    /**
     * The price of each product is listed in the product table. Query the maximum price of the products that are produced in Zhejiang. 
     * SQL statement: SELECT sum(column_price) FROM product where place_of_production = "Zhejiang". 
     */
    public void sum(SyncClient client) {
        // Use builder to create a query statement. 
        {
            SearchRequest searchRequest = SearchRequest.newBuilder()
                    .tableName("<TABLE_NAME>")
                    .indexName("<SEARCH_INDEX_NAME>")
                    .searchQuery(
                            SearchQuery.newBuilder()
                                    .query(QueryBuilders.term("place_of_production", "Zhejiang Province"))
                                    .limit(0) // If you want to obtain only the aggregation results instead of specific data, you can set limit to 0 to improve query performance. 
                                    .addAggregation(AggregationBuilders.sum("sum_agg_1", "column_number").missing(10))
                                    .build())
                    .build();
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            System.out.println(resp.getAggregationResults().getAsSumAggregationResult("sum_agg_1").getValue());
        }
    
        // Create a query statement without using builder. 
        {
            SearchRequest searchRequest = new SearchRequest();
            searchRequest.setTableName("<TABLE_NAME>");
            searchRequest.setIndexName("<SEARCH_INDEX_NAME>");
    
            SearchQuery searchQuery = new SearchQuery();
            TermQuery query = new TermQuery();
            query.setTerm(ColumnValue.fromString("Zhejiang Province"));
            query.setFieldName("place_of_production");
            // In the following comment, builder is used to create a query statement. The method that uses builder to create a query statement has the same effect as the method that uses TermQuery to create a query statement. 
            // Query query2 = QueryBuilders.term("place_of_production", "Zhejiang Province").build();
    
            searchQuery.setQuery(query);
            searchQuery.setLimit(0);
    
            SumAggregation aggregation = new SumAggregation();
            aggregation.setAggName("sum_agg_1");
            aggregation.setFieldName("column_number");
            aggregation.setMissing(ColumnValue.fromLong(100));
            // In the following comment, builder is used to create an aggregation statement. The method that uses builder to create an aggregation statement has the same effect as the method that uses TermQuery to create an aggregation statement. 
            // SumAggregation aggregation2 = AggregationBuilders.sum("sum_agg_1", "column_number").missing(10).build();
            List<Aggregation> aggregationList = new ArrayList<Aggregation>();
            aggregationList.add(aggregation);
            searchQuery.setAggregationList(aggregationList);
    
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            System.out.println(resp.getAggregationResults().getAsSumAggregationResult("sum_agg_1").getValue());
        }
    }

Average value

The aggregation method that can be used to return the average of all values for a numeric field. This method can be used in a similar manner as the SQL AVG function.

  • Parameters

    Parameter

    Description

    aggregationName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    fieldName

    The name of the field that is used to perform the aggregation operation. Only the LONG, DOUBLE, and DATE types are supported.

    missing

    The default value for the field on which the aggregation operation is performed, which is applied to rows in which the field value is empty.

    • If you do not specify a value for missing, the row is ignored.

    • If you specify a value for missing, the value of this parameter is used as the field value of the row.

  • Example

    /**
     * The sales volume of each product is listed in the product table. Query the average price of the products that are produced in Zhejiang. 
     * EquivalentSQL statement: SELECT avg(column_price) FROM product where place_of_production = "Zhejiang Province". 
     */
    public void avg(SyncClient client) {
        // Use builder to create a query statement. 
        {
            SearchRequest searchRequest = SearchRequest.newBuilder()
                    .tableName("<TABLE_NAME>")
                    .indexName("<SEARCH_INDEX_NAME>")
                    .searchQuery(
                            SearchQuery.newBuilder()
                                    .query(QueryBuilders.term("place_of_production", "Zhejiang Province"))
                                    .limit(0) // If you want to obtain only the aggregation results instead of specific data, you can set limit to 0 to improve query performance. 
                                    .addAggregation(AggregationBuilders.avg("avg_agg_1", "column_price"))
                                    .build())
                    .build();
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            System.out.println(resp.getAggregationResults().getAsAvgAggregationResult("avg_agg_1").getValue());
        }
    
        // Create a query statement without using builder. 
        {
            SearchRequest searchRequest = new SearchRequest();
            searchRequest.setTableName("<TABLE_NAME>");
            searchRequest.setIndexName("<SEARCH_INDEX_NAME>");
    
            SearchQuery searchQuery = new SearchQuery();
            TermQuery query = new TermQuery();
            query.setTerm(ColumnValue.fromString("Zhejiang Province"));
            query.setFieldName("place_of_production");
            // In the following comment, builder is used to create a query statement. The method that uses builder to create a query statement has the same effect as the method that uses TermQuery to create a query statement. 
            // Query query2 = QueryBuilders.term("place_of_production", "Zhejiang Province").build();
    
            searchQuery.setQuery(query);
            searchQuery.setLimit(0);
    
            AvgAggregation aggregation = new AvgAggregation();
            aggregation.setAggName("avg_agg_1");
            aggregation.setFieldName("column_price");
            // In the following comment, builder is used to create an aggregation statement. The method that uses builder to create an aggregation statement has the same effect as the method that uses TermQuery to create an aggregation statement. 
            // AvgAggregation aggregation2 = AggregationBuilders.avg("avg_agg_1", "column_price").build();
            List<Aggregation> aggregationList = new ArrayList<Aggregation>();
            aggregationList.add(aggregation);
            searchQuery.setAggregationList(aggregationList);
    
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            System.out.println(resp.getAggregationResults().getAsAvgAggregationResult("avg_agg_1").getValue());
        }
    }
                        

Count

The aggregation method that can be used to return the total number of values for a field or the total number of rows in a search index. This method can be used in a similar manner as the SQL COUNT function.

Note

You can use the following methods to query the total number of rows in a search index or the total number of rows that meet the query conditions:

  • Use the count feature of aggregation and specify count(*) in the request.

  • Use the query feature to obtain the number of rows that meet the query conditions. Set the setGetTotalCount parameter to true in the query. Use MatchAllQuery to obtain the total number of rows in a search index.

You can use the name of a column as the value of the count expression to query the number of rows that contain the column in a search index. This method is suitable for scenarios that involve sparse columns.

  • Parameters

    Parameter

    Description

    aggregationName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    fieldName

    The name of the field that is used to perform the aggregation operation. Only the following data types are supported: LONG, DOUBLE, BOOLEAN, KEYWORD, GEOPOINT, and DATE.

  • Example

    /**
     * Penalty records of merchants are recorded in the merchant table. You can query the number of merchants who are located in Zhejiang and for whom penalty records exist. If no penalty records exist for a merchant, the field that corresponds to penalty records also does not exist for the merchant. 
     * EquivalentSQL statement: SELECT count(column_history) FROM product where place_of_production = "Zhejiang Province". 
     */
    public void count(SyncClient client) {
        // Use builder to create a query statement. 
        {
            SearchRequest searchRequest = SearchRequest.newBuilder()
                    .tableName("<TABLE_NAME>")
                    .indexName("<SEARCH_INDEX_NAME>")
                    .searchQuery(
                            SearchQuery.newBuilder()
                                    .query(QueryBuilders.term("place_of_production", "Zhejiang Province"))
                                    .limit(0) // If you want to obtain only the aggregation results instead of specific data, you can set limit to 0 to improve query performance. 
                                    .addAggregation(AggregationBuilders.count("count_agg_1", "column_history"))
                                    .build())
                    .build();
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            System.out.println(resp.getAggregationResults().getAsCountAggregationResult("count_agg_1").getValue());
        }
    
        // Create a query statement without using builder. 
        {
            SearchRequest searchRequest = new SearchRequest();
            searchRequest.setTableName("<TABLE_NAME>");
            searchRequest.setIndexName("<SEARCH_INDEX_NAME>");
    
            SearchQuery searchQuery = new SearchQuery();
            TermQuery query = new TermQuery();
            query.setTerm(ColumnValue.fromString("Zhejiang Province"));
            query.setFieldName("place_of_production");
            // In the following comment, builder is used to create a query statement. The method that uses builder to create a query statement has the same effect as the method that uses TermQuery to create a query statement. 
            // Query query2 = QueryBuilders.term("place_of_production", "Zhejiang Province").build();
    
            searchQuery.setQuery(query);
            searchQuery.setLimit(0);
    
            CountAggregation aggregation = new CountAggregation();
            aggregation.setAggName("count_agg_1");
            aggregation.setFieldName("column_history");
            // In the following comment, builder is used to create an aggregation statement. The method that uses builder to create an aggregation statement has the same effect as the method that uses TermQuery to create an aggregation statement. 
            // CountAggregation aggregation2 = AggregationBuilders.count("count_agg_1", "column_history").build();
            List<Aggregation> aggregationList = new ArrayList<Aggregation>();
            aggregationList.add(aggregation);
            searchQuery.setAggregationList(aggregationList);
    
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            System.out.println(resp.getAggregationResults().getAsCountAggregationResult("count_agg_1").getValue());
        }
    }

Distinct count

The aggregation method that can be used to return the number of distinct values for a field. This method can be used in a similar manner as the SQL COUNT(DISTINCT) function.

Note

The number of distinct values is an approximate number.

  • If the total number of rows before the distinct count feature is used is less than 10,000, the calculated result is an exact value.

  • If the total number of rows before the distinct count feature is used is greater than or equal to 100 million, the error rate is approximately 2%.

  • Parameters

    Parameter

    Description

    aggregationName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    fieldName

    The name of the field that is used to perform the aggregation operation. Only the following data types are supported: LONG, DOUBLE, BOOLEAN, KEYWORD, GEOPOINT, and DATE.

    missing

    The default value for the field on which the aggregation operation is performed, which is applied to rows in which the field value is empty.

    • If you do not specify a value for missing, the row is ignored.

    • If you specify a value for missing, the value of this parameter is used as the field value of the row.

  • Example

    /**
     * Query the number of distinct provinces from which the products are produced. 
     * EquivalentSQL statement: SELECT count(distinct column_place) FROM product. 
     */
    public void distinctCount(SyncClient client) {
        // Use builder to create a query statement. 
        {
            SearchRequest searchRequest = SearchRequest.newBuilder()
                    .tableName("<TABLE_NAME>")
                    .indexName("<SEARCH_INDEX_NAME>")
                    .searchQuery(
                            SearchQuery.newBuilder()
                                    .query(QueryBuilders.matchAll())
                                    .limit(0) // If you want to obtain only the aggregation results instead of specific data, you can set limit to 0 to improve query performance. 
                                    .addAggregation(AggregationBuilders.distinctCount("dis_count_agg_1", "column_place"))
                                    .build())
                    .build();
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            System.out.println(resp.getAggregationResults().getAsDistinctCountAggregationResult("dis_count_agg_1").getValue());
        }
    
        // Create a query statement without using builder. 
        {
            SearchRequest searchRequest = new SearchRequest();
            searchRequest.setTableName("<TABLE_NAME>");
            searchRequest.setIndexName("<SEARCH_INDEX_NAME>");
    
            SearchQuery searchQuery = new SearchQuery();
            MatchAllQuery query = new MatchAllQuery();
            // In the following comment, builder is used to create a query statement. The method that uses builder to create a query statement has the same effect as the method that uses TermQuery to create a query statement. 
            // Query query2 = QueryBuilders.matchAll().build();
    
            searchQuery.setQuery(query);
            searchQuery.setLimit(0);
    
            DistinctCountAggregation aggregation = new DistinctCountAggregation();
            aggregation.setAggName("dis_count_agg_1");
            aggregation.setFieldName("column_place");
            // In the following comment, builder is used to create an aggregation statement. The method that uses builder to create an aggregation statement has the same effect as the method that uses TermQuery to create an aggregation statement. 
            // DistinctCountAggregation aggregation2 = AggregationBuilders.distinctCount("dis_count_agg_1", "column_place").build();
            List<Aggregation> aggregationList = new ArrayList<Aggregation>();
            aggregationList.add(aggregation);
            searchQuery.setAggregationList(aggregationList);
    
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            System.out.println(resp.getAggregationResults().getAsDistinctCountAggregationResult("dis_count_agg_1").getValue());
        }
    }

Percentile statistics

A percentile value indicates the relative position of a value in a dataset. For example, when you collect statistics about the response time of each request during routine O&M of your system, you must analyze the response time distribution by using percentiles such as p25, p50, p90, and p99.

Note

To improve the accuracy of the results, we recommend that you specify extreme percentile values such as p1 and p99. If you use extreme percentile values instead of other values such as p50, the returned results are more accurate.

  • Parameters

    Parameter

    Description

    aggregationName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    fieldName

    The name of the field that is used to perform the aggregation operation. Only the LONG, DOUBLE, and DATE types are supported.

    percentiles

    The percentiles such as p50, p90, and p99. You can specify one or more percentiles.

    missing

    The default value for the field on which the aggregation operation is performed, which is applied to rows in which the field value is empty.

    • If you do not specify a value for missing, the row is ignored.

    • If you specify a value for missing, the value of this parameter is used as the field value of the row.

  • Example

    /**
     * Analyze the distribution of the response time of each request that is sent to the system by using percentiles. 
     */
    public void percentilesAgg(SyncClient client) {
        // Use builder to create a query statement. 
        {
            SearchRequest searchRequest = SearchRequest.newBuilder()
                    .tableName("<TABLE_NAME>")
                    .indexName("indexName")
                    .searchQuery(
                            SearchQuery.newBuilder()
                                    .query(QueryBuilders.matchAll())
                                    .limit(0) // If you want to obtain only the aggregation results instead of specific data, you can set limit to 0 to improve query performance. 
                                    .addAggregation(AggregationBuilders.percentiles("percentilesAgg", "latency")
                                            .percentiles(Arrays.asList(25.0d, 50.0d, 99.0d))
                                            .missing(1.0))
                                    .build())
                    .build();
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the results. 
            PercentilesAggregationResult percentilesAggregationResult = resp.getAggregationResults().getAsPercentilesAggregationResult("percentilesAgg");
            for (PercentilesAggregationItem item : percentilesAggregationResult.getPercentilesAggregationItems()) {
                System.out.println("key: " + item.getKey() + " value:" + item.getValue().asDouble());
            }
        }
    
        // Create a query statement without using builder. 
        {
            SearchRequest searchRequest = new SearchRequest();
            searchRequest.setTableName("<TABLE_NAME>");
            searchRequest.setIndexName("<SEARCH_INDEX_NAME>");
    
            SearchQuery searchQuery = new SearchQuery();
            MatchAllQuery query = new MatchAllQuery();
            // In the following comment, builder is used to create a query statement. The method that uses builder to create a query statement has the same effect as the method that uses TermQuery to create a query statement. 
            // Query query2 = QueryBuilders.matchAll().build();
    
            searchQuery.setQuery(query);
            searchQuery.setLimit(0);
    
            PercentilesAggregation aggregation = new PercentilesAggregation();
            aggregation.setAggName("percentilesAgg");
            aggregation.setFieldName("latency");
            aggregation.setPercentiles(Arrays.asList(25.0d, 50.0d, 99.0d));
            // In the following comment, builder is used to create an aggregation statement. The method that uses builder to create an aggregation statement has the same effect as the method that uses TermQuery to create an aggregation statement. 
            // AggregationBuilders.percentiles("percentilesAgg", "latency").percentiles(Arrays.asList(25.0d, 50.0d, 99.0d)).missing(1.0).build();
            List<Aggregation> aggregationList = new ArrayList<Aggregation>();
            aggregationList.add(aggregation);
            searchQuery.setAggregationList(aggregationList);
    
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the results. 
            PercentilesAggregationResult percentilesAggregationResult = resp.getAggregationResults().getAsPercentilesAggregationResult("percentilesAgg");
            for (PercentilesAggregationItem item : percentilesAggregationResult.getPercentilesAggregationItems()) {
                System.out.println("key: " + item.getKey() + " value:" + item.getValue().asDouble());
            }
        }
    }

Group by field value

The aggregation method that can be used to group query results based on field values. The values that are the same are grouped together. Identical values for each group and the number of identical values in each group are returned.

Note
  • The calculated number may be different from the actual number if the number of values in a group is very large.

  • If you want to group query results based on multiple fields, you can use the groupBy parameter in nested mode or use the GroupByComposite parameter. For information about the differences between the groupBy and GroupByComposite parameters, see Appendix: different methods for multi-field grouping.

  • Parameters

    Parameter

    Description

    groupByName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    fieldName

    The name of the field that is used to perform the aggregation operation. Only the LONG, DOUBLE, BOOLEAN, KEYWORD, and DATE types are supported.

    groupBySorter

    The sorting rules for groups. By default, groups are sorted based on the number of items in the groups in descending order. If you configure multiple sorting rules, the groups are sorted based on the order in which the rules are configured. The following sorting rules are supported:

    • Sort by value in alphabetical order.

    • Sort by value in reverse alphabetical order

    • Sort by row count in ascending order

    • Sort by row count in descending order

    • Sort by the values that are obtained from sub-aggregation results in ascending order

    • Sort the values that are obtained from sub-aggregation results in descending order

    size

    The number of groups that you want to return. Default value: 10. Maximum value: 2000. If the number of groups exceeds 2,000, only the first 2,000 groups are returned.

    subAggregation and subGroupBy

    The sub-aggregation operation. You can perform the sub-aggregation operation based on the grouping results.

    • Scenario

      Query the number of products in each category, and the maximum and minimum product prices in each category.

    • Method

      Group query results by product category to obtain the number of products in each category. Then, perform two sub-aggregation operations to obtain the maximum and minimum product prices in each category.

    • Examples:

      • Fruits: 5. The maximum price is 15. The minimum price is 3.

      • Toiletries: 10. The maximum price is 98. The minimum price is 1.

      • Electronic devices: 3. The maximum price is 8,699. The minimum price is 2,300.

      • Other products: 15. The maximum price is 1,000. The minimum price is 80.

  • Examples

    Group by single field

    /**
     * Query the number of products, and the maximum and minimum product prices in each category. 
     * Example of returned results: Fruits: 5. The maximum price is 15, and the minimum price is 3. Toiletries: 10. The maximum price is 98, and the minimum price is 1. Electronic devices: 3. The maximum price is 8699, and the minimum price is 2300. 
     * Other products: 15. The maximum price is 1000, and the minimum price is 80. 
     */
    public void groupByField(SyncClient client) {
        // Use builder to create a query statement. 
        {
            SearchRequest searchRequest = SearchRequest.newBuilder()
                    .tableName("<TABLE_NAME>")
                    .indexName("indexName")
                    .searchQuery(
                            SearchQuery.newBuilder()
                                    .query(QueryBuilders.matchAll())
                                    .limit(0) // If you want to obtain only the aggregation results instead of specific data, you can set limit to 0 to improve query performance. 
                                    .addGroupBy(GroupByBuilders
                                            .groupByField("name1", "column_type")
                                            .addSubAggregation(AggregationBuilders.min("subName1", "column_price"))
                                            .addSubAggregation(AggregationBuilders.max("subName2", "column_price"))
                                    )
                                    .build())
                    .build();
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            for (GroupByFieldResultItem item : resp.getGroupByResults().getAsGroupByFieldResult("name1").getGroupByFieldResultItems()) {
                // Display values. 
                System.out.println(item.getKey());
                // Display the number of rows. 
                System.out.println(item.getRowCount());
                // Display the minimum prices. 
                System.out.println(item.getSubAggregationResults().getAsMinAggregationResult("subName1").getValue());
                // Display the maximum prices. 
                System.out.println(item.getSubAggregationResults().getAsMaxAggregationResult("subName2").getValue());
            }
        }
    
        // Create a query statement without using builder. 
        {
            SearchRequest searchRequest = new SearchRequest();
            searchRequest.setTableName("<TABLE_NAME>");
            searchRequest.setIndexName("<SEARCH_INDEX_NAME>");
    
            SearchQuery searchQuery = new SearchQuery();
            MatchAllQuery query = new MatchAllQuery();
            // In the following comment, builder is used to create a query statement. The method that uses builder to create a query statement has the same effect as the method that uses TermQuery to create a query statement. 
            // Query query2 = QueryBuilders.matchAll().build();
    
            searchQuery.setQuery(query);
            searchQuery.setLimit(0);
    
            GroupByField groupByField = new GroupByField();
            groupByField.setGroupByName("name1");
            groupByField.setFieldName("column_type");
            // Configure sub-aggregation operations. 
            MinAggregation minAggregation = AggregationBuilders.min("subName1", "column_price").build();
            MaxAggregation maxAggregation = AggregationBuilders.max("subName2", "column_price").build();
            groupByField.setSubAggregations(Arrays.asList(minAggregation, maxAggregation));
    
            // In the following comment, builder is used to create an aggregation statement. The method that uses builder to create an aggregation statement has the same effect as the method that uses TermQuery to create an aggregation statement. 
            // GroupByBuilders.groupByField("name1", "column_type")
            //        .addSubAggregation(AggregationBuilders.min("subName1", "column_price"))
            //        .addSubAggregation(AggregationBuilders.max("subName2", "column_price").build());
            List<GroupBy> groupByList = new ArrayList<GroupBy>();
            groupByList.add(groupByField);
            searchQuery.setGroupByList(groupByList);
            searchRequest.setSearchQuery(searchQuery);
    
            // Execute the query statement. 
            SearchResponse resp = client.search(searchRequest);
            // Obtain the aggregation results. 
            for (GroupByFieldResultItem item : resp.getGroupByResults().getAsGroupByFieldResult("name1").getGroupByFieldResultItems()) {
                // Display values. 
                System.out.println(item.getKey());
                // Display the number of rows. 
                System.out.println(item.getRowCount());
                // Display the minimum prices. 
                System.out.println(item.getSubAggregationResults().getAsMinAggregationResult("subName1").getValue());
                // Display the maximum prices. 
                System.out.println(item.getSubAggregationResults().getAsMaxAggregationResult("subName2").getValue());
            }
        }
    }

    Group by multiple fields in nested mode

    /**
     * Example of grouping query results by multiple fields in nested mode. 
     * In a search index, you can use two groupBy fields in nested mode to achieve the same effect as using multiple groupBy fields in an SQL statement. 
     * Equivalent SQL statement: select a,d, sum(b),sum(c) from user group by a,d. 
     */
    public void GroupByMultiField() {
        SearchRequest searchRequest = SearchRequest.newBuilder()
            .tableName("<TABLE_NAME>")
            .indexName("<SEARCH_INDEX_NAME>")
            .returnAllColumns(true)   // You can set returnAllColumns to false and specify a value for addColumesToGet to have better query performance. 
            //.addColumnsToGet("col_1","col_2")
            .searchQuery(SearchQuery.newBuilder()
                .query(QueryBuilders.matchAll())   // Specify query conditions. Query conditions can be used in the same manner as the WHERE clause in SQL. You can use QueryBuilders.bool() to perform nested queries. 
                .addGroupBy(
                    GroupByBuilders
                        .groupByField("unique name_1", "field_a")
                        .size(20)
                        .addSubGroupBy(
                            GroupByBuilders
                                .groupByField("unique name_2", "field_d")
                                .size(20)
                                .addSubAggregation(AggregationBuilders.sum("unique name_3", "field_b"))
                                .addSubAggregation(AggregationBuilders.sum("unique name_4", "field_c"))
                        )
                )
                .build())
            .build();
        SearchResponse response = client.search(searchRequest);
        // Query rows that meet the specified conditions. 
        List<Row> rows = response.getRows();
        // Obtain the aggregation results. 
        GroupByFieldResult groupByFieldResult1 = response.getGroupByResults().getAsGroupByFieldResult("unique name_1");
        for (GroupByFieldResultItem resultItem : groupByFieldResult1.getGroupByFieldResultItems()) {
            System.out.println("field_a key:" + resultItem.getKey() + " Count:" + resultItem.getRowCount());
            // Obtain the sub-aggregation results. 
            GroupByFieldResult subGroupByResult = resultItem.getSubGroupByResults().getAsGroupByFieldResult("unique name_2");
            for (GroupByFieldResultItem item : subGroupByResult.getGroupByFieldResultItems()) {
                System.out.println("field_a " + resultItem.getKey() + " field_d key:" + item.getKey() + " Count: " + item.getRowCount());
                double sumOf_field_b = item.getSubAggregationResults().getAsSumAggregationResult("unique name_3").getValue();
                double sumOf_field_c = item.getSubAggregationResults().getAsSumAggregationResult("unique name_4").getValue();
                System.out.println("sumOf_field_b:" + sumOf_field_b);
                System.out.println("sumOf_field_c:" + sumOf_field_c);
            }
        }
    }

    Sort groups for aggregation

    /**
     * Example of configuring sorting rules for aggregation. 
     * Method: Configure the sorting rules by specifying GroupBySorter. If you configure multiple sorting rules, the groups are sorted based on the order in which the rules are configured. GroupBySorter supports sorting in ascending or descending order. 
     * By default, GroupBySorter.rowCountSortInDesc() is used, and the groups are sorted by row count in descending order. 
     */
    public void groupByFieldWithSort(SyncClient client) {
        // Create a query statement. 
        SearchRequest searchRequest = SearchRequest.newBuilder()
            .tableName("<TABLE_NAME>")
            .indexName("<SEARCH_INDEX_NAME>")
            .searchQuery(
                SearchQuery.newBuilder()
                    .query(QueryBuilders.matchAll())
                    .limit(0)
                    .addGroupBy(GroupByBuilders
                        .groupByField("name1", "column_type")
                        //.addGroupBySorter(GroupBySorter.subAggSortInAsc("subName1")) // Sort the groups in ascending order based on the values that are obtained from sub-aggregation results. 
                        .addGroupBySorter(GroupBySorter.groupKeySortInAsc())           // Sort the groups in ascending order based on the values that are obtained from aggregation results. 
                        //.addGroupBySorter(GroupBySorter.rowCountSortInDesc())        // Sort the groups in descending order based on the number of rows that are obtained from the aggregation results in each group. 
                        .size(20)
                        .addSubAggregation(AggregationBuilders.min("subName1", "column_price"))
                        .addSubAggregation(AggregationBuilders.max("subName2", "column_price"))
                    )
                    .build())
            .build();
        // Execute the query statement. 
        SearchResponse resp = client.search(searchRequest);
    }

Group by multiple fields

The aggregation method that can be used to group query results based on multiple fields. You can use tokens to perform paging.

Note
  • Only Tablestore SDKs for Java and Go support this feature.

  • If you want to group query results based on multiple fields, you can use the groupBy parameter in nested mode or use the GroupByComposite parameter. For information about the differences between the groupBy and GroupByComposite parameters, see Appendix: different methods for multi-field grouping.

  • Parameters

    Parameter

    Description

    groupByName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    sources

    The fields by which you want to group query results. You can group query results by up to 32 fields and perform aggregation operations on the resulting groups. The following group types are supported:

    Important
    • For group types specified in the SourceGroupByList parameter, you can set the GroupBySorter parameter to only groupKeySort.

    • By default, the groups are sorted in descending order.

    • If a field value in a column does not exist, the system returns NULL.

    • For the GroupByField type, you can configure only the following parameters: groupByName, fieldName, and groupBySorter.

    • For the GroupByHistogram type, you can configure only the following parameters: groupByName, fieldName, interval, and groupBySorter.

    • For the GroupByDateHistogram type, you can configure only the following parameters: groupByName, fieldName, interval, timeZone, and groupBySorter.

    nextToken

    The pagination token that is used in the next request to retrieve a new page of groups. You can obtain the value of the NextToken parameter from the output of the GroupByCompositeResult operation. The NextToken parameter allows you to obtain all the grouping results.

    size

    The number of groups per page. If the number of groups that meet the requirements exceeds the value of the Size parameter, you must use the NextToken parameter to obtain the groups on the next page.

    Important

    If you want to limit the number of groups to return, you cannot configure the Size and SuggestedSize parameters at the same time. In most cases, we recommend that you configure the Size parameter.

    In scenarios where you want to interconnect Tablestore with a high-throughput computing engine such as Apache Spark or PrestoSQL, we recommend that you configure the SuggestedSize parameter.

    suggestedSize

    The expected number of groups per page. You can set a value greater than the maximum number of groups allowed at the server side or -1. The server side returns the actual number of groups based on its capacity. This parameter is applicable in scenarios where you interconnect Tablestore with a high-throughput computing engine, such as Apache Spark or PrestoSQL.

    If you set this parameter to a value that is greater than the maximum number of groups allowed at the server side, the system adjusts the value to the maximum number of groups allowed at the server side. The actual number of groups that are returned equals min(suggestedSize, maximum number of groups allowed at the server side, total number of groups).

    subAggregation and subGroupBy

    The sub-aggregation operation. You can perform the sub-aggregation operation based on the grouping results.

    Important

    The GroupByComposite type is not supported in the SubGroupByList parameter.

  • Example

    /**
     * Group and aggregate query results: Group the query results and perform the aggregation operation on the resulting groups based on the parameters such as groupbyField, groupByHistogram, and groupByDataHistogram that are passed to the SourceGroupBy parameter.
     * Return the aggregated results of multiple fields in a flat structure. 
     */
    public static void groupByComposite(SyncClient client) {
        GroupByComposite.Builder compositeBuilder = GroupByBuilders
                .groupByComposite("groupByComposite")
                .size(2000)
                .addSources(GroupByBuilders.groupByField("groupByField", "Col_Keyword")
                        .addGroupBySorter(GroupBySorter.groupKeySortInAsc()).build())
                .addSources(GroupByBuilders.groupByHistogram("groupByHistogram", "Col_Long")
                        .addGroupBySorter(GroupBySorter.groupKeySortInAsc())
                        .interval(5)
                        .build())
                .addSources(GroupByBuilders.groupByDateHistogram("groupByDateHistogram", "Col_Date")
                        .addGroupBySorter(GroupBySorter.groupKeySortInAsc())
                        .interval(5, DateTimeUnit.DAY)
                        .timeZone("+05:30").build());
    
        SearchRequest searchRequest = SearchRequest.newBuilder()
                .indexName("<SEARCH_INDEX_NAME>")
                .tableName("<TABLE_NAME>")
                .returnAllColumnsFromIndex(true)
                .searchQuery(SearchQuery.newBuilder()
                        .addGroupBy(compositeBuilder.build())
                        .build())
                .build();
    
        SearchResponse resp = client.search(searchRequest);
    
        while (true) {
            if (resp.getGroupByResults() == null || resp.getGroupByResults().getResultAsMap().size() == 0) {
                System.out.println("groupByComposite Result is null or empty");
                return;
            }
    
            GroupByCompositeResult result = resp.getGroupByResults().getAsGroupByCompositeResult("groupByComposite");
    
            if(!result.getSourceNames().isEmpty()) {
                for (String sourceGroupByNames: result.getSourceNames()) {
                    System.out.printf("%s\t", sourceGroupByNames);
                }
                System.out.print("rowCount\t\n");
            }
    
    
            for (GroupByCompositeResultItem item : result.getGroupByCompositeResultItems()) {
                for (String value : item.getKeys()) {
                    String val = value == null ?  "NULL" : value;
                    System.out.printf("%s\t", val);
    
                }
                System.out.printf("%d\t\n", item.getRowCount());
            }
    
            // Use the Token parameter to page through groups.
            if (result.getNextToken() != null) {
                searchRequest.setSearchQuery(
                        SearchQuery.newBuilder()
                                .addGroupBy(compositeBuilder.nextToken(result.getNextToken()).build())
                                .build()
                );
                resp = client.search(searchRequest);
            } else {
                break;
            }
        }
    }

Group by range

The aggregation method that can be used to group query results based on the value ranges of a field. Field values that are within a specified range are grouped together. The number of values in each range is returned.

  • Parameters

    Parameter

    Description

    groupByName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    fieldName

    The name of the field that is used to perform the aggregation operation. Only the LONG and DOUBLE types are supported.

    range[double_from, double_to)

    The value ranges for grouping.

    The value range can start from Double.MIN_VALUE and end with Double.MAX_VALUE.

    subAggregation and subGroupBy

    The sub-aggregation operation. You can perform the sub-aggregation operation based on the grouping results.

    For example, after you group query results by sales volume and by province, you can obtain the province that accounts for the largest proportion of sales volume in a specific range. You must specify a value for GroupByField in GroupByRange to perform this query.

  • Example

    /**
     * Group sales volumes based on ranges [0, 1000), [1000, 5000), and [5000, Double.MAX_VALUE) to obtain the sales volume in each range. 
     */
    public void groupByRange(SyncClient client) {
        // Create a query statement. 
        SearchRequest searchRequest = SearchRequest.newBuilder()
            .tableName("<TABLE_NAME>")
            .indexName("<SEARCH_INDEX_NAME>")
            .searchQuery(
                SearchQuery.newBuilder()
                    .query(QueryBuilders.matchAll())
                    .limit(0)
                    .addGroupBy(GroupByBuilders
                        .groupByRange("name1", "column_number")
                        .addRange(0, 1000)
                        .addRange(1000, 5000)
                        .addRange(5000, Double.MAX_VALUE)
                    )
                    .build())
            .build();
        // Execute the query statement. 
        SearchResponse resp = client.search(searchRequest);
        // Obtain the aggregation results. 
        for (GroupByRangeResultItem item : resp.getGroupByResults().getAsGroupByRangeResult("name1").getGroupByRangeResultItems()) {
    
            // Display the number of rows. 
            System.out.println(item.getRowCount());
        }
    }

Group by geographical location

The aggregation method that can be used to group query results based on distances from geographical locations to a central point. Query results in distances that are within a specified range are grouped together. The number of items in each range is returned.

  • Parameters

    Parameter

    Description

    groupByName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    fieldName

    The name of the field that is used for the aggregation operation. Only the GEOPOINT type is supported.

    origin(double lat, double lon)

    The longitude and latitude of the central point.

    double lat specifies the latitude of the central point. double lon specifies the longitude of the central point.

    range[double_from, double_to)

    The distance ranges that are used for grouping. Unit: meters.

    The value range can start from Double.MIN_VALUE and end with Double.MAX_VALUE.

    subAggregation and subGroupBy

    The sub-aggregation operation. You can perform the sub-aggregation operation based on the grouping results.

  • Example

    /**
     * Group users based on their geographical distances to a Wanda Plaza to obtain the number of users in each distance range. The distance ranges are [0, 1000), [1000, 5000), and [5000, Double.MAX_VALUE). Unit: meters. 
     */
    public void groupByGeoDistance(SyncClient client) {
        // Create a query statement. 
        SearchRequest searchRequest = SearchRequest.newBuilder()
            .tableName("<TABLE_NAME>")
            .indexName("<SEARCH_INDEX_NAME>")
            .searchQuery(
                SearchQuery.newBuilder()
                    .query(QueryBuilders.matchAll())
                    .limit(0)
                    .addGroupBy(GroupByBuilders
                        .groupByGeoDistance("name1", "column_geo_point")
                        .origin(3.1, 6.5)
                        .addRange(0, 1000)
                        .addRange(1000, 5000)
                        .addRange(5000, Double.MAX_VALUE)
                    )
                    .build())
            .build();
        // Execute the query statement. 
        SearchResponse resp = client.search(searchRequest);
        // Obtain the aggregation results. 
        for (GroupByGeoDistanceResultItem item : resp.getGroupByResults().getAsGroupByGeoDistanceResult("name1").getGroupByGeoDistanceResultItems()) {
           // Display the number of rows. 
            System.out.println(item.getRowCount());
        }
    }

Group by filter

The aggregation method that can be used to filter the query results and group them based on each filter to obtain the number of matching results. Results are returned in the order in which the filters are specified.

  • Parameters

    Parameter

    Description

    groupByName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    filter

    The filters that can be used for the query. Results are returned in the order in which the filters are specified.

    subAggregation and subGroupBy

    The sub-aggregation operation. You can perform the sub-aggregation operation based on the grouping results.

  • Example

    /**
     * Specify the following filters to obtain the number of items that match each filter: The sales volume exceeds 100, the place of origin is Zhejiang, and the description contains Hangzhou. 
     */
    public void groupByFilter(SyncClient client) {
        // Create a query statement. 
        SearchRequest searchRequest = SearchRequest.newBuilder()
            .tableName("<TABLE_NAME>")
            .indexName("<SEARCH_INDEX_NAME>")
            .searchQuery(
                SearchQuery.newBuilder()
                    .query(QueryBuilders.matchAll())
                    .limit(0) 
                    .addGroupBy(GroupByBuilders
                        .groupByFilter("name1")
                        .addFilter(QueryBuilders.range("number").greaterThanOrEqual(100))
                        .addFilter(QueryBuilders.term("place","Zhejiang Province"))
                        .addFilter(QueryBuilders.match("text","Hangzhou"))
                    )
                    .build())
            .build();
        // Execute the query statement. 
        SearchResponse resp = client.search(searchRequest);
        // Obtain the aggregation results based on the order of filters. 
        for (GroupByFilterResultItem item : resp.getGroupByResults().getAsGroupByFilterResult("name1").getGroupByFilterResultItems()) {
            // Display the number of rows. 
            System.out.println(item.getRowCount());
        }
    }

Group by histogram

The aggregation method can be used to group query results based on specific data intervals. Field values that are within the same range are grouped together. The value range of each group and the number of values in each group are returned.

  • Parameters

    Parameter

    Description

    groupByName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    fieldName

    The name of the field that is used to perform the aggregation operation. Only the LONG and DOUBLE types are supported.

    interval

    The data interval that is used to obtain aggregation results.

    fieldRange[min,max]

    The range that is used together with the interval parameter to limit the number of groups. The value that is calculated by using the (fieldRange.max-fieldRange.min)/interval formula cannot exceed 2000.

    minDocCount

    The minimum number of rows. If the number of rows in a group is less than the minimum number of rows, the aggregation results for the group are not returned.

    missing

    The default value for the field on which the aggregation operation is performed, which is applied to rows in which the field value is empty.

    • If you do not specify a value for missing, the row is ignored.

    • If you specify a value for missing, the value of this parameter is used as the field value of the row.

  • Example

    /**
     * Collect statistics on the distribution of users by age group. 
     */
    public static void groupByHistogram(SyncClient client) {
        // Create a query statement. 
        SearchRequest searchRequest = SearchRequest.newBuilder()
            .tableName("<TABLE_NAME>")
            .indexName("<SEARCH_INDEX_NAME>")
            .searchQuery(
                SearchQuery.newBuilder()
                    .addGroupBy(GroupByBuilders
                        .groupByHistogram("groupByHistogram", "age")
                        .interval(10)
                        .minDocCount(0L)
                        .addFieldRange(0, 99))
                    .build())
            .build();
        // Execute the query statement. 
        SearchResponse resp = ots.search(searchRequest);
        // Obtain the results that are returned when the aggregation operation is performed. 
        GroupByHistogramResult results = resp.getGroupByResults().getAsGroupByHistogramResult("groupByHistogram");
        for (GroupByHistogramItem item : results.getGroupByHistogramItems()) {
            System.out.println("key:" + item.getKey().asLong() + " value:" + item.getValue());
        }
    }

Group by date histogram

The aggregation method that can be used to group query results based on specific date intervals. Field values that are within the same range are grouped together. The value range of each group and the number of values in each group are returned.

Important

This feature is supported by Tablestore SDK for Java V5.16.1 and later.

  • Parameters

    Parameter

    Description

    groupByName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    fieldName

    The name of the field that is used to perform the aggregation operation. Only the DATE type is supported.

    interval

    The data interval that is used to obtain aggregation results.

    fieldRange[min,max]

    The range that is used together with the interval parameter to limit the number of groups. The value that is calculated by using the (fieldRange.max-fieldRange.min)/interval formula cannot exceed 2000.

    minDocCount

    The minimum number of rows. If the number of rows in a group is less than the minimum number of rows, the aggregation results for the group are not returned.

    missing

    The default value for the field on which the aggregation operation is performed, which is applied to rows in which the field value is empty.

    • If you do not specify a value for missing, the row is ignored.

    • If you specify a value for missing, the value of this parameter is used as the field value of the row.

    timeZone

    The time zone in the +hh:mm or -hh:mm format, such as +08:00 or -09:00. This parameter is required only when the field is of the DATE type.

    If this parameter is not specified for the field of the DATE type, the offset of N hours may occur in the aggregation results. You can specify the timeZone parameter to prevent this issue.

  • Example

    /**
     * Collect statistics on the daily distribution of data of the col_date field from 10:00 on May 1, 2017 to 13:00:00 May 21, 2017. 
     */
    public static void groupByDateHistogram(SyncClient client) {
        // Create a query statement. 
        SearchRequest searchRequest = SearchRequest.newBuilder()
        .returnAllColumns(false)
        .indexName("<SEARCH_INDEX_NAME>")
        .tableName("<TABLE_NAME>")
        .searchQuery(
            SearchQuery.newBuilder()
            .query(QueryBuilders.matchAll())
            .limit(0)
            .getTotalCount(false)
            .addGroupBy(GroupByBuilders
                        .groupByDateHistogram("groupByDateHistogram", "col_date")
                        .interval(1, DateTimeUnit.DAY)
                        .minDocCount(1)
                        .missing("2017-05-01 13:01:00")
                        .fieldRange("2017-05-01 10:00", "2017-05-21 13:00:00"))
            .build())
        .build();
        // Execute the query statement. 
        SearchResponse resp = ots.search(searchRequest);
        // Obtain the results that are returned when the aggregation operation is performed. 
        List<GroupByDateHistogramItem> items = resp.getGroupByResults().getAsGroupByDateHistogramResult("groupByDateHistogram").getGroupByDateHistogramItems();
        for (GroupByDateHistogramItem item : items) {
            System.out.printf("millisecondTimestamp:%d, count:%d \n", item.getTimestamp(), item.getRowCount());
        }
    }

Count

After you group query results, you can query the rows in each group. This method can be used in a similar manner as ANY_VALUE(field) in MySQL.

Note

When you query the rows that are obtained from the results of an aggregation operation in each group, the returned results contain only the primary key information if the search index contains the NESTED, GEOPOINT, or ARRAY field. To obtain the required field, you must query the data table.

  • Parameters

    Parameter

    Description

    aggregationName

    The unique name of the aggregation operation. You can query the results of a specific aggregation operation based on this name.

    limit

    The maximum number of rows that can be returned for each group. By default, only one row of data is returned.

    sort

    The sorting method that is used to sort data in groups.

    columnsToGet

    The fields that you want to return. Only fields in search indexes are supported. ARRAY, DATE, GEOPOINT, and NESTED fields are not supported.

    The value of this parameter is the same as the value of the columnsToGet parameter in SearchRequest. You need to only specify a value for the columnsToGet parameter in SearchRequest.

  • Example

    /**
     * An activity application form of a school contains fields in which information such as the names of students, classes, head teachers, and class presidents can be specified. You can group students by class to view the application statistics and the property information of each class. 
     * Equivalent SQL statement: select className, teacher, monitor, COUNT(*) as number from table GROUP BY className. 
     */
    public void testTopRows(SyncClient client) {
        SearchRequest searchRequest = SearchRequest.newBuilder()
                .indexName("<SEARCH_INDEX_NAME>")
                .tableName("<TABLE_NAME>")
                .searchQuery(
                        SearchQuery.newBuilder()
                                .query(QueryBuilders.matchAll())
                                .limit(0) 
                                .addGroupBy(GroupByBuilders.groupByField("groupName", "className")
                                        .size(5)  // Specify the number of groups that you want to return.  For information about the maximum value that you can specify for the number of groups to return, see the description of the number of groups returned by GroupByField in the "Search index limits" topic. 
                                        .addSubAggregation(AggregationBuilders.topRows("topRowsName")
                                                .limit(1)
                                                .sort(new Sort(Arrays.asList(new FieldSort("teacher", SortOrder.DESC)))) // Sort rows by teacher in descending order.
                                        )
                                )
                                .build())
                .addColumnsToGet(Arrays.asList("teacher", "monitor"))
                .build();
        SearchResponse resp = client.search(searchRequest);
        List<GroupByFieldResultItem> items = resp.getGroupByResults().getAsGroupByFieldResult("groupName").getGroupByFieldResultItems();
        for (GroupByFieldResultItem item : items) {
            String className = item.getKey();
            long number = item.getRowCount();
            List<Row> topRows = item.getSubAggregationResults().getAsTopRowsAggregationResult("topRowsName").getRows();
            Row row = topRows.get(0);
            String teacher = row.getLatestColumn("teacher").getValue().asString();
            String monitor = row.getLatestColumn("monitor").getValue().asString();
        }
    }

Nesting

GroupBy supports nesting. You can perform sub-aggregation operations by using GroupBy.

You can use nesting to perform sub-aggregation operations in a group. For example, you can perform nesting aggregation operations up to two levels.

  • GroupBy + SubGroupBy: Items are grouped by province and by city to obtain data for each city in each province.

  • GroupBy + SubAggregation: Items are grouped by province to obtain the maximum value of a metric for each province.

Note

To ensure high performance of complex GroupBy operations, you can specify only a small number of levels for nesting. For more information, see Search index limits.

public void subGroupBy(SyncClient client) {
    // Create a query statement. 
    SearchRequest searchRequest = SearchRequest.newBuilder()
            .indexName("index_name")
            .tableName("table_name")
            .returnAllColumns(true)
            .searchQuery(
                    SearchQuery.newBuilder()
                            .query(QueryBuilders.match("textField", "hello"))
                            .limit(10)
                            .addAggregation(AggregationBuilders.min("name1", "fieldName1"))
                            .addAggregation(AggregationBuilders.max("name2", "fieldName2"))
                            .addGroupBy(GroupByBuilders
                                    .groupByField("name3", "fieldName3")
                                    .addSubAggregation(AggregationBuilders.max("subName1", "fieldName4"))
                                    .addSubAggregation(AggregationBuilders.sum("subName2", "fieldName5"))
                                    .addSubGroupBy(GroupByBuilders
                                            .groupByRange("subName3", "fieldName6")
                                            .addRange(12, 90)
                                            .addRange(100, 900)
                                    ))
                            .build())
            .build();
    // Execute the query statement. 
    SearchResponse resp = client.search(searchRequest);
    // Obtain the maximum and minimum values for the first level. 
    AggregationResults aggResults = resp.getAggregationResults();
    System.out.println(aggResults.getAsMinAggregationResult("name1").getValue());
    System.out.println(aggResults.getAsMaxAggregationResult("name2").getValue());

    // Obtain the GroupByField results of the first level and the results of the aggregations that are nested in GroupByField. 
    GroupByFieldResult results = resp.getGroupByResults().getAsGroupByFieldResult("name3");
    for (GroupByFieldResultItem item : results.getGroupByFieldResultItems()) {
        System.out.println("count:" + item.getRowCount());
        System.out.println("key:" + item.getKey());

        // Obtain the sub-aggregation results. 
        // Display the maximum value that is obtained from the results of the sub-aggregation operation. 
        System.out.println(item.getSubAggregationResults().getAsMaxAggregationResult("subName1"));
        // Display the sum that is obtained from the results of the sub-aggregation operation. 
        System.out.println(item.getSubAggregationResults().getAsSumAggregationResult("subName2"));
        // Display the GroupByRange values that are obtained from the results of the sub-aggregation operation. 
        GroupByRangeResult subResults = item.getSubGroupByResults().getAsGroupByRangeResult("subName3");
        for (GroupByRangeResultItem subItem : subResults.getGroupByRangeResultItems()) {
            System.out.println(String.format("from:%s, to:%s, count:%s", subItem.getFrom(), subItem.getTo(), subItem.getRowCount()));
        }
    }
}

Multiple aggregations

You can perform multiple aggregation operations.

Note

If you perform multiple complex aggregation operations at the same time, a long period of time may be required.

Combine multiple aggregations

public void multipleAggregation(SyncClient client) {
    // Create a query statement. 
    SearchRequest searchRequest = SearchRequest.newBuilder()
        .tableName("<TABLE_NAME>")
        .indexName("<SEARCH_INDEX_NAME>")
        .searchQuery(
            SearchQuery.newBuilder()
                .query(QueryBuilders.matchAll())
                .limit(0) 
                .addAggregation(AggregationBuilders.min("name1", "long"))
                .addAggregation(AggregationBuilders.sum("name2", "long"))
                .addAggregation(AggregationBuilders.distinctCount("name3", "long"))
                .build())
        .build();
    // Execute the query statement. 
    SearchResponse resp = client.search(searchRequest);
    // Obtain the minimum value from the results of the aggregation operation. 
    System.out.println(resp.getAggregationResults().getAsMinAggregationResult("name1").getValue());
    // Obtain the sum from the results of the aggregation operation. 
    System.out.println(resp.getAggregationResults().getAsSumAggregationResult("name2").getValue());
    // Obtain the number of distinct values from the results of the aggregation operation. 
    System.out.println(resp.getAggregationResults().getAsDistinctCountAggregationResult("name3").getValue());
}

Combine Aggregation and GroupBy

public void multipleGroupBy(SyncClient client) {
    // Create a query statement. 
    SearchRequest searchRequest = SearchRequest.newBuilder()
        .tableName("<TABLE_NAME>")
        .indexName("<SEARCH_INDEX_NAME>")
        .searchQuery(
            SearchQuery.newBuilder()
                .query(QueryBuilders.matchAll())
                .limit(0)
                .addAggregation(AggregationBuilders.min("name1", "long"))
                .addAggregation(AggregationBuilders.sum("name2", "long"))
                .addAggregation(AggregationBuilders.distinctCount("name3", "long"))
                .addGroupBy(GroupByBuilders.groupByField("name4", "type"))
                .addGroupBy(GroupByBuilders.groupByRange("name5", "long").addRange(1, 15))
                .build())
        .build();
    // Execute the query statement. 
    SearchResponse resp = client.search(searchRequest);
    // Obtain the minimum value from the results of the aggregation operation. 
    System.out.println(resp.getAggregationResults().getAsMinAggregationResult("name1").getValue());
    // Obtain the sum from the results of the aggregation operation. 
    System.out.println(resp.getAggregationResults().getAsSumAggregationResult("name2").getValue());
    // Obtain the number of distinct values from the results of the aggregation operation. 
    System.out.println(resp.getAggregationResults().getAsDistinctCountAggregationResult("name3").getValue());
    // Obtain the values of GroupByField from the results of the aggregation operation. 
    for (GroupByFieldResultItem item : resp.getGroupByResults().getAsGroupByFieldResult("name4").getGroupByFieldResultItems()) {
        // Display the keys. 
        System.out.println(item.getKey());
        // Display the number of rows. 
        System.out.println(item.getRowCount());
    }
    // Obtain the GroupByRange values from the results of the aggregation operation. 
    for (GroupByRangeResultItem item : resp.getGroupByResults().getAsGroupByRangeResult("name5").getGroupByRangeResultItems()) {
        // Display the number of rows. 
        System.out.println(item.getRowCount());
    }
}

Appendix: different methods for multi-field grouping

If you want to group query results by multiple fields, you can use the groupBy parameter in nested mode or use the GroupByComposite parameter. The following table describes the difference between the groupBy parameter in nested mode and the groupByComposite parameter.

Feature

groupBy (nested)

groupByComposite

size

2000

2000

Limits on fields

Up to 5 layers supported

Up to 32 fields supported

Pagination

Not supported

Supported by using the nextToken parameter

Sort group items.

  • In alphabetical order or reverse alphabetical order

  • By row count in ascending order or row count in descending order

  • By the values that are obtained from sub-aggregation results in ascending order or the values that are obtained from sub-aggregation results in descending order

In alphabetical order or reverse alphabetical order

Supports aggregation

Yes

Yes

Compatibility

For fields of the DATE type, the query results are returned in the specified format.

For fields of the DATE type, the query results are returned as timestamp strings.