Lindorm:Count the number of rows in a table

To count the exact number of rows in a wide table, you can use HBase Shell to run the COUNT command. The COUNT command can be used to scan an entire wide table in batches and collect the results for statistical analysis. We recommend that you run the COUNT command on an Elastic Compute Service (ECS) client that is connected to the same virtual private cloud (VPC) as LindormTable. If you run the COUNT command over the Internet, the operation that is performed consumes a large amount of network bandwidth and the statistical analysis that is performed based on the operation is inefficient. The time required to scan a table varies based on the schema of the table. If you run the COUNT command to scan the entire table, less than 100,000 rows can be scanned per second. You can run the following command to count the number of rows in a wide table named table:

count 'table'

The following command output is returned:

RowCounter runs a MapReduce job in pseudo-distributed mode to perform the COUNT operation in an on-premises environment. By default, triggered operations are performed by using a single thread. In terms of the efficiency in running the COUNT command, this method is similar to the preceding method that uses HBase Shell. If you want to perform an efficient statistical analysis, use multiple threads. You can set Dmapreduce.local.map.tasks.maximum to the number of threads that you want to use to perform concurrent operations. Take note of the following items before you specify the number of threads:

• Set the number of threads to a value that is less than or equal to the number of regions in your wide table.

• If the number of threads increases, the loads on your cluster may become excessive and affect online services in a negative manner. We recommend that you specify the number of threads based on your business requirements.

Run the following commands in HBase Shell to count the number of rows in a specified wide table in different scenarios:

• Count the number of rows in a table named table.

  ./alihbase-2.0.18/bin/hbase org.apache.hadoop.hbase.mapreduce.RowCounter  "table"

• Count the number of rows in a table named table by using 16 concurrent threads.

  ./alihbase-2.0.18/bin/hbase org.apache.hadoop.hbase.mapreduce.RowCounter -Dmapreduce.local.map.tasks.maximum=16 "table"

• Count the number of rows in a table named table. This table is created in the ns namespace.

  ./alihbase-2.0.18/bin/hbase org.apache.hadoop.hbase.mapreduce.RowCounter  "ns:table"

The output is stored in the hbase.log file in the log directory.

You can execute the following statement to count the number of rows in a wide table named table:

SELECT COUNT(*) FROM table;

The following output is returned:

+--------+
| EXPR$0 |
+--------+
| 16000  |
+--------+

In the cluster management system of Lindorm, you can view the estimated number of rows in your wide table on the Overview page. The value in the EstimateRowCount column is calculated by accumulating the metadata of the number of rows in each data file. If you perform update operations and delete operations, the same row of data may be stored in multiple data files. The metadata of the number of rows in data files is collected when these files are generated. If you enable the time-to-live (TTL) feature, specific data in these files may expire. As a result, the estimated number of rows may be inaccurate. The estimated number of rows is a rough value. If the data in your wide table is not updated or deleted and the data does not expire, the estimated number of rows that is displayed on the Overview page is accurate. The estimated value can be used to check data integrity after historical data migration is complete.

To view the estimated number of rows in your wide table, perform the following operations: In the left-side navigation pane of the cluster management system, click Overview. In the Current IDC section, click View in the EstimateRowCount column corresponding to the wide table.