ApsaraDB for HBase is highly flexible and adaptable to multiple scenarios. The following section describes optimizations for read operations. During production, you may encounter errors such as Full GC, Out of Memory (OOM), Region in Transition (RIT), and high read latency. While upgrading your hardware may solve some of these problems, a more feasible approach is to optimize HBase based on your needs.
We divide the optimizations into:
Client optimization, server optimization, and platform optimization (implemented in ApsaraDB for HBase).
This greatly decreases the number of RPC calls between the client and server, significantly increasing throughput.
Result[] re= table.get(List<Get> gets)
Check whether an appropriate cache size is set for SCAN operations
SCAN operations require the server to return a large amount of data per request. When the client calls a SCAN operation, the server returns data to the client in batches. This is designed to reduce workloads on both the server and client when a large amount of data is transmitted at a time. The data is cached locally. The default maximum number of cached records is 100. Some SCAN operations may return large amounts of data (hundreds or tens of thousands) over RPC requests. We recommend that you extend the size of the cache based on your needs.
scan.setCaching(int caching) // Set this parameter to 1000 if SCAN operations may return large amounts of data.
Request a specified column family or column name
ApsaraDB for HBase is a column-oriented database. Data is stored based on column families, where each column family is a separate database. We recommend that you specify the column family or column name when you perform queries to reduce Input/Output (I/O).
Disable caching when you access ApsaraDB for HBase offline
When you access ApsaraDB for HBase offline, the entire data is read at one time. In this case, caching is not required. We recommend that you disable BlockCache for offline reads.
scan.setBlockCache(false)
Server optimization
Request balancing
Check the status of the read workload during peak hours. You can view the status in the ApsaraDB for HBase console. If there is obvious hotspotting, the ultimate solution is to redesign the rowkey to balance the workloads. An intermediate solution is to split the hotspotting regions.
Whether the BlockCache is set properly
BlockCache is important for read performance as a read cache. If a lot of data is requested, we recommend that you use a server with a core-to-memory ratio of 1:4. For example, a machine with 8 cores and 32 GB memory or a machine with 16 cores and 64 GB memory. You can increase the value of BlockCache and decrease the value of Memstore.
You can set hfile.block.cache.size to 0.5 and set hbase.regionserver.global.memstore.size to 0.3 in the console of ApsaraDB for HBase, and then click Restart.
The number of HFiles
During read operations, HFiles need to be opened frequently. The number of I/O operations increases in proportion to the number of HFiles, increasing read latency. To reduce this effect, we recommend that you perform a major compaction during off-peak hours.
Whether compaction consumes a large amount of system resources
Compaction is mainly used to merge multiple smaller HFiles into one larger HFile. This operation improves the read performance of subsequent read operations, but also consumes large amounts of resources. Typically, a minor compaction does not consume a large amount of system resources unless the configuration is inappropriate. Do not perform a major compaction during peak hours. We recommend that you perform a major compaction during off-peak hours.
Whether the Bloomfilter is properly set
Bloomfilter is mainly used to filter HFiles during queries to avoid unnecessary I/O operations. Bloomfilter can improve the read performance. Generally, when you create a table, the default value of BLOOMFILTER is set to ROW.
Platform optimization
Whether the rate of data localization is too low ( The platform has been optimized by Alibaba Cloud.)
If you have local HFiles, we recommend that you use Short-Circuit Local Read. During restart or expand operations, ApsaraDB for HBase automatically merges regions that have been moved. This ensures that the localization rate is not negatively affected. Furthermore, performing a regular major compaction helps improve the localization rate.
Short-Circuit Local Read (enabled by default)
Normally, Hadoop Distributed File System (HDFS) performs read operations through DataNode. Short-Circuit Local Read can be enabled to bypass DataNode and allow the client to directly read local data.
Hedged Read (enabled by default)
The system prioritizes Short-Circuit Local Read to read local data. However, in some special cases, the local read operation may fail for a short time due to disk problems or network problems. The Hedged Read operation has been developed to solve this problem. The basic working principle of Hedged Read is as follows: when a client initiates a local read request, and no result is returned after a period of time, the client will send the same request to DataNodes. If a result is returned, all subsequent results are discarded.
Disable swap (disabled by default)
When there is insufficient physical memory, part of the hard disk space is used as swap. This operation may cause high latency. Swap is disabled by default for ApsaraDB for HBase. However, disabling swap will cause high anon-rss, and the page reclaim operation cannot reclaim enough pages. This may cause the kernel to become unresponsive. To this effect, ApsaraDB for HBase has taken the relevant isolation measures to avoid this situation.