Tair (Redis® OSS-Compatible):Troubleshoot high CPU utilization on an instance

What factors can cause an abnormal increase in CPU utilization?

• CPU-intensive commands: commands that have a time complexity of O(N), where N is a large value. Examples: KEYS, HGETALL, MGET, MSET, HMSET, and HMGET. In most cases, a command that has a higher time complexity consumes more CPU resources. This increases CPU utilization.

  If an instance runs CPU-intensive commands, pending requests accumulate in the queue due to single-threading. This slows down the response of applications. In specific cases, an instance may be overwhelmed by pending requests. An application may be disconnected due to these requests timing out. In addition, requests may be directly forwarded to backend databases and cause a cache avalanche.

  Note

  For information about the time complexity of each command, visit the Redis official website.

• Hotkeys: If a single key or a subset of keys is accessed more frequently than other keys, a possible reason is that one or more hotkeys were generated. Hotkeys can consume a substantial amount of CPU resources. This affects the access latency of other keys. If hotkeys in a cluster instance are concentrated on specific data shards, CPU utilization skew may occur. In this case, the CPU utilization of specific shards is much higher than the CPU utilization of other shards.

• Large keys: Large keys consume more memory. Access to large keys significantly increases the CPU load and traffic. Large keys are more prone to becoming hotspots, which can result in high CPU utilization. If large keys are concentrated on specific data shards, the CPU utilization, bandwidth usage, and memory usage may be skewed.

• Short-lived connections: Frequent establishment of connections can lead to a substantial consumption of resources on an instance due to the overhead associated with connection handling.

• AOFs: By default, append-only file (AOF) persistence is enabled for instances. If an instance is under heavy load, writing data to AOFs leads to increased CPU utilization and higher overall response latency of the instance.

Scenarios of high CPU utilization

The following scenarios may result in high CPU utilization:

• During a specific period of time, the CPU utilization suddenly spikes to a high level, even reaching 100%. For information about the causes and solutions, see Sudden increase in CPU utilization.

• A data node in an instance has higher CPU utilization compared with other data nodes. For information about the causes and solutions, see Inconsistent CPU utilization across data nodes.

• A proxy node in an instance has higher CPU utilization compared with other proxy nodes. For information about the causes and solutions, see Inconsistent CPU utilization across proxy nodes.

Take appropriate measures to reduce the CPU utilization based on the scenario.

Sudden increase in CPU utilization

If the overall CPU utilization of an instance increases, perform the following steps to troubleshoot the issue:

Troubleshoot and disable CPU-intensive commands

Troubleshooting procedure

1. Use the performance monitoring feature to identify the period of time during which CPU utilization is high. For more information, see View performance monitoring data.

2. Use the following methods to identify CPU-intensive commands:

   • The latency insights feature records the latency of all commands and custom special events on an instance. You can identify CPU-intensive commands that have longer response times based on the latency information collected over a specific period of time across different nodes. For more information, see Latency insights.

     Figure 1. Usage example of latency insights

   • Slow query logs record commands that run longer than the specified threshold. You can identify CPU-intensive commands based on the slow queries and their execution durations that are collected over a specific period of time across different nodes. For more information, see Query slow logs.

     Figure 2. Slow log query example

Solutions

• Evaluate and disable high-risk commands and CPU-intensive commands, such as FLUSHALL, KEYS, and HGETALL. For more information, see Disable high-risk commands.

• Optional: Use one of the following methods to change the instance architecture based on your business requirements:

  • Change the architecture of the instance to read/write splitting to evenly distribute CPU-intensive commands or application requests. For more information about the read/write splitting architecture, see Read/write splitting instances.

  • Convert the instance into a DRAM-based instance and use the multi-threading feature of DRAM-based instances to reduce the CPU utilization of the instance.

  Note

  For information about how to change the architecture and series type of an instance, see Change the configurations of an instance.

Troubleshoot and optimize short-lived connections

Troubleshooting procedure

1. Use the performance monitoring feature to identify the period of time during which CPU utilization is high. For more information, see View performance monitoring data.

2. On the performance monitoring dashboard, check whether CPU utilization is high and a large number of connections that do not have the expected queries per second (QPS) exist. If the preceding scenario is true, short-lived connections may exist. In this case, use one of the following solutions:

Solutions

• Change short-lived connections to persistent connections. For example, create a JedisPool connection pool. For more information, see Use a client to connect to an instance.

• Convert the instance into a DRAM-based instance to optimize the processing of short-lived connections.

Disable AOF persistence

By default, AOF persistence is enabled for instances. If an instance is under heavy load, frequent AOF operations may increase CPU utilization.

Disable AOF persistence if this does not adversely affect your business. In addition, you can back up the data of an instance during off-peak hours or during the maintenance window to minimize the impact.

Evaluate the service performance

The preceding methods are used to optimize the performance of an instance. If the average CPU utilization still exceeds 70% during business operations, the instance may have a performance bottleneck.

To resolve this issue, check whether commands and requests from application hosts that may degrade the instance performance exist. If such commands or requests exist, you must optimize your business system. If no such commands or requests exist but the CPU utilization is still high, we recommend that you upgrade the instance specifications to ensure business stability. You can also upgrade the instance to a cluster instance or read/write splitting instance. For more information about how to upgrade an instance, see Change the configurations of an instance.

Inconsistent CPU utilization across data nodes

If specific data shards in an instance that uses the cluster architecture or read/write splitting architecture have high CPU utilization whereas other data shards have low CPU utilization, perform the following steps to troubleshoot the issue:

Troubleshoot and optimize hotkeys

Troubleshooting procedure

1. Use the performance monitoring feature to identify the period of time during which CPU utilization is high. For more information, see View performance monitoring data.

2. On the History tab of the Real-time Key Statistics page, select the data nodes that have high CPU utilization, select time filter conditions based on Step 1, and then click Search. You can view the hotkeys that show high CPU utilization within the time range. For more information, see Use the real-time key statistics feature.

Solutions

• Enable the proxy query cache feature. After you enable this feature, proxy nodes cache the request and response data of hotkeys. If a proxy node receives a duplicate request during the validity period of the cached data, the proxy node directly returns a response to the client without the need to interact with backend data shards. This helps prevent skewed requests caused by hotkeys that receive a large number of read requests. For more information, see Use proxy query cache to address issues caused by hotkeys

  Note

  The proxy query cache feature is available only for DRAM-based and persistent memory-optimized instances.

• If hotkeys are generated from read requests, you can change the instance into a read/write splitting instance to reduce the read pressure imposed on each data shard of the instance.

  Note

  If a large number of requests are sent to a read/write splitting instance, a specific amount of latency is unavoidable, and dirty data may be read from the instance. Therefore, the read/write splitting architecture is not the optimal solution for scenarios that have high requirements for read and write capabilities and data consistency.

Troubleshoot and disable CPU-intensive commands

Troubleshooting procedure

1. Use the performance monitoring feature to identify the period of time during which CPU utilization is high. For more information, see View performance monitoring data.

2. Use the following methods to identify CPU-intensive commands:

   • The latency insights feature records the latency of all commands and custom special events on an instance. You can identify CPU-intensive commands that have longer response times based on the latency information collected over a specific period of time across different nodes. For more information, see Latency insights.

     Figure 1. Usage example of latency insights

   • Slow query logs record commands that run longer than the specified threshold. You can identify CPU-intensive commands based on the slow queries and their execution durations that are collected over a specific period of time across different nodes. For more information, see Query slow logs.

     Figure 2. Slow log query example

Solution

Evaluate and disable high-risk commands and CPU-intensive commands, such as FLUSHALL, KEYS, and HGETALL. For more information, see Disable high-risk commands.

Troubleshoot and optimize large keys

Troubleshooting procedure

1. Use the performance monitoring feature to identify the period of time during which CPU utilization is high. For more information, see View performance monitoring data.

2. On the Cache Analysis page, click Analyze. Select a data node that has high CPU utilization and click OK. You can view the large keys that show high CPU utilization within the time range. For more information, see Use the offline key analysis feature.

Solution

Split large keys into small keys based on actual business requirements to evenly distribute the request load.

Inconsistent CPU utilization across proxy nodes

If specific proxy nodes in an instance that uses the cluster architecture or read/write splitting architecture have high CPU utilization whereas other proxy nodes have low CPU utilization, perform the following steps to troubleshoot the issue:

Troubleshooting procedure

On the Proxy Node tab of the Performance Trends page, check whether connection usage is evenly distributed. For more information, see View performance trends.

Solutions

Perform one of the following operations based on whether connection usage is evenly distributed:

• If connection usage is evenly distributed, restart the client or proxy node on which business applications are deployed to redistribute connections. For more information about how to restart a proxy node, see Restart or rebuild proxy nodes.

• If connection usage is unevenly distributed, the uneven distribution is usually caused by a large scale of pipeline or batch operations. You can reduce the scale of pipeline or batch operations, for example, by splitting the operations into multiple smaller operations.