Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Optimizing High CPU Usage for Elasticsearch on Alibaba Cloud

Optimizing High CPU Usage for Elasticsearch on Alibaba Cloud

This guide provides practical tips and code examples to ensure your clusters maintain peak performance and reliability.

Introduction

In this series of articles, we address common Elasticsearch cluster issues and provide solutions for efficient troubleshooting. Our focus now shifts to a critical problem many administrators face: sudden spikes in CPU usage within Elasticsearch clusters. Let's dive into understanding and resolving this challenge using Alibaba Cloud Elasticsearch.

The Implications of High CPU Usage in Elasticsearch

High CPU utilization in Elasticsearch often indicates that one or more thread pools cannot support ongoing business demands. Thread pools manage the CPU resources for concurrent operations, and if they are exhausted, Elasticsearch will reject requests associated with the affected thread pool. For instance, if the search thread pool is overloaded, searches will be denied until additional threads become available.

To illustrate, consider a search thread pool scenario:

  • With fewer requests, the thread pool can handle the load efficiently.
  • As requests increase, the thread pool queue starts to fill up, requiring requests to wait.
  • An overwhelming number of requests exceed the thread and queue capacity, resulting in errors.

Diagnosing High CPU Usage in Elasticsearch

Checking CPU Usage

Retrieve current CPU usage for each node by using the cat nodes API.

GET _cat/nodes?v=true&s=cpu:desc

You can visualize CPU usage via Kibana Stack Monitoring for an intuitive monitoring experience.

Identifying Hot Threads

For nodes exhibiting high CPU usage, examine resource-intensive threads using the hot threads API.

GET _nodes/my-node,my-other-node/hot_threads

This API returns details of any hot threads as plain text.

Practical Solutions to Lower CPU Usage

The most common reasons for high CPU usage and their solutions include:

Expanding the Cluster

  • Heavy data indexing and search loads can overwhelm small thread pools.
  • Add more nodes or upscale existing ones for better workload handling.

Distributing Bulk Requests

  • Large bulk writes or multiple search requests require substantial CPU resources.
  • Opt for smaller requests spaced out over time to reduce strain.

Canceling Long-running Searches

  • Ongoing searches can block threads in the search thread pool.
  • Use the tasks management API to examine these searches.
GET _tasks?actions=*search&detailed

Responding to requests with detailed querying can be managed using the _cancel API to free up resources:

POST _tasks/oTUltX4IQMOUUVeiohTt8A:464/_cancel

Avoiding Resource-intensive Searches

To minimize resource consumption, strategies include:

  • Limiting script searches.
  • Avoiding queries like fuzzy, regexp, prefix, and wildcard when possible.
  • Refraining from range queries on text and keyword types.
  • Avoiding join operations.
  • Utilizing index.max_result_window index settings to lower the size limit.
  • Using search.max_buckets cluster settings to reduce maximum allowable aggregation buckets.
  • Disabling expensive queries with search.allow_expensive_queries cluster settings.

Conclusion

Proactive monitoring and alerting for your Elasticsearch clusters are critical to preventing high CPU usage issues. Maximizing thread pool and queue usage relative to node CPU cores is key to maintaining balance. If you've encountered high CPU utilization, how did you address it? Share your experiences in the comments.

Ready to start your journey with Elasticsearch on Alibaba Cloud? Explore our tailored Cloud solutions and services to take the first step towards transforming your data into a visual masterpiece.

Please Click here, Embark on Your 30-Day Free Trial

Alibaba Cloud Elasticsearch Thread Pools High CPU Usage Cluster Optimization
0 1 0
Share on

Read previous post:

A Pro Guide to Upgrading to Elasticsearch 8.X

Read next post:

How to Mitigate High Disk Usage in Elasticsearch

Data Geek

97 posts | 4 followers

You may also like

Comments

Data Geek

97 posts | 4 followers

Related Products

More Posts by Data Geek

See All