Elasticsearch:Query logs

Limits

• You can view the access logs only of Elasticsearch V6.7.0 and V7.10 clusters in the Elasticsearch console.

• You can view the audit logs only of Elasticsearch clusters of V7.X or later in the following regions in the Elasticsearch console.

  Country or district	Region
  China	China (Beijing), China (Hangzhou), China (Shanghai), and China (Zhangjiakou)
  Asia Pacific	Singapore, Malaysia (Kuala Lumpur), Indonesia (Jakarta), and Japan (Tokyo)
  Europe & Americas	US (Virginia), US (Silicon Valley), Germany (Frankfurt), and UK (London)

Procedure

1. Log on to the Alibaba Cloud Elasticsearch console.

2. In the left-side navigation pane, click Elasticsearch Clusters.

3. Navigate to the desired cluster.

   1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.

   2. On the Elasticsearch Clusters page, find the cluster and click its ID.

4. In the left-side navigation pane of the page that appears, click Logs. Then, you can view the logs of the cluster.

   Alibaba Cloud Elasticsearch provides the following types of logs for an Elasticsearch cluster: cluster logs, slow search logs, slow indexing logs, garbage collection (GC) logs, access logs, and audit logs. The following table describes each type of log and their use scenarios. For more information about the logs, see Common types of logs.

   Log type	Description	Use scenario
   Cluster log	This type of log records the health status of an Elasticsearch cluster and the information about read and write operations performed on the cluster. For example, logs for write operations include logs generated for index creation, index mapping updates, and full write queues, and logs for read operations include logs generated for query queues and query exceptions.	If you want to view the status of each node in an Elasticsearch cluster or view information about read and write operations performed on the cluster, such as information about network connectivity between nodes, full GC, index creation or deletion, or errors reported for queries, you can view the cluster logs of the cluster. Important If errors occur in your business, we recommend that you first view the cluster logs and monitoring data of your cluster to troubleshoot performance or configuration issues.
   Slow search log	This type of log records information about slow read operations. If the time that is required to complete a read operation exceeds a specific threshold, the read operation is considered as a slow read operation, and the system displays information about the operation in slow search logs. The threshold is configured in the scenario-based index template of your Elasticsearch cluster. By default, the configurations in the template are optimal, and you can directly apply the template. For more information about the scenario-based index template of an Elasticsearch cluster, see Modify the index template of a cluster.	If a long period of time is required to complete a read operation in your business, you can view slow search logs to troubleshoot the issue. Read operations that require a longer period of time to complete consume more cluster resources. If a large number of slow logs are generated for a cluster, check the resource usage and loads of the cluster to identify the items that cause bottleneck issues. Then, replenish resources for the cluster based on the items or use the aliyun-qos plug-in to perform throttling for the cluster to ensure cluster stability.
   Slow indexing log	This type of log records information about slow write operations. If the time that is required to complete a write operation exceeds a specific threshold, the write operation is considered as a slow write operation, and the system displays information about the operation in slow indexing logs. The threshold is configured in the scenario-based index template of your Elasticsearch cluster. By default, the configurations in the template are optimal, and you can directly apply the template. For more information about the scenario-based index template of an Elasticsearch cluster, see Modify the index template of a cluster.	If a long period of time is required to complete a write operation in your business, you can view slow indexing logs to troubleshoot the issue. Write operations that require a longer period of time to complete consume more cluster resources. If a large number of slow logs are generated for a cluster, check the resource usage and loads of the cluster to identify the items that cause bottleneck issues. Then, replenish resources for the cluster based on the items or use the aliyun-qos plug-in to perform throttling for the cluster to ensure cluster stability.
   GC log	This type of log records information about GC for an Elasticsearch cluster. GC logs contain information about GC triggered by JVM heap memory usage. You can obtain GC details, including information about GC based on the Old GC, Concurrent Mark Sweep (CMS) GC, Full GC, and Minor GC mechanisms.	If a performance bottleneck occurs on an Elasticsearch cluster, you can view the GC logs of the cluster to obtain GC details and check whether GC operations require a long period of time to complete or are frequently performed. If such GC operations exist, replenish resources for the cluster at your earliest opportunity or use the aliyun-qos plug-in to perform throttling for the cluster to ensure cluster stability. Important By default, an Alibaba Cloud Elasticsearch cluster uses the CMS garbage collector. If the volume of data stored on each data node in a cluster is greater than or equal to 32 GB, we recommend that you use the G1 garbage collector to improve GC efficiency. For more information, see Configure a garbage collector.
   Access log	This type of log records information about access to an Elasticsearch cluster. The access logs of an Elasticsearch cluster contain details of requests initiated by using the RestSearchAction API, such as the URIs of the requests, the sizes of the request bodies, and the time when the requests are initiated. Important You can view the access logs only of Elasticsearch V6.7.0 and V7.10 clusters in the Elasticsearch console. Elasticsearch access logs do not support the following query scenarios: SQL queries, multi queries, scroll queries, and queries triggered by some Kibana visualization tools. If you want to obtain more complete information about query and write requests for your cluster, we recommend that you enable the Audit Log Indexing feature for the cluster. For more information, see Configure the Audit Log Indexing feature.	If you want to have a command of clients that send query requests to an Elasticsearch cluster, you can view the access logs of the cluster.
   Audit log	This type of log records information about auditing of operations that are performed on an Elasticsearch cluster, such as the create, delete, modify, and query operations. Important You can view the audit logs only of Elasticsearch clusters of V7.X or later in regions listed in Limits in the Elasticsearch console. For an Elasticsearch cluster other than the preceding clusters, you can enable the Audit Log Indexing feature for the cluster in the YML Configuration section of the Cluster Configuration page. Then, the audit logs of the cluster are written to indexes in the cluster. These indexes are named in the .security_audit_log-* format. You can query such indexes in the Kibana console of the cluster to view the audit logs. For more information, see Configure the YML file. Before you view the audit logs of an Elasticsearch cluster, you must click Log Configuration on the Logs page of the cluster in the Elasticsearch console and turn on the Audit Log Collection switch. By default, the system collects logs from the following types of audit events: access_denied, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, and run_as_granted. If you want to change the types of audit events from which you want to collect logs, you must modify the xpack.security.audit.logfile.events.include parameter in the YML configuration file of your Elasticsearch cluster. For more information, see Configure the Audit Log Indexing feature.	If your identity fails to be verified when you access an Elasticsearch cluster, a connection to the cluster is denied, you need to view the access events of the cluster, or you need to check whether suspicious events exist, you can troubleshoot the issue or perform the desired operation based on the audit logs of the cluster. A modification to data access permissions or user security configurations may be a suspicious event.

5. On a tab of the Logs page, enter a query string, select the start time and end time, and then click Search.

   After you click Search, the logs that match your query string are displayed.

   • You can query logs that were generated during the previous seven days. By default, the logs are displayed by time in descending order.

   • The Lucene query syntax is supported. For more information, see Query string syntax.

   • AND in the query string must be uppercase.

   • If you do not specify an end time, the current system time is used as the end time. If you do not specify a start time, the start time is 1 hour earlier than the end time.

   In this example, the logs that meet the following conditions are queried on the Cluster Log tab: The value of the level field is INFO, the value of the host field is 172.16.xx.xx, and the value of the content field contains the health keyword. In this case, the query string is host:172.16.xx.xx AND content:health AND level:info.

   Important

   • Alibaba Cloud Elasticsearch can return a maximum of 10,000 logs for each query.

     If the returned logs do not contain the logs that you want to view, you can shorten the specified time range and perform another query.

   • A maximum of 10,000 characters can be displayed in a log.

Common types of logs

Cluster logs

Slow logs

Slow logs include slow search logs and slow indexing logs. If the time that is required to complete a read or write operation exceeds a specific threshold, a slow log is generated for the operation. The Search Slow Log tab displays slow search logs, and the Indexing Slow Log tab displays slow indexing logs. By default, slow log collection is enabled. If unbalanced loads, read or write exceptions, or slow data processing issues occur on your cluster, you can troubleshoot issues based on the slow logs.

GC logs

Access logs

Audit logs

The Audit Log tab displays the audit logs generated for operations that are performed on an Elasticsearch cluster, such as the create, delete, modify, and query operations. By default, audit log collection is disabled. To enable audit log collection and view audit logs, perform the following steps:

1. On the Logs page of an Elasticsearch cluster, click Log Configuration on the right side.

2. In the Log Configuration dialog box, turn on Audit Log Collection.

   Important

   • After you turn on Audit Log Collection, you can view the audit logs of the cluster on the Audit Log tab of the Logs page. If you want to change the types of audit events from which you want to collect logs, modify the xpack.security.audit.logfile.events.include parameter in the YML configuration file of the cluster. For more information, see Configure the Audit Log Indexing feature.

   • After you turn on or off Audit Log Collection, the system restarts the cluster. The system uses the rolling restart method to restart a cluster. Before the restart, make sure that the cluster is in the Normal state (indicated by the color green), each primary shard of each index in the cluster has at least one replica shard, and the resource usage of the cluster is not high. If all of the preceding conditions are met, the cluster can still provide services during the restart. However, we recommend that you perform this operation during off-peak hours.

3. Read the risk warning, select the check box, and then click OK.

   The system restarts the cluster. You can view the restart progress in the Tasks dialog box. After the cluster is restarted, the system starts to collect audit logs.

   Important

   Audit logs occupy the disk space of your cluster. If the disk space that is occupied by audit logs is excessively large, the performance of the cluster may be affected. If you do not need to view the audit logs of the cluster, you can turn off Audit Log Collection in the Log Configuration dialog box.

4. On the Logs page, click the Audit Log tab and view the audit logs of the cluster.

   

References

ListSearchLog

FAQ

• Can I specify a retention period for the .security indexes of an Elasticsearch cluster?

• How do I store the logs of an Elasticsearch cluster on my on-premises machine?

• I am unable to view the search and update logs of an Elasticsearch cluster. What do I do?

• How do I configure slow log collection for and view the slow logs of an Elasticsearch cluster?

• How do I obtain the slow logs of an Elasticsearch cluster on a regular basis?