Elasticsearch:Use SkyWalking to implement end-to-end monitoring on Elasticsearch

Prerequisites

• An Alibaba Cloud Elasticsearch cluster is created. In this topic, an Elasticsearch V7.4.0 cluster is used.

  For more information, see Create an Alibaba Cloud Elasticsearch cluster.

• A Linux server on which JDK 1.8.0 or later is installed is prepared.

  We recommend that you use an Alibaba Cloud Elastic Compute Service (ECS) instance. For more information about how to purchase an ECS instance, see Step 1: Create an ECS instance.

  Note

  For more information about how to install a JDK, see Step 3: Install the JDK. If the JDK is not correctly installed and you start SkyWalking to view logs, the error message "Java not found" or "java-xxx: No such file or directory" is reported.

• Ports 8080, 10800, 11800, and 12800 on the Linux server are not occupied.

• The firewall and Security-Enhanced Linux (SELinux) are disabled for the Linux server.

Procedure

1. Step 1: Download and install SkyWalking

2. Step 2: Configure SkyWalking to connect to the Elasticsearch cluster

3. Step 3: Verify the results

Step 1: Download and install SkyWalking

1. Download the SkyWalking package to the Linux server.

   An Elasticsearch V7.4.0 cluster is used in this topic. Therefore, you must select Binary Distribution for ElasticSearch 7. The following code provides the command that is used to download the package:

   wget https://archive.apache.org/dist/skywalking/7.0.0/apache-skywalking-apm-es7-7.0.0.tar.gz

2. Run the following command to decompress the package:

   tar -zxvf apache-skywalking-apm-es7-7.0.0.tar.gz

3. Run the following command to view the decompressed files:

   ls apache-skywalking-apm-bin-es7/

   The following result is returned:

   total 92
   agent
   bin
   config
   LICENSE
   licenses
   NOTICE
   oap-libs
   README.txt
   tools
   webapp

Step 2: Configure SkyWalking to connect to the Elasticsearch cluster

1. Run the following code to open the application.yml file in the config folder:

   cd apache-skywalking-apm-bin-es7/config/
   vi application.yml

2. Locate storage, change H2 to elasticsearch7, and configure the file based on the following instructions:

   storage:
     selector: ${SW_STORAGE:elasticsearch7}
     elasticsearch7:
       nameSpace: ${SW_NAMESPACE:"skywalking-index"}
       clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:es-cn-4591kzdzk000i****.public.elasticsearch.aliyuncs.com:9200}
       protocol: ${SW_STORAGE_ES_HTTP_PROTOCOL:"http"}
      # trustStorePath: ${SW_SW_STORAGE_ES_SSL_JKS_PATH:"../es_keystore.jks"}
      # trustStorePass: ${SW_SW_STORAGE_ES_SSL_JKS_PASS:""}
       enablePackedDownsampling: ${SW_STORAGE_ENABLE_PACKED_DOWNSAMPLING:true} # Hour and Day metrics will be merged into minute index.
       dayStep: ${SW_STORAGE_DAY_STEP:1} # Represent the number of days in the one minute/hour/day index.
       user: ${SW_ES_USER:"elastic"}
       password: ${SW_ES_PASSWORD:"es_password"}

   Note

   SkyWalking stores data in H2 by default. However, H2 does not support persistent data storage. Therefore, you must change H2 to Elasticsearch.

   Parameter	Description
   selector	The storage selector. For this example, set the value to elasticsearch7.
   nameSpace	The namespace. The value of this parameter is used as the prefix for the names of all indexes in the Elasticsearch cluster.
   clusterNodes	The endpoint of the Elasticsearch cluster. The Elasticsearch cluster is not in the same virtual private cloud (VPC) as SkyWalking. You must use the public endpoint to access the Elasticsearch cluster. For more information about how to obtain the public endpoint of the Elasticsearch cluster, see View the basic information of a cluster.
   user	The username that is used to access the Elasticsearch cluster. The default username is elastic.
   password	The password that is used to access the Elasticsearch cluster. The password of the elastic user is specified when you create the Elasticsearch cluster. If you forget the password, you can reset it. For more information about how to reset the password, see Reset the access password for an Elasticsearch cluster.

   Important

   Specify only the username and password. Comment out trustStorePath and trustStorePass. Otherwise, the error message "NoSuchFileException:../es_keystore.jks" is reported.

3. Optional:Change the IP address or port number for listening.

   By default, SkyWalking communicates with Elasticsearch over port 12800 for RESTful API operations and over port 11800 for gRPC API operations. The IP address or port number can be changed in the core part of the application.yml file. In this topic, the default values are used.

   core:
     selector: ${SW_CORE:default}
     default:
       # Mixed: Receive agent data, Level 1 aggregate, Level 2 aggregate
       # Receiver: Receive agent data, Level 1 aggregate
       # Aggregator: Level 2 aggregate
       role: ${SW_CORE_ROLE:Mixed} # Mixed/Receiver/Aggregator
       restHost: ${SW_CORE_REST_HOST:0.0.0.0}
       restPort: ${SW_CORE_REST_PORT:12800}
       restContextPath: ${SW_CORE_REST_CONTEXT_PATH:/}
       gRPCHost: ${SW_CORE_GRPC_HOST:0.0.0.0}
       gRPCPort: ${SW_CORE_GRPC_PORT:11800}

4. Optional:In the webapp folder, modify the configurations in the webapp.yml file.

   Default configurations are used in this topic. You can modify the configurations based on your business requirements.

   server:
     port: 8080
   collector:
     path: /graphql
     ribbon:
       ReadTimeout: 10000
       # Point to all backend's restHost:restPort, split by ,
       listOfServers: 127.0.0.1:12800

Step 3: Verify the results

1. Run the following commands to start SkyWalking on the Linux server:

   cd ../bin
   ./startup.sh

   Important

   • Before you start SkyWalking, make sure that the Elasticsearch cluster is normally running.

   • SkyWalking Collector and SkyWalking UI are also started after you run the startup.sh command.

   If SkyWalking is started, the following result is returned:

   SkyWalking OAP started successfully!
   SkyWalking Web Application started successfully!

2. Enter http://<IP address of the Linux server>:8080/ in the address bar of your browser.

   

   Note

   If this is the first time you use SkyWalking to connect to Elasticsearch, the startup is slow. This is because SkyWalking needs to create a large number of indexes in Elasticsearch. Before the creation is completed, the accessed page is blank. You can view the logs that are stored in <SkyWalking installation path>logs/skywalking-oap-server.log to check whether the creation is completed.

3. Log on to the Kibana console of the Elasticsearch cluster. For more information, see Log on to the Kibana console. Then, run the GET _cat/indices?v command to view index data.

   In the returned results, a large number of indexes whose names start with skywalking-index exist.