Elasticsearch:Configure a DingTalk chatbot to receive alert notifications from X-Pack Watcher

Step 1: Configure a DingTalk chatbot

1. Create a DingTalk group that is used to receive alert notifications.

2. Click the icon in the upper-right corner of the chat window of the DingTalk group. In the Group Settings panel, click Bot.

3. In the Robot Management panel, click Add Robot.

4. In the Robot dialog box, click Add Robot.

5. Click the Custom card. In the Robot details dialog box, click Add.

6. In the Add Robot dialog box, select Custom Keywords for Security Settings. In the field that appears, enter one or more keywords.

   Important

   The keywords must be included in the alert information you configured.

7. Read and agree to the terms of service, and click Finished.

8. Click Copy on the right side of Webhook to copy the webhook URL of the chatbot.

   Keep the webhook URL for later use. In addition, you must keep the webhook URL confidential. If the webhook URL is leaked, risks may arise.

Step 2: Configure an NGINX proxy and configure a security group rule for the ECS instance

1. Configure an NGINX proxy on the ECS instance.

   The NGINX proxy is used to forward alerts notifications. X-Pack Watcher sends alert notifications to the proxy, which then forwards the notifications to DingTalk or WeCom.

   1. Install NGINX on the ECS instance.

   2. Configure the nginx.conf file.

      Replace the server configuration in the nginx.conf file with the following code:

      server
        {
          listen 8080;# The listening port.
          server_name localhost;# The domain name.
          index index.html index.htm index.php;
          root /usr/local/webserver/nginx/html;# The website directory.
            location ~ .*\.(php|php5)?$
          {
            #fastcgi_pass unix:/tmp/php-cgi.sock;
            fastcgi_pass 127.0.0.1:9000;
            fastcgi_index index.php;
            include fastcgi.conf;
          }
          location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|ico)$
          {
            expires 30d;
            # access_log off;
          }
          location / {
            proxy_pass <Webhook URL of the DingTalk chatbot>;
          }
          location ~ .*\.(js|css)?$
          {
            expires 15d;
            # access_log off;
          }
          access_log off;
        }

      Replace <Webhook URL of the DingTalk chatbot> with the webhook URL of the DingTalk chatbot that you configured to receive alert notifications.

   3. Reload the NGINX configuration file and restart NGINX.

      /usr/local/webserver/nginx/sbin/nginx -s reload            # Reload the NGINX configuration file.
      /usr/local/webserver/nginx/sbin/nginx -s reopen            # Restart NGINX.

2. Configure a security group rule for the ECS instance.

   The security group rule is used to allow the NGINX proxy on the ECS instance to receive alert notifications from the Elasticsearch cluster.

   1. Log on to the ECS console.

   2. In the left-side navigation pane, choose Instances & Images > Instances.

   3. On the Instances page, find the ECS instance and click its name.

   4. On the instance details page, click the Security Groups tab.

   5. On the Security Groups tab, click the name of the desired security group.

   6. On the Inbound tab of the Access Rule section, click Add Rule.

   7. Configure the parameters.

      Parameter	Description
      Action	Select Allow.
      Priority	Retain the default value.
      Protocol Type	Select Custom TCP.
      Port Range	Set this parameter to the port that you frequently use. In this example, port 8080 is used.
      Authorization Object	Enter the IP addresses of all nodes in the Elasticsearch cluster. Note For more information about how to obtain the IP addresses of the nodes, see View the basic information of nodes.
      Description	The description of the rule.

   8. Click Save.

Step 3: Configure a watch for alerting

1. Log on to the Kibana console of your Elasticsearch cluster and go to the homepage of the Kibana console as prompted.
   For more information about how to log on to the Kibana console, see Log on to the Kibana console.

   Note In this example, an Elasticsearch V6.7.0 cluster is used. Operations on clusters of other versions may differ. The actual operations in the console prevail.
2. In the left-side navigation pane of the page that appears, click Dev Tools.

3. On the Console tab of the page that appears, run the following command to create a watch.

   In this example, a watch named log_error_watch is created to search the logs index for errors every 10 seconds. If more than 0 errors are found, an alert is triggered.

   PUT _xpack/watcher/watch/log_error_watch
   {
     "trigger": {
       "schedule": {
         "interval": "10s"
       }
     },
     "input": {
       "search": {
         "request": {
           "indices": ["logs"],
           "body": {
             "query": {
               "match": {
                 "message": "error"
               }
             }
           }
         }
       }
     },
     "condition": {
       "compare": {
         "ctx.payload.hits.total": {
           "gt": 0
         }
       }
     },
     "actions" : {
     "test_issue" : {
       "webhook" : {
         "method" : "POST",
         "url" : "http://<yourAddress>:8080",
         "body" : "{\"msgtype\": \"text\", \"text\": { \"content\": \"An error is found. Handle the issue immediately.\"}}"
       }
     }
   }
   }

   Table 1. Parameters

   Parameter	Network architecture	Description
   url	New network architecture	The domain name of the endpoint. Requests are forwarded based on the domain name. For more information about how to obtain the domain name of an endpoint, see Configure a private connection for an Elasticsearch cluster.
   	Original network architecture	Set this parameter to one of the following items: IP address of the NGINX proxy. In this case, requests are forwarded over the Internet by using the NGINX proxy that resides in the same VPC as the Elasticsearch cluster. Webhook URL of the DingTalk chatbot.
   body	New network architecture Original network architecture	You must configure this parameter based on the security settings of the DingTalk chatbot. For example, in Step 1: Configure a DingTalk chatbot, Security Settings is set to Custom Keywords and the error keyword is specified. In this case, the DingTalk chatbot sends alert notifications only if the content field in the body parameter contains error.

   Note

   If the error No handler found for uri [/_xpack/watcher/watch/log_error_watch_2] and method [PUT] is returned after you run the preceding command, X-Pack Watcher is disabled for the Elasticsearch cluster. In this case, enable X-Pack Watcher and run the command again. For more information, see Configure the YML file.

Step 4: View the alert notifications

In normal cases, if the conditions specified in Step 3: Configure a watch for alerting are met, an alert notification similar to An error is found. Handle the issue immediately is sent to the DingTalk group.

DELETE _xpack/watcher/watch/log_error_watch