Use Uptime to monitor Alibaba Cloud Elasticsearch clusters in real time - Elasticsearch

Heartbeat shippers can detect the statuses of network endpoints based on the HTTP or HTTPS, TCP, and ICMP services on a regular basis. They can also send the collected detection data to the Uptime application in Kibana. Then, the Uptime application monitors the availability and response time of applications and services in real time and reports errors before your business is affected. This topic describes how to use Uptime to monitor an Alibaba Cloud Elasticsearch cluster in real time.

Background information

Uptime must be used with the following services:

Heartbeat
Elasticsearch
Kibana

Note

You can also use the Alerting and Actions feature of Kibana V7.7 to configure monitoring and alerting. For more information, see Alerting and Actions.

Deployment architecture

Deployment of a single Heartbeat shipper
A single Heartbeat shipper is deployed at a single position to monitor a single service. The Heartbeat shipper sends monitoring data to Elasticsearch. You can view the heartbeat data of the service on the Uptime page of Kibana and determine the service status based on the heartbeat data.
Deployment of multiple Heartbeat shippers
Two Heartbeat shippers are deployed at different positions to monitor the same service. The two Heartbeat shippers send monitoring data to Elasticsearch. You can view the heartbeat data of the service on the Uptime page of Kibana and determine the service status based on the heartbeat data. If the Heartbeat shipper at one position becomes faulty, the Heartbeat shipper at the other position can help locate the fault.

For more information about the deployment architecture, see Deployment Architecture.

Make preparations

Create an Alibaba Cloud Elasticsearch cluster and enable the Auto Indexing feature for the cluster.
For more information, see Create an Alibaba Cloud Elasticsearch cluster and Configure the YML file.
Create an Elastic Compute Service (ECS) instance, which is used to deploy a Heartbeat shipper. The ECS instance must reside in the same virtual private cloud (VPC) as the Elasticsearch cluster.
For more information, see Create an instance on the Custom Launch tab.
Important
Beats supports only the following operating systems: Alibaba Cloud Linux, Red Hat Enterprise Linux (RHEL), and CentOS. Therefore, you must select one of the preceding operating systems when you create the ECS instance.
Install Cloud Assistant and Docker on the ECS instance.
For more information, see Install Cloud Assistant Agent and Deploy and use Docker on ECS instances.

Create a Heartbeat shipper

Log on to the Alibaba Cloud Elasticsearch console.
In the left-side navigation pane, click Beats Data Shippers.
In the Create Shipper section of the Beats Data Shippers page, click Heartbeat.

Install and configure a shipper.

For more information, see Collect the logs of an ECS instance and Prepare a YML configuration file for a shipper.

The following figure shows the heartbeat.monitors parameter configuration on the heartbeat.yml tab in this example.

Parameter	Description
type	In this example, http is used. Note A Heartbeat shipper can monitor HTTP or HTTPS, TCP, and ICMP services. If you use an HTTP or HTTPS monitor, response code, request bodies, and request headers can be monitored. If you use a TCP monitor, port numbers and strings can be monitored.
urls	The URLs that you want to check. You can specify multiple HTTP services. In this example, an Alibaba Cloud Elasticsearch cluster is checked. This parameter is set to the internal endpoint of the Elasticsearch cluster.
schedule	The interval at which the check is performed. The value @every 10s indicates that the check is performed every 10 seconds.

Click Next.
In the ECS instance list, select the instance on which you want to install the shipper.
Click Start to start the shipper.
If the state of the Heartbeat shipper is Enabled and the installation state of the Heartbeat shipper is Normal Heartbeat, the Heartbeat shipper is installed. To view the installation state of the shipper, click View Instances in the Actions column that corresponds to the shipper.

View the monitoring information on the Uptime page

Log on to the Kibana console.
The Kibana console is the one that corresponds to the Elasticsearch cluster that you specified for Output when you create the Heartbeat shipper. For more information, see Log on to the Kibana console.
In the left-side navigation pane, click Uptime. On the Uptime page, you can view the monitoring information.
- Color red: indicates that the Elasticsearch cluster is in an abnormal state. Check the communication status of the Heartbeat shipper or the status of the Elasticsearch cluster.
- Color blue: indicates that the Elasticsearch cluster is in a normal state.