Use Metricbeat to collect the metrics of a machine - Elasticsearch

Beats are lightweight, single-purpose data shippers designed to send operational data from many machines and systems to Elasticsearch or Logstash. Metricbeat is a specific Beat that collects various system and service metrics. This includes general system metrics (CPU, memory) as well as service-specific metrics (e.g., Redis, NGINX). This topic explains how to configure Metricbeat to collect system and service metrics, send them to an Alibaba Cloud Elasticsearch cluster, and visualize them using Kibana dashboards.

Prerequisites

An Alibaba Cloud Elasticsearch cluster is created.
Note
To access the Elasticsearch cluster via its internal endpoint, you must have an Elastic Compute Service (ECS) instance that resides in the same virtual private cloud (VPC) as the Elasticsearch cluster. For more information, see Create an instance.
Download the Metricbeat installation package for your operating system. This guide uses Metricbeat 6.7.0 and macOS as an example.

Procedure

Follow these steps to set up Metricbeat monitoring:

Step 1: Configure an Alibaba Cloud Elasticsearch cluster

Log on to the Alibaba Cloud Elasticsearch console.
In the left navigation menu, choose Elasticsearch Clusters.
Navigate to the target cluster.
1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
2. On the Elasticsearch Clusters page, find the cluster and click its ID.
Enable public network access and configure IP whitelist (if using public endpoint).
1. In the left navigation pane, select Security.
2. On the Security page, toggle on the Public Network Access switch to green.
  Wait for a while until the settings take effect.
3. Click Modify next to Public IP Address Whitelist.
4. In the panel, add the public IP address of the computer or server where Metricbeat will run to the whitelist.
  Important
  If your Metricbeat host is behind a public network (e.g., Wi-Fi, corporate VPN), you must add the IP address of its outbound proxy/gateway.
  Adding 0.0.0.0/1,128.0.0.0/1 (which collectively covers all public IP addresses) is strongly discouraged for production environments due to significant security risks. This grants public access to your cluster from anywhere on the internet. Use it only for temporary testing or when no other option is available, and remove it immediately after use.
Obtain your Elasticsearch cluster's public endpoint:
1. In the left navigation pane, select Basic Information.
2. In the Basic Information section, note down the public endpoint of the Elasticsearch cluster. You will need this for Metricbeat configuration.
Enable auto indexing:
1. In the left navigation pane, select Cluster Configuration.
2. In the YML File Configuration section, click Modify Configuration.
3. In the panel, set Auto Indexing to Enable.
  Important
  This change requires a cluster restart to take effect, which may temporarily impact your services. Plan this operation during off-peak hours.
Select This operation will restart the cluster. Continue? the checkbox and click OK.
Monitor the restart progress in the Tasks dialog box. The new configuration takes effect after the cluster restarts.

Step 2: Configure Metricbeat

Decompress Metricbeat and navigate to the decompressed Metricbeat folder.

Edit metricbeat.yml:

Locate the Elasticsearch output section and uncomment the relevant lines. Configure it as follows:

编辑metricbeat.yml文件

Parameter	Description
hosts	The endpoint of your Elasticsearch cluster. Use the public endpoint obtained earlier. If Metricbeat is running on an ECS instance in the same VPC, you can use the internal endpoint.
protocol	Set this parameter to http.
username	The username for your Elasticsearch cluster. The default is elastic.
password	The password corresponding to the `username`.

Start Metricbeat:
```
./metricbeat -e -c metricbeat.yml
```
If Metricbeat starts successfully, it will begin collecting and sending data to your Elasticsearch cluster. You should see output indicating data transfer.

Step 3: View Metricbeat dashboards in Kibana

Log on to the Kibana console of the Elasticsearch cluster.
(Optional) In the left menu bar, click Management and create an index pattern:
Note
If you have already created an index pattern, skip this step.
1. In the Kibana section, click Index Patterns.
2. On the Create index pattern page, enter a name for the index pattern. This is the name of the index that you want to query.
3. Click Next step.
4. Click Create index pattern.
In the left menu bar, select Dashboard.
On the Dashboards page, view the collected data.
- The following figure shows relevant metrics.
- Click Metricbeat-cpu to view CPU metrics.
  Note
  You can configure metrics to be refreshed at 5-second intervals. The system generates reports for the metrics collected at these intervals. You can also connect to WebHook to configure alerts.