Use Metricbeat to collect system data and NGINX service data - Elasticsearch

This guide describes how to use the Alibaba Cloud Metricbeat collector to monitor system metrics (CPU, memory, disk, and network) and Nginx service data, and then visualize these metrics using Kibana dashboards.

Prerequisites

Before you begin, ensure the following requirements are met:

An Alibaba Cloud Elasticsearch cluster is created.
Note
Version: 6.0.x to 7.17.x (Version 7.10.0 is highly recommended for optimal compatibility).
Automatic indexing is enabled for your Alibaba Cloud Elasticsearch cluster.
This feature is required by Metricbeat. For more information, see Quick access and configuration.
An ECS instance is created within the same Virtual Private Cloud (VPC) as the Elasticsearch cluster. See Custom launch.
Important
Operating System: Alibaba Cloud Linux (Alinux), Red Hat, or CentOS.
Cloud Assistant Agent and Docker must be installed and running on the ECS instance.
See Install the Cloud Assistant Agent and Deploy and use Docker.
Network: If accessing Kibana via the internet, ensure Public Network Access is enabled for the Kibana node.

Collect system data

Log on to the Alibaba Cloud Elasticsearch console.
In the left navigation menu, click Beats Data Shippers.
In the Create Shipper area, click Metricbeat.
Install and configure the collector.
For more information, see Collect ECS service logs and Collector YML configuration. The configuration for this topic is shown in the following figure.
Note
- Select Enable Kibana Monitoring to start monitoring the Metricbeat service in the Kibana console.
- Select Enable Kibana Dashboard. This option generates charts in the Kibana console without requiring additional YML configuration. Because Alibaba Cloud Kibana is configured within a VPC, you must enable internal-facing access on the Kibana configuration page to access the console. For more information, see Connect to a cluster through Kibana.
- Because the system module is enabled by default, you can skip Shipper YML Configuration.
Click Next.
Select the ECS instance where you want to install the collector.
Note
If you are creating a collector for the first time, click Authorize Now and follow the prompts to grant Alibaba Cloud ES access to Alibaba Cloud ECS.
Start the collector and check its installation status.
1. Click Start.
  The Start Shipper dialog box appears after the collector starts.
2. Click Back to Beats Shippers to go to the Beats Data Shippers page. The Metricbeat collector that you started is displayed in the Manage Shippers area.
3. Wait for the Shipper Status to change to Enabled. Then, click View Instances in the Actions column.
4. On the View Instances page, check the Installed Shippers. The installation is successful if the status is Normal Heartbeat.
View the results.
1. Log on to the Kibana console of the target Alibaba Cloud Elasticsearch cluster.
2. In the navigation menu, click Dashboard.
3. In the Dashboard list, click [Metricbeat System] Overview. Then, click a system to view its monitoring dashboard.

Collect Nginx service data

Prerequisite: You must enable stub_status for the Nginx service. The ngx_http_stub_status_module module in Nginx counts the number of requests that the Nginx service receives and processes. To use this module, you must enable stub_status in the nginx.conf file. For information about how to modify the nginx.conf file, see How to modify the listener port address for web services such as Nginx and Tomcat on an ECS instance that runs Linux.

location /status {
           stub_status on;
           access_log off;
        }

Important

The server_status_path configured in the metricbeat.yml file must match the status path in the nginx.conf file.

Log on to the Alibaba Cloud Elasticsearch console.
In the left navigation menu, click Beats Data Shippers.
In the Create Shipper area, click Metricbeat.
Install and configure the collector.
For more information, see Collect ECS service logs and Collector YML configuration.
Add the following script to the metricbeat.yml file.
```
metricbeat.modules:
- module: nginx
  metricsets: ["stubstatus"]
  enabled: true
  period: 10s
  # Nginx hosts
  hosts: ["http://121.41.**.**"]
  # Path to server status. Default server-status
  server_status_path: "status"
```
Note
- Select Enable Kibana Monitoring to monitor the Metricbeat service in the Kibana console.
- Select Enable Kibana Dashboard. This automatically generates charts in the Kibana console without requiring manual YML configuration. Because Alibaba Cloud Kibana is configured within a VPC, you must first enable internal-facing access on the Kibana configuration page. For more information, see Connect to a cluster through Kibana.
Click Next.
Select the ECS instance where you want to install the collector.
Note
If you are creating a collector for the first time, click Authorize Now and follow the prompts to grant Alibaba Cloud ES access to Alibaba Cloud ECS.
Start the collector and check its installation status.
For more information, see Collect system data.
View the results.
1. In a browser, go to <Nginx hosts>/status to view the monitoring page.
2. Log on to the Kibana console of the target Alibaba Cloud ES instance.
  For more information, see Log on to the Kibana console.
3. In the left navigation pane, click Dashboard.
4. In the Dashboard list, click [Metricbeat Nginx] Overview to view the Nginx monitoring dashboard.