Use Metricbeat to collect system data and NGINX service data - Elasticsearch

This topic describes how to use Alibaba Cloud Metricbeat to collect system data and NGINX service data and then generate visual charts. The system data includes CPU utilization, memory usage, disk I/O, and network I/O.

Prerequisites

The following operations are performed:

Create an Alibaba Cloud Elasticsearch cluster.
For more information, see Create an Alibaba Cloud Elasticsearch cluster.
Enable the Auto Indexing feature for the Elasticsearch cluster.
For security purposes, Alibaba Cloud Elasticsearch disables the Auto Indexing feature by default. However, Beats depends on this feature. If you select Elasticsearch for Output when you create a shipper, you must enable the Auto Indexing feature. For more information, see Access and configure an Elasticsearch cluster.
Create an Alibaba Cloud Elastic Compute Service (ECS) instance in the virtual private cloud (VPC) where the Elasticsearch cluster resides.
For more information, see Create an instance on the Custom Launch tab.
Important
Beats supports only Alibaba Cloud Linux (Alinux), Red Hat Enterprise Linux (RHEL), and CentOS.
Install Cloud Assistant and Docker on the ECS instance.
For more information, see Install Cloud Assistant Agent and Deploy and use Docker on ECS instances.

Use Metricbeat to collect system data

Log on to the Alibaba Cloud Elasticsearch console.
In the left-side navigation pane, click Beats Data Shippers.
In the Create Shipper section of the page that appears, click Metricbeat.
Install and configure a shipper.
For more information, see Collect the logs of an ECS instance and Prepare a YML configuration file for a shipper. The following figure shows the configurations that are used in this topic.
Note
- If you select Enable Kibana Monitoring, the system enables Metricbeat service monitoring in the Kibana console.
- If you select Enable Kibana Dashboard, the system generates charts in the Kibana console. You do not need to configure the YML file. Alibaba Cloud Kibana is deployed in a VPC. You must enable the Private Network Access feature for Kibana on the Kibana Configuration page. For more information, see Configure a public or private IP address whitelist for Kibana.
- The system module is enabled by default. You do not need to specify Shipper YML Configuration.
Click Next.
Select the ECS instance on which you want to install the shipper.
Note
If you are installing a shipper for the first time, click Authorize Now. Then, follow the instructions to authorize Elasticsearch to access ECS instances.
Enable the shipper and check whether the shipper is installed.
1. Click Enable.
  Then, the Enable Shipper message appears.
2. Click Back to Beats Shippers. In the Manage Shippers section of the Beats Data Shippers page, view the installed shipper.
3. After the state of the shipper changes to Enabled 1/1, click View Instances in the Actions column.
4. In the View Instances pane, check whether the shipper is installed on the ECS instance. If the value of Installed Shippers is Heartbeat Normal, the shipper is installed.
View the result.
1. Log on to the Kibana console of the Elasticsearch cluster.
  For more information, see Log on to the Kibana console.
2. In the left-side navigation pane, click Dashboard.
3. In the Dashboards section, click [Metricbeat System] Overview. On the page that appears, click the desired Metricbeat system and view the dashboard for the system.

Use Metricbeat to collect NGINX service data

Prerequisites: The stub_status module is enabled for the NGINX service. The ngx_http_stub_status_module module measures the number of requests received by the NGINX service and the number of requests processed by the NGINX service. Therefore, you must enable stub_status in the nginx.conf file.

location /status {
           stub_status on;
           access_log off;
        }

Important

The value of server_status_path in the metricbeat.yml file must be the same as that of status in the nginx.conf file.

Log on to the Alibaba Cloud Elasticsearch console.
In the left-side navigation pane, click Beats Data Shippers.
In the Create Shipper section of the page that appears, click Metricbeat.
Install and configure a shipper.
For more information, see Collect the logs of an ECS instance and Prepare a YML configuration file for a shipper.
Add the following script in the metricbeat.yml file:
```
metricbeat.modules:
- module: nginx
  metricsets: ["stubstatus"]
  enabled: true
  period: 10s
  # Nginx hosts
  hosts: ["http://121.41.**.**"]
  # Path to server status. Default server-status
  server_status_path: "status"
```
Note
- If you select Enable Kibana Monitoring, the system enables Metricbeat service monitoring in the Kibana console.
- If you select Enable Kibana Dashboard, the system generates charts in the Kibana console. You do not need to configure the YML file. Alibaba Cloud Kibana is deployed in a VPC. You must enable the Private Network Access feature for Kibana on the Kibana Configuration page. For more information, see Configure a public or private IP address whitelist for Kibana.
Click Next.
Select the ECS instance on which you want to install the shipper.
Note
If you are installing a shipper for the first time, click Authorize Now. Then, follow the instructions to authorize Elasticsearch to access ECS instances.
Enable the shipper and check whether the shipper is installed.
For more information, see Use Metricbeat to collect system data.
View the result.
1. In the address bar of your browser, enter <IP address of the NGINX server>/status to access the monitoring page of the NGINX server.
2. Log on to the Kibana console of the Elasticsearch cluster.
  For more information, see Log on to the Kibana console.
3. In the left-side navigation pane, click Dashboard.
4. In the Dashboards section, click [Metricbeat Nginx] Overview. On the page that appears, view the dashboard for the NGINX service.