How to Collect and Analyze NGINX Log using Alibaba Cloud Elasticsearch and Fleet

In the rapidly evolving digital landscape, NGINX serves as the backbone for countless high-performance web applications. Understanding the intricacies of NGINX log data is crucial for monitoring stability, optimizing performance, and enhancing security. Leveraging the powerful combination of a Fleet agent and Alibaba Cloud Elasticsearch, developers and system administrators can transform raw data into actionable insights. This article walks you through the necessary steps to harness these technologies effectively.

Prepare Your Environment

The journey begins with setting up an Alibaba Cloud Elasticsearch cluster, tailored for this purpose. Note, our Elasticsearch cluster needs to be version V7.10 or later, with V7.16 as the only exception. For detailed guidance, visit Creating an Alibaba Cloud Elasticsearch Cluster.

Simultaneously, create two Elastic Compute Service (ECS) instances within the same Virtual Private Cloud (VPC) as your Elasticsearch cluster, ensuring they are equipped with Elastic IP addresses (EIPs) for unhindered access.

Create an Agent Policy and Add Integrations

Step 1: Creating an Agent Policy

First, log into the Kibana console of your Elasticsearch cluster. Navigate to Management > Fleet and create a new agent policy named nginx-log, omitting system logs and metrics collection for a focused approach on NGINX logs.

 Click "Create agent policy."
 Name: nginx-log
 Uncheck: Collect system logs and metrics, Collect agent logs, Collect agent metrics.
 Click "Create agent policy."

Step 2: Adding a Fleet Server Integration

Under your newly created agent policy, add a Fleet server integration. This crucial step ensures your Fleet agent can transfer NGINX log data to Elasticsearch efficiently.

11. Find nginx-log agent policy > Add integration.
22. Search "Fleet Server" > Install Fleet Server assets.
33. Click "Add Fleet Server" > Save and continue.

Step 3: Integrating NGINX Logging

Now, within the same agent policy, introduce NGINX logging by adding the NGINX integration, enabling the collection and analysis of NGINX logs.

 Add integration > Search "Nginx" > Install Nginx assets.
 Click "Add Nginx" > Configure integration based on your needs > Save and continue.

Deploying a Fleet Agent

With your integrations in place, it's time to deploy a Fleet agent on each ECS instance, gathering NGINX logs effectively.

1）Configure Fleet Server Host:Set the Fleet Server host to collect data, utilizing the private IP address of your ECS instance with the specified port.

PUT Fleet Server hosts
{
  "hosts": ["https://172.16.*.***:8220"]
}

2）Add a Fleet Agent:Navigate to the Fleet page, elect your nginx-log policy, and add a Fleet agent by executing the autogenerated code on your ECS instance.

sudo Fleet agent install --url=https://172.16.*.***:8220 --enrollment-token=<YourTokenHere>

Viewing Collected Data

With the Fleet agent diligently collecting data, explore the rich insights within your logs through Kibana's View Dashboards, Discover page, or directly querying Elasticsearch from the Dev Tools console.

1GET logs-nginx.access-*/_search
2
3GET logs-nginx.error-*/_search

By masterfully orchestrating Alibaba Cloud Elasticsearch and Fleet, you can elevate your NGINX log analysis, driving informed decision-making and robust server management.

Ready to start your journey with Elasticsearch on Alibaba Cloud? Explore our tailored Cloud solutions and services to take the first step towards transforming your data into a visual masterpiece. Click here, Embark on Your 30-Day Free Trial

Community

How to Collect and Analyze NGINX Log using Alibaba Cloud Elasticsearch and Fleet

Prepare Your Environment

Create an Agent Policy and Add Integrations

Step 1: Creating an Agent Policy

Step 2: Adding a Fleet Server Integration

Step 3: Integrating NGINX Logging

Deploying a Fleet Agent

Viewing Collected Data

Read previous post:

Read next post:

Data Geek

You may also like

Comments

Data Geek

Related Products

Alibaba Cloud Elasticsearch

Simple Log Service

Log Management for AIOps Solution

Hologres