Elasticsearch:Use Filebeat to collect Apache log data

Step 1: Make preparations

Step 2: Configure and install a Filebeat shipper

Step 3: Configure a Logstash pipeline to filter and synchronize data

Step 4: View the collected data

Create an Elasticsearch cluster and a Logstash cluster that are of the same version and are deployed in the same virtual private cloud (VPC).

Enable the Auto Indexing feature for the Elasticsearch cluster.

Create an Alibaba Cloud Elastic Compute Service (ECS) instance in the same VPC as the Elasticsearch cluster and Logstash cluster.

Install HTTP Daemon (HTTPd) on the ECS instance.

LogFormat "{\"@timestamp\":\"%{%Y-%m-%dT%H:%M:%S%z}t\",\"client_ip\":\"%{X-Forwa rded-For}i\",\"direct_ip\": \"%a\",\"request_time\":%T,\"status\":%>s,\"url\":\"%U%q\",\"method\":\"%m\",\"http_host\":\"%{Host}i\",\"server_ip\":\"%A\",\"http_referer\":\"%{Referer}i\",\"http_user_agent\":\"%{User-agent}i\",\"body_bytes_sent\":\"%B\",\"total_bytes_sent\":\"%O\"}"  access_log_json
# Change the original CustomLog configuration to CustomLog "logs/access_log" access_log_json.

Install Cloud Assistant and Docker on the ECS instance.

Log on to the Alibaba Cloud Elasticsearch console.

Navigate to the Beats Data Shippers page.

1. In the top navigation bar, select a region.

2. In the left-side navigation pane, click Beats Data Shippers.

3. Optional: If this is the first time you go to the Beats Data Shippers page, view the information displayed in the message that appears and click OK to authorize the system to create a service-linked role for your account.

   Note

   When Beats collects data from various data sources, Beats depends on the service-linked role and the rules specified for the role. Do not delete the service-linked role. Otherwise, the use of Beats is affected. For more information, see Elasticsearch service-linked roles.

In the Create Shipper section, move the pointer over Filebeat and click ECS Logs.

Configure and install a shipper.

Click Next.

In the Install Shipper step, select the ECS instance on which you want to install the shipper.

Start the shipper and check whether the shipper is installed.

1. Click Start.

   Then, the Start Shipper message appears.

2. Click Back to Beats Shippers. In the Manage Shippers section of the Beats Data Shippers page, view the installed shipper.

3. After the state of the shipper changes to Enabled 1/1, click View Instances in the Actions column.

4. In the View Instances panel, check whether the shipper is installed on the ECS instance. If the value of Installed Shippers is Normal Heartbeat, the shipper is installed.

In the left-side navigation pane of the Alibaba Cloud Elasticsearch console, click Logstash Clusters.

On the page that appears, find your Logstash cluster and click Manage Pipeline in the Actions column.

On the Pipelines page, click Create Pipeline.

Configure a pipeline.

input {
  beats {
      port => 8000
    }
}
filter {
  json {
        source => "message"
        remove_field => "@version"
        remove_field => "prospector"
        remove_field => "beat"
        remove_field => "source"
        remove_field => "input"
        remove_field => "offset"
        remove_field => "fields"
        remove_field => "host"
        remove_field => "message"
      }

}
output {
  elasticsearch {
    hosts => ["http://es-cn-mp91cbxsm00******.elasticsearch.aliyuncs.com:9200"]
    user => "elastic"
    password => "<your_password>"
    index => "<your_index>"
  }
}

Log on to the Kibana console of your Elasticsearch cluster and go to the homepage of the Kibana console as prompted.

In the left-side navigation pane of the page that appears, click Dev Tools.

On the Console tab of the page that appears, run the following command to view the collected data:

GET <your_index>/_search

In the left-side navigation pane, click Discover. On the page that appears, specify a period in the upper-right corner. Then, view the details of the collected data within the specified period.