Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog How to Integrate Alibaba Cloud WAF Log with Splunk

How to Integrate Alibaba Cloud WAF Log with Splunk

This article shows you how to integrate Alibaba Cloud WAF log with Splunk to ensure all related logs can be ingested into your Security Operation Center.

By Victor Mak, Solutions Architect

This article describes how to integrate Alibaba Cloud Web Application Firewall (WAF) log with Splunk to ensure all compliance, auditing, and other related logs can be ingested into your Security Operation Center.

The following figure illustrates the Splunk integration architecture:

1

Alibaba Cloud Log Service is a one-stop service for log data, Log Service experiences massive big data scenarios of Alibaba Group. Log Service allows you to quickly complete the collection, consumption, shipping, query, and analysis of log data without the need for development, which improves the Operation & Maintenance (O&M) efficiency and the operational efficiency, and builds the processing capabilities to handle massive logs in the DT (data technology) era. For more information, see Log Service (SLS) Production Introduction.

We will be using Python on an Alibaba Cloud Elastic Compute Service (ECS) instance, integrated with Splunk HEC, to deliver WAF log to Splunk. The consumer library is an advanced mode of log consumption in Log Service, and provides the consumer group concept to abstract and manage the consumption end. Compared with using SDKs directly to read data, you can only focus on the business logic by using the consumer library, without caring about the implementation details of Log Service, or the load balancing or failover between consumers. For more information, see consumer group Introduction.

Splunk HEC is a Splunk Http Event Collector, a HTTP(s) interface to receive logs.

Prerequisites

Before you begin, make sure you have the following:

  1. An Alibaba Cloud Account. If you don't have one already, visit the Free Trial Page to sign up for a free account.
  2. You have purchased WAF business edition or above to protect your website. If not, visit the WAF Product Page to learn more about Alibaba Cloud WAF.
  3. You have a Linux ECS server with hardware spec is recommended:
    1. Operating System with Ubuntu
    2. 8 vCPUs with 2.0+ Ghz
    3. 32GB Memory
    4. at least 2GB available disk space, suggest 10GB or more
  4. You have a Splunk Enterprise server.

Procedure Overview

  1. Enable Web Application Firewall (WAF) logging.
  2. Configure Splunk Http Event Collector (HEC).
  3. Setup Python environment in ECS.
  4. Configure Python program send log to Splunk.

Step 1: Enable Web Application Firewall (WAF) Logging

Follow these steps to enable Web Application Firewall (WAF) logging in the WAF console:

  1. Log on to the Alibaba Cloud WAF console
  2. In the left-side, navigate to the App Management under App Market.
  3. Click Upgrade in the right-side to enable WAF Real-time logging

    2

  4. Enable Access log service, select log storage period and log storage size depends on your business actual usage.

    3

  5. Click Authorize after enabled access log service

    4

  6. Click Confirm Authorization Policy

    5

  7. Click Configure in Real-time Log Query and Analysis Service:

    6

  8. Enable the website you want to enable Log services in drop-down list.

    7

Step 2: Configure Splunk HTTP Event Collector (HEC)

Follow these steps to configure Http Event Collector (HEC) in the Splunk console:

  1. Log on to the Splunk Enterprise Console
  2. In the upper-right, navigate to the Data inputs under Settings

    8

  3. Navigate to HTTP Event Collector and click Add new

    9

  4. In the task of Select Source, give a Name for HTTP Event Collector and click Next

    10

  5. In the task of Add Data, add index to HTTP Event Collector. In this example, we use main index and click Review and Submit

    11

  6. The HTTP Event Collector has been created successfully.

    12

  7. Navigate to HTTP Event Collector, you should see the token now

    13

  8. Navigate to HEC token you created and click Edit, input source and source type

    14

Step 3: Setup Python Environment in ECS

Follow these steps to install Log Service Python SDK in ECS:

  1. SSH or console login to ECS
  2. Install Python3, pip and Python SDK of Log Service. For more information on Log Services Python SDK, see User Guide.
apt-get update
apt-get install -y python3-pip python3-dev
cd /usr/local/bin
ln -s /usr/bin/python3 python
pip3 install --upgrade pip
pip install aliyun-log-python-sdk

Step 4: Configure Python to Send Logs to Splunk

  1. Before you begin, make sure you have following information:
    1. Project is the Log Service's resource management unit, used to isolate and control resources. You can find the Project Name in Alibaba Cloud Log Services console:

      15

    2. Log Service Endpoint is a URL used to access a project and logs within the project, and is associated with the Alibaba Cloud region where the project resides and the project name. You can find the Endpoint URL in Service endpoint.
    3. Logstore is a unit in Log Service for the collection, storage, and query of log data. Each Logstore belongs to a project, and each project can create multiple Logstores. You can find the Logstore Name under your Log Service Project in Alibaba Cloud Log Services console:

      16

    4. AccessKey is a "secure password" designed for you to access your cloud resources by using APIs (not the console). You can use the AccessKey to sign API request content to pass the security authentication in Log Service. For more information, see AccessKey Introduction. You can find your Accesskey in User Management console:

      17

    5. Splunk Enterprise Host is same as the IP address/Hostname you access Splunk Enterprise Server web console
    6. Splunk HEC port, you can find the HEC port in Splunk Global Setting under HEC console:

      18

    7. HEC Token is the token you use to integrate with Alibaba Cloud Log Services and Splunk. You can find token under HTTP Event Collector console:

      19

  2. Download the latest example of integration code from GitHub:
    wget https://raw.githubusercontent.com/aliyun/aliyun-log-python-sdk/master/tests/consumer_group_examples/sync_data_to_splunk.py

  3. Replace Log Services and Splunk related settings in Python Program includes:
    1. SLS Endpoint
    2. SLS accessKeyId
    3. SLS accessKey
    4. SLS Project
    5. SLS Logstore
    6. SLS Consumer Group
    7. Splunk Host
    8. Splunk HEC Port
    9. Splunk HEC Token

  4. Suppose the Python program is saved as "sync_data.py", you could launch it as:
    python sync_data.py

  5. The Python program log should show successful sent log to remote Splunk server.
    *** start to consume data...
consumer worker "WAF-SLS-1" start 
heart beat start
heart beat result: [] get: [0, 1]
Get data from shard 0, log count: 6
Complete send data to remote
Get data from shard 0, log count: 2
Complete send data to remote
heart beat result: [0, 1] get: [0, 1]

  6. You are able to search the WAF log in Splunk server now

    20

WAF Security Log Service Tutorial Splunk
2 1 0
Share on

Read previous post:

Boost Your E-Commerce Business in China Using Global Acceleration

Read next post:

Kerberos-Based Big Data Security Solution

Alibaba Clouder

2,599 posts | 764 followers

You may also like

Comments

Raja_KT February 8, 2019 at 7:54 am

I think most of the vCPUs are 2 GHz and above , right now, right?

wjo1212 February 13, 2019 at 9:12 am

very nice article, to gain better performance, it's recommended to use pypy3 to run the program in parallel. refer to this article for more detail:https://community.alibabacloud.com/blog/integrating-alibaba-cloud-log-service-with-splunk_594335

Alibaba Clouder

2,599 posts | 764 followers

Related Products

More Posts by Alibaba Clouder

See All