By Victor Mak, Solutions Architect
This article describes how to integrate Alibaba Cloud Web Application Firewall (WAF) log with Splunk to ensure all compliance, auditing, and other related logs can be ingested into your Security Operation Center.
The following figure illustrates the Splunk integration architecture:
Alibaba Cloud Log Service is a one-stop service for log data, Log Service experiences massive big data scenarios of Alibaba Group. Log Service allows you to quickly complete the collection, consumption, shipping, query, and analysis of log data without the need for development, which improves the Operation & Maintenance (O&M) efficiency and the operational efficiency, and builds the processing capabilities to handle massive logs in the DT (data technology) era. For more information, see Log Service (SLS) Production Introduction.
We will be using Python on an Alibaba Cloud Elastic Compute Service (ECS) instance, integrated with Splunk HEC, to deliver WAF log to Splunk. The consumer library is an advanced mode of log consumption in Log Service, and provides the consumer group concept to abstract and manage the consumption end. Compared with using SDKs directly to read data, you can only focus on the business logic by using the consumer library, without caring about the implementation details of Log Service, or the load balancing or failover between consumers. For more information, see consumer group Introduction.
Splunk HEC is a Splunk Http Event Collector, a HTTP(s) interface to receive logs.
Before you begin, make sure you have the following:
Follow these steps to enable Web Application Firewall (WAF) logging in the WAF console:
Follow these steps to configure Http Event Collector (HEC) in the Splunk console:
Follow these steps to install Log Service Python SDK in ECS:
apt-get update
apt-get install -y python3-pip python3-dev
cd /usr/local/bin
ln -s /usr/bin/python3 python
pip3 install --upgrade pip
pip install aliyun-log-python-sdk
wget https://raw.githubusercontent.com/aliyun/aliyun-log-python-sdk/master/tests/consumer_group_examples/sync_data_to_splunk.py
python sync_data.py
*** start to consume data...
consumer worker "WAF-SLS-1" start
heart beat start
heart beat result: [] get: [0, 1]
Get data from shard 0, log count: 6
Complete send data to remote
Get data from shard 0, log count: 2
Complete send data to remote
heart beat result: [0, 1] get: [0, 1]
Boost Your E-Commerce Business in China Using Global Acceleration
2,599 posts | 768 followers
FollowAlibaba Clouder - September 2, 2020
Alibaba Cloud Community - October 19, 2021
wjo1212 - January 3, 2019
Alibaba Clouder - April 20, 2018
Alibaba Clouder - July 12, 2019
Alibaba Cloud Native Community - October 31, 2023
very nice article, to gain better performance, it's recommended to use pypy3 to run the program in parallel. refer to this article for more detail:https://community.alibabacloud.com/blog/integrating-alibaba-cloud-log-service-with-splunk_594335
2,599 posts | 768 followers
FollowA cloud firewall service utilizing big data capabilities to protect against web-based attacks
Learn MoreAn all-in-one service for log-type data
Learn MoreLearn More
More Posts by Alibaba Clouder
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
Get Started for Free Get Started for Free
Raja_KT February 8, 2019 at 7:54 am
I think most of the vCPUs are 2 GHz and above , right now, right?