This topic describes how to use Python Program to integrate Web Application Firewall (WAF) logs into a Syslog server to meet regulatory and audit requirements. This allows you to manage all the related logs in your security operations center.

Background information

The following figure shows the integration architecture.Architecture

Log Service is an end-to-end logging service developed by Alibaba Cloud and is widely used by Alibaba Group in big data scenarios. Log Service allows you to complete the collection, consumption, delivery, query, and analysis of log data without the need for development. This improves the O&M efficiency and the operational efficiency and delivers capabilities of processing a large number of logs in the Data Technology (DT) era. WAF is integrated with Log Service. The Log Service for WAF feature allows you to collect, query, and analyze website access logs. For more information, see Overview.

Python Program is a program running on ECS instances to deliver WAF logs to a Syslog server. The consumer library is an advanced mode provided for LogHub consumers. It uses consumer groups to manage the consumption end. Compared with the mode in which data is read by using SDKs, the consumer library enables you to focus only on the business logic. You do not need to concern about the implementation details of Log Service or the fault tolerance among multiple consumers. For more information, see Use consumer groups to consume logs.

The Syslog server centrally manages log messages. It can receive data from multiple Syslog sources.

Prerequisites

  • Log Service for WAF is enabled. The log collection feature is enabled for your domain name. For more information, see Get started with the Log Service for WAF feature.
  • A Linux ECS instance with the following recommended configurations is deployed:
    • Ubuntu operating system
    • 2.0 GHz processor or above, with eight cores
    • 32 GB of memory
    • Available disk space greater than 2 GB (More than 10 GB of available disk space is recommended.)
  • A Syslog server is deployed, and the UDP port 514 is enabled on the server to receive Syslog data.

Procedure

Install Log Service SDK for Python on your ECS instance and configure Python Program to deliver WAF logs to the Syslog server. Perform the following steps:

  1. Connect to the ECS instance by using SSH or in the ECS console. For more information, see Connect to an ECS instance.
  2. Install Python 3, pip, and aliyun-log-python-sdk. For more information about Log Service SDK for Python, see User Guide. 
    apt-get update
apt-get install -y python3-pip python3-dev
cd /usr/local/bin
ln -s /usr/bin/python3 python
pip3 install --upgrade pip
pip install aliyun-log-python-sdk
  3. Run the following command to download the latest integration sample code from GitHub: 
    wget https://raw.githubusercontent.com/aliyun/aliyun-log-python-sdk/master/tests/consumer_group_examples/sync_data_to_syslog.py
  4. Replace Log Service and Syslog parameters in Python Program. The following table describes the parameters.
    Parameter Meaning Description
    SLS Project Log project name A project is the basic unit to isolate and control resources in Log Service.

    You can log on to the Log Service console to view the log projects of WAF.

    The name of a WAF log project starts with waf-project. Projects that reside in the China (Hangzhou) region are the log projects of WAF instances in the Chinese mainland. Projects that reside in the Singapore region are the log projects of WAF instances outside the Chinese mainland.

    Projects
    SLS Endpoint Log Service endpoint The Log Service endpoint is a URL used to access a project and logs in the project. The endpoint varies based on the Alibaba Cloud region where the project resides and the project name. To view the URL, see Endpoints.
    SLS Logstore Logstore A Logstore is a unit in Log Service to collect, store, and query log data. Each Logstore belongs to a single project. Each project can have multiple Logstores.

    You can log on to the Log Service console and click a WAF log project to view the Logstore name.

    Logstore
    SLS AccessKey ID and AccessKey Secret AccessKey pair An AccessKey pair consists of an AccessKey ID and an AccessKey secret and is designed to access your cloud resources by using APIs instead of the console. You can use the AccessKey pair to sign API requests so that the requests can pass the security authentication in Log Service. For more information, see AccessKey.

    You can log on to the User Management console to view the information of your AccessKey pair.

    AccessKey
    Syslog Host Syslog host The IP address or hostname of the Syslog server.
    Syslog Port Syslog port The port used to receive Syslog data. The UDP port 514 and the TCP port 1468 are supported.
    Syslog protocol Syslog protocol The UDP or TCP protocol that is used to receive Syslog data. The parameter value varies based on the configurations of the Syslog server.
    Syslog separator Syslog delimiter The delimiter used to separate Syslog key-value pairs.
    The following code provides an example of how to configure Python Program:
    • Log Service configurations 
      endpoint = os.environ.get('SLS_ENDPOINT', 'http://ap-southeast-1.log.aliyuncs.com')
accessKeyId = os.environ.get('SLS_AK_ID', 'Your AccessKey ID')
accessKey = os.environ.get('SLS_AK_KEY', 'Your AccessKey secret')
project = os.environ.get('SLS_PROJECT', 'waf-project-548613414276****-ap-southeast-1')
logstore = os.environ.get('SLS_LOGSTORE', 'waf-logstore')
consumer_group = os.environ.get('SLS_CG', 'WAF-SLS')
    • Syslog configurations 
      settings = {
                "host": "1.2.xx.xx",
                "port": 514,       
                "protocol": "udp", 
                "sep": ",",       
                "cert_path": None, 
                "timeout": 120,    
                "facility": syslogclient.FAC_USER,  
                "severity": syslogclient.SEV_INFO,  
                "hostname": None,  
                "tag": None        
            }
  5. Start Python Program. Assume that Python Program is saved as sync_data_to_syslog.py. Run the following command to start it: 
    python sync_data_to_syslog.py
    The following command output shows that logs are delivered to the Syslog server after the start of Python Program: 
    *** start to consume data...
consumer worker "WAF-SLS-1" start 
heart beat start
heart beat result: [] get: [0, 1]
Get data from shard 0, log count: 6
Complete send data to remote
Get data from shard 0, log count: 2
Complete send data to remote
heart beat result: [0, 1] get: [0, 1]

You can query WAF logs in the Syslog server.