Add logs of security services to the CTDR feature - Security Center

You can add logs from on-premises security services, such as Chaitin Web Application Firewall (WAF) and FortiGate Firewall, to the Cloud Threat Detection and Response (CTDR) feature for centralized risk management in Security Center. This improves risk visibility and response efficiency. This topic describes how to add the logs of security services to the CTDR feature.

Overview

To add the logs of a security service to the CTDR feature, perform the following steps:

Create a Logstore dedicated for the CTDR feature: In the Security Center console, create a Logstore dedicated for a specific log type. The Logstore is used to store the logs that you want to add to the CTDR feature.
Collect logs from a security service to the Logstore: Collect the logs of a security service to the Logstore of Simple Log Service (SLS) that is dedicated for the CTDR feature. If raw logs are contained in one field, you must use Logtail plug-ins to parse the raw logs into structured data in the SLS console. The parsed logs are stored as key-value pairs in the Logstore.
Configure a parsing rule: Define the mappings between the logs stored in the Logstore and the standard log fields of the CTDR feature. This step standardizes the logs that you want to add to the CTDR feature.
Specify a data source and associate the Logstore with the parsing rule: In the Security Center console, specify a data source and associate the Logstore with the parsing rule. The logs are added to the CTDR feature.

Supported security services

Security service	Supported log type
FortiGate Firewall	Alert logs of FortiGate Firewall Flow logs of FortiGate Firewall
Chaitin WAF	Alert logs of Chaitin WAF Flow logs of Chaitin WAF
Microsoft Active Directory (AD)	AD logs
F5 BIG-IP Local Traffic Manager (LTM)	Alert logs of F5 BIG-IP LTM
Others	Firewall alert logs Firewall flow logs WAF alert logs WAF flow logs

Note

If the security service whose logs you want to add is not included in the preceding table, you can add the logs to a supported security service and then add the logs to the CTDR feature. If you have questions about log collection and parsing rule settings, submit a ticket to contact Security Center technical support.

Billing rules

The following table describes the billable items that are involved when you add the logs of security services to Security Center.

Billable item		Payer	Description
Volume of logs that are added to the CTDR feature		Alibaba Cloud account that purchased the CTDR feature	Bills are generated based on the volume of logs that are added from the dedicated Logstore to the CTDR feature on a daily basis.
Log storage capacity of the CTDR feature		Alibaba Cloud account that purchased the CTDR feature	Bills are generated based on the volume of standardized logs that are delivered to the log management feature for storage.
Logstore dedicated for the CTDR feature	Storage location: hot storage Read and write traffic	Alibaba Cloud account that purchased Security Center	The time to live (TTL) of the Logstore is fixed as 1 to 3 days. Your permissions on the log data in the Logstore are limited. You do not have the permissions to modify the log data or create indexes in the Logstore.
Logstore dedicated for the CTDR feature	Read traffic over the Internet Operations by using an Alibaba Cloud account, such as consumption, transformation, and shipping	Alibaba Cloud account that purchased the CTDR feature	The fee for the read traffic over the Internet is generated when the logs of security services are shipped to the Logstore. If an Alibaba Cloud account is used to perform operations that generate fees, such as consumption, transformation, and shipping, the fees are billed to the Alibaba Cloud account. For more information, see Billing overview.

Prerequisites

A specific volume of log data that you can add to the CTDR feature is purchased, and the CTDR feature is enabled. For more information, see Purchase and enable the CTDR feature.

1. Create a Logstore dedicated for the CTDR feature

To receive and temporarily store the logs of security services, perform the following operations to create a dedicated Logstore:

Log on to the Security Center console. In the top navigation bar, select the region of the asset that you want to manage. You can select China or Outside China.
In the left-side navigation pane, choose CTDR > Service Integration.
On the Service Integration page, find the required security service and click Access Settings in the Actions column.
In the panel that appears, find the required log type and click the number in the Associated Accounts column.
In the Add Account panel, configure the Import Account and Region parameters. Then, click Automatically Create Logstore.
Note
After the Logstore is created, the system displays the names of the project and the Logstore, which can be used to collect logs from security services to Alibaba Cloud.
Record the names of the project and Logstore that are displayed in the Add Account panel.

2. Collect logs to the Logstore

In this step, you must use the data collection method provided by SLS to parse the raw log samples of the required security service into key-value pairs and store the key-value pairs in the Logstore. In the next step, you can define the log parsing rules in the Security Center console to map the parsed fields to the standard fields required by CTDR, which is the log standardization process.

Important

If you have questions when you collect logs to the Logstore, you can submit a ticket to contact the technical support. When you submit the ticket, select Simple Log Service from the Product Catalog drop-down list.

Select a collection method

SLS allows you to collect data from multiple sources, such as servers, applications, open source software, IoT devices, mobile devices, and Alibaba Cloud services. You can also collect data that is transferred over standard protocols. You can use Logtail to collect log data. You can also use other collection methods, such as the Kafka protocol and the Syslog protocol, to upload logs to the automatically created Logstore. For information about log collection methods, see Data collection overview.

If you use Logtail to collect logs, you must select a network type before you can perform the following operations. For more information, see Select a network type.

Configure Logtail to collect logs

In this section, Chaitin WAF is used an example. For more information about how to collect logs from other security services, such as FortiGate Firewall, see the "References" section.

Servers on which Chaitin WAF is deployed do not support Logtail. Before you can collect the logs of Chaitin WAF, you must deploy an intermediate server.

Configure an intermediate server and forward the logs of Chaitin WAF to the intermediate server by using the Syslog forwarding feature of Chaitin.
Install Logtail on the intermediate server. For more information, see Install Logtail on a Linux server and Install Logtail on a Windows server.
If the server that stores the logs of a security service supports Logtail, you can install Logtail on the server.
Configure a user identifier for the server on which Logtail is installed, and then grant your Alibaba Cloud account the permissions to collect logs from the server by using Logtail. For more information, see Configure a user identifier.
Create a machine group in the Logstore.
- For more information about how to create an IP address-based machine group, see Create an IP address-based machine group.
- For more information about how to create a custom identifier-based machine group, see Create a custom identifier-based machine group.
Configure data access.
1. Log on to the Simple Log Service console.
2. In the Quick Data Import section, click Import Data. In the Import Data dialog box, select Syslog-Plug-in.
3. In the Select Logstore step, set the Project and Logstore parameters to the names that you record in the 1. Create a Logstore dedicated for the CTDR feature section and click Next.
4. In the Create Machine Group step, click Use Existing Machine Groups.
5. In the Machine Group Settings step, configure the Scenario and Installation Environment parameters, select the created machine group, and then click Next.
6. In the Data Source Configuration step, configure the Plug-in Config parameter by using the following sample code and click Next.
  Important
  - You must remove the comments from the following sample code.
  - In the following sample code, Logtail plug-ins expand the JSON object in the _content_ field of logs of Chaitin WAF log into key-value pairs. This allows the CTDR feature to parse the fields that store the key-value pairs. You can also modify Logtail plug-ins to parse logs into key-value pairs based on your business requirements. For more information, see Expand JSON fields and Overview of Logtail plug-ins for data processing.
```
{
	"inputs": [{
		"type": "service_syslog",
		"detail": {
			"Address": "udp://0.0.0.0:9001", # If you use a server to forward multiple types of logs, change the port number based on the log type.
			"ParseProtocol": "rfc3164"
		}
	}],
	"processors": [{
		"detail": {
			"ExpandArray": false,
			"ExpandConnector": ".", # Configure this field based on your business requirements. 
			"ExpandDepth": 0,
			"IgnoreFirstConnector": true,
			"KeepSource": false, # This field specifies whether to retain the original log fields. Configure this field based on your business requirements.
			"KeepSourceIfParseError": true,
			"NoKeyError": true,
			"Prefix": "", # Configure this field based on your business requirements.
			"SourceKey": "_content_", # The name of the JSON-formatted field that you want to expand.
			"UseSourceKeyAsPrefix": false # Configure this field based on your business requirements.
		},
		"type": "processor_json"
	}]
}
```
7. Preview data, configure indexes, and then click Next.
In the Simple Log Service console, check whether logs are added to the Logstore.

Use other collection methods

In addition to Logtail, you can use the Kafka protocol and the Syslog protocol to upload logs to the Logstore. The following topics describe the methods:

3. Configure a log parsing rule

Log on to the Security Center console. In the top navigation bar, select the region of the asset that you want to manage. You can select China or Outside China.
In the left-side navigation pane, choose CTDR > Service Integration.
In the upper-right corner of the Service Integration page, click Log Parsing Rule.
On the Log Parsing Rule page, click Create Rule.

On the Add Log Parsing Rule page, configure the parameters.

Parameter type	Parameter	Description
Basic Information Settings	Rule Name	Enter a name for the rule.
	Activity Category	Select the types of the logs that you want to add. The following list describes the mappings between the log types and logs. Firewall alert logs: Security logs > Firewall alert logs Firewall flow logs: Network logs > 5-tuple logs WAF alert logs: Security logs > WAF alert logs WAF flow logs: Network logs > HTTP logs
	Remarks	Enter remarks for the rule.
View Sample Log	Region ID	Select the region in which the Logstore is created.
	Project	Enter the name of the project to which the Logstore belongs.
	Logstore	Enter the name of the Logstore.
	Load Samples	Click Load Samples to load the log fields that are collected from the security service to the rule list. After you load the log fields, you can directly select the imported log fields to configure mappings without the need to manually configure log fields. You can also click View More Sample Logs to go to the SLS console to view the details of the log fields that are collected from the security service.
Log Parsing and Mapping Settings	Parameters used to configure the parsing rule	Configure the mapping fields to map the collected log fields to the standard log fields of the CTDR feature. You cannot change the values of the Target Standard Field field to the fields that you want to configure. You can configure other fields based on your business requirements. Log Filter: If you want to filter the log fields that are mapped to the CTDR feature based on the values of specific fields, select Advanced Settings and configure a filter condition. A filter condition consists of fields and a filter operator. To add another filter condition, click Add Filter Condition. View Standard Fields: Click this button to view the standard log fields that can be parsed by the CTDR feature and the log field details in the Standard Fields panel. Manage Mapping Dictionary: Click this button to manage mapping dictionaries in the Manage Mapping Dictionary panel. Parsing Function Documentation: Click this button to go to the documentation center of Security Center to view the details and scenarios of parsing functions.

4. Associate the Logstore with the parsing rule

On the Service Integration page, find the required security service and click Access Settings in the Actions column.
In the panel that appears, find the required log type and click the number in the Associated Accounts column.
In the Add Account panel, configure the Import Account and Region parameters. Then, click Automatically Create Logstore.
Note
If a Logstore of the specified log type is created in the specified region, the system does not create another Logstore even if you re-click Automatically Create Logstore.
Select the Log Parsing Rule that you create in the 3. Configure a log parsing rule section, click Check Validity, and then click Save.

What to do next

Configure custom detection rules to check and analyze the logs of security services. For more information, see Create a custom detection rule.
Note
Predefined detection rules are not supported by the logs of security services.
To view the log data that is standardized by the CTDR feature, use the log management feature. Before you can use the log management feature, you must purchase log storage capacity and enable log delivery for the required log type. For more information, see Step 1: Enable log delivery.

References

Logstores dedicated for the CTDR feature

After you perform the operations in the 1. Create a Logstore dedicated for the CTDR feature section, Security Center automatically creates a project and a Logstore in SLS that are dedicated for the CTDR feature. The Logstore is used to store the logs of security services and is not recommended for other purposes. Your permissions on the log data in the Logstore are limited. You do not have the permissions to modify the log data or create indexes in the Logstore. You can consume, transform, and ship log data in the Logstore.

Only one project can be created in each region. If you re-click Automatically Create Logstore, the system does not create another project or Logstore.

The name of the project created in the China (Hangzhou) region is aliyun-cloudsiem-data-Alibaba Cloud Account ID-cn-hangzhou. Security Center creates different Logstores based on the types of logs that you add. The following table describes the Logstores. Data access in the SLS console must be configured for the required Logstore of a security service.

Logstore	Description
cloud_siem_chaitin_waf_alert_log	Used to collect the alert logs of Chaitin WAF and store the key-value pairs that are parsed from the logs for structured storage.
cloud_siem_chaitin_waf_flow_log	Used to collect the flow logs of Chaitin WAF and store the key-value pairs that are parsed from the logs for structured storage.
cloud_siem_f5_ltm_alert_log	Used to collect the alert logs of F5 BIG-IP LTM and store the key-value pairs that are parsed from the logs for structured storage.
cloud_siem_fortinet_forigate_alert_log	Used to collect the alert logs of FortiGate Firewall and store the key-value pairs that are parsed from the logs for structured storage.
cloud_siem_fortinet_forigate_flow_log	Used to collect the flow logs of FortiGate Firewall and store the key-value pairs that are parsed from the logs for structured storage.
cloud_siem_microsoft_active_directory_log	Used to collect the logs of Microsoft AD and store the key-value pairs that are parsed from the logs for structured storage.
cloud_siem_other_cfw_alert_log	Used to collect the alert logs of firewalls from other security providers and store the key-value pairs that are parsed from the logs for structured storage.
cloud_siem_other_cfw_flow_log	Used to collect the flow logs of firewall services from other security providers and store the key-value pairs that are parsed from the logs for structured storage.
cloud_siem_other_waf_alert_log	Used to collect the alert logs of WAF from other security providers and store the key-value pairs that are parsed from the logs for structured storage.
cloud_siem_other_waf_flow_log	Used to collect the flow logs of WAF from other security providers and store the key-value pairs that are parsed from the logs for structured storage.

Example of Logtail plug-ins for collecting logs of FortiGate Firewall

You can use Logtail to collect the logs of FortiGate Firewall to the Logstore dedicated for the CTDR feature in the same manner as the log collection of Chaitin WAF. The following sample code shows how to configure Logtail plug-ins.

Note

In the following sample code, Logtail plug-ins expand the JSON object in the _content_ field of logs of FortiGate Firewall into key-value pairs. This allows the CTDR feature to parse the fields that store the key-value pairs. You can also modify Logtail plug-ins to parse logs into key-value pairs based on your business requirements. For more information, see Extract content from log fields and Overview of Logtail plug-ins for data processing.

{
    "inputs": [
        {
            "type": "service_syslog",
            "detail": {
                "Address": "udp://0.0.0.0:9002",
                "ParseProtocol": "rfc5424"
            }
        }
    ],
    "processors": [
        {
            "detail": {
                "Delimiter": " ",
                "Separator": "=",
                "KeepSource": true, 
                "SourceKey": "_content_",  
                "UseSourceKeyAsPrefix": false 
            },
            "type": "processor_split_key_value"
        }
    ]
}

Use a server to collect logs of multiple security services

If you want to add the logs of multiple security services to the CTDR feature, you can forward the logs from the log sources to a server by using a specified IP address and multiple ports, install Logtail on the server, and then configure data access for the related Logstores in the SLS console. This way, you can collect different types of logs to the Logstores dedicated for the CTDR feature.

For example, if you want to add the alert logs of Chaitin WAF and FortiGate Firewall to the CTDR feature, you can use syslog agents such as rsyslog to forward the required syslogs to a server by using a specified IP address and different ports. For example, you can forward the alert logs of Chaitin WAF to the server over port 9001, and the alert logs of FortiGate Firewall over port 9003. After you install Logtail on the server, use the following information to complete the Logtail configuration in the related Logstores in the SLS console.

Alert logs of Chaitin WAF: In the Logstore named cloud_siem_chaitin_waf_alert_log, enter the following content to configure Logtail:

{
     "inputs": [
         {
             "type": "service_syslog",
             "detail": {
                 "Address": "udp://0.0.0.0:9001",
                 "ParseProtocol": "rfc3164"
             }
         }
     ]
 }

Alert logs of FortiGate Firewall: In the Logstore named cloud_siem_fortinet_forigate_alert_log, enter the following content to configure Logtail:

{
     "inputs": [
         {
             "type": "service_syslog",
             "detail": {
                 "Address": "udp://0.0.0.0:9003",
                 "ParseProtocol": "rfc3164"
             }
         }
     ]
 }

Log collection

In addition to the log collection method by using Logtail described in this topic, you can use other collection methods based on your business requirements. For more information, see Data collection overview.
For more information about how to troubleshoot issues when no heartbeat is detected in machine groups while data is collected from the machine groups, see How do I troubleshoot an error related to a Logtail machine group in a host environment?