Build a Cloud SIEM Scheme Based on the SLS Platform and Log Audit

By Yemo

What Is SIEM?

Definition of SIEM

Security information and event management (SIEM) performs threat detection, security event management, and compliance detection by collecting and analyzing security events from various data sources. SIEM is developed based on security information management (SIM). It collects, analyzes, and reports log data, and security event management (SEM), which performs real-time analysis of logs and event data to provide threat monitoring, event correlation, and event response.

2021 Gartner's Magic Quadrant for SIEM analysis report gives a new definition of the SIEM market.

Collects security event logs and telemetry data in real-time (streams and data packets) for threat detection and compliance use cases
Analyzes data in real-time and detects attacks and other activities of interest continuously
Investigates security events to determine their potential severity and impact on the business
Reports activities above
Stores related events and logs

SIEM technology focuses on event data generated by security devices, network infrastructure, systems, and applications. Data sources are mainly log data, but SIEM technology can also process other forms of data, such as network telemetry data (streams and data packets.) SIEM also provides real-time event analysis, advanced analysis of users and entity behaviors for security monitoring, historical data analysis for large time spans, event investigation and management, and reports (such as compliance requirements.)

SIEM Magic Quadrant

Magic Quadrant is a tool for Gartner to evaluate and compare vendors in the industry. It generates Magic Quadrant diagrams and corresponding analysis reports based on the ability to execute and the completeness of vision of each vendor in the industry market. In the diagram, the vertical axis shows the vendor's ability to execute, which is used to evaluate the vendor's ability to turn its vision into a market reality. The horizontal axis represents the vendor's completeness of vision. Vendors are positioned into four quadrants (LEADERS, CHALLENGERS, VISIONARIES, and Niche Players) based on the vendor's ability to execute and the completeness of vision.

The picture above shows the SIEM Magic Quadrant provided by Gartner. Next, we will focus on several important vendors.

Important SIEM Vendors

Exabeam is a leader in the Magic Quadrant. Exabeam's SIEM solution can be used as SaaS (Exabeam Fusion SIEM) or used for hybrid and joint deployment. It includes Exabeam data lake, advanced analysis, threat detection, entity analysis, case management, and event response. Users can purchase it flexibly based on the modular architecture of the customized deployment. Exabeam's machine learning (ML)-driven user and entity behavior detection can provide users with risk scoring and automatic context enrichment capabilities.

Securonix is in the leading position in the Magic Quadrant. Its SIEM solution includes next-generation SIEM, secure data lake, UEBA, SOAR, NDR, threat intelligence, competitor behavior analysis, and several applications for specific use cases. It provides good support in data privacy control and threat intelligence.

Splunk is a leader in the Magic Quadrant. Splunk's security product portfolio has been rated as an industry-leading technology by Gartner Market Research for many years. Through the platform of "Data-to-Everything," a complete set of solutions integrating SIEM, UEBA, and SOAR is provided.

Elastic is a niche player in the Magic Quadrant. Based on open-source, Elastic has developed basic data collection, analysis, and visualization capabilities through the combination of Logstash, Elasticsearch, and Kibana. As a log aggregator, Logstash can collect and process data from almost any data source.

Elasticsearch is a storage engine for parsing large amounts of data. Kibana is used as a visualization layer for visualized processing and problem analysis. Elastic 7.14 released the first free and open Limitless XDR, which can provide integrated SIEM and Endpoint Security functions in one platform.

SLS vs. Splunk

Note: Splunk offers out-of-the-box solutions on a platform basis (such as its complete SIEM, UEBA, and SOAR solutions.)

SLS is more of a platform that provides support and secondary customization and development. This article mainly compares the functions of SLS and Splunk at the platform level.

Splunk Data-to-Everything Platform

Splunk provides a powerful "Splunk Data-to-Everything Platform" to help users break through the barriers between data and actions. Moreover, it empowers IT, security, Internet of Things, and business operations teams to understand business in real-time and transform data into business results. The platform mainly includes:

Data Stream Processor provides a real-time stream processing solution, which realizes unified collection and processing of data and supports multi cloud.
A high-availability cluster architecture that can be elastically scaled to respond to business changes quickly
Flexible SPL statements realize quick searches in long-time spans and massive data. It enables efficient problem analysis and cause location.
Schema on read design can apply data to a schema and perform data processing when data is being exported from the database (rather than when data is being imported to the database), realizing quick data fetch.
Monitor and analyze applications and centralized consoles to detect exceptions in the early warning process promptly and give notifications
Based on role-based access control, it controls access to applications, dashboards, views, and field-level data to ensure the security of user access and data usage.
Provides AI functions out-of-the-box based on ML and SPL
Provides a complete security orchestration mechanism and automation and response (SOAR) solutions to help users automate security operations

This platform provides data support for decision-making related to IT, security, observability, problems, and actions.

SLS Cloud-Native Observability Platform

SLS is a log middle office product of Alibaba and Ant Group. It is also a basic product that serves tens of thousands of Alibaba Cloud customers. SLS processes dozens of petabytes of Log/Metric/Trace data every day. It provides support for AIOps, big data analysis, operations services, big data security, and other scenarios and enhances observability for engineers.

The core platform capability of SLS is to provide unified storage and computing capabilities related to various monitoring data of observability.

Unified storage capacity provides unified storage capabilities for log, metric, trace, and other data.
Powerful analysis engine
- Data Processing Engine (DSL): It is mainly for data processing and preprocessing scenarios and solving the problem of diverse formats.
- SQL Query Analysis Engine (SQL): It provides clear computing capabilities for stored data.
- Intelligent Analysis Engine (AIOps): It provides intelligent algorithms for specific problems.
- Rich Visualization Capabilities: It visualizes the overall status of the system.
- It provides event management and on-call management mechanisms to monitor and notify exceptions promptly.

Based on this platform, it provides rich scenario support in DevOps, ITOps, SecOps, BusinessOps, and other fields.

Function Comparison of SLS and Splunk

In the following table, the sentences highlighted in red indicate better performance:

Based on the comparison above, SLS already has relatively comprehensive capabilities in the functional points in building SIEM solutions. It performs well in some aspects, including service availability, price, data processing and analysis capabilities, and alarm management. Next, we will focus on how to use SLS to build a SIEM scheme.

SLS Features and Scenarios

SIEM Functions and SLS Capability Mapping

It can be seen from this figure that SLS already has the basic ability to build SIEM. Next, we focus on the key points of using SLS to build SIEM scenarios from the following aspects:

Rich data collection and processing capabilities
Unified Query and Analysis Capabilities: It offers interactive query analysis syntax, ML algorithm support, and visualization analysis capabilities.
Threat Detection and Response: It adopts built-in alarm rules and customized rules for threat detection. It notifies users of discovered threat events and manages the events.

Data Collection and Processing Capabilities

Real-Time Data Collection

SLS provides a wide range of collection methods to collect data from multiple sources, such as servers and applications, open-source software, Internet of Things devices, mobile terminals, standard protocols, and Alibaba Cloud products.

The main access methods for application logs are:

Client writes directly using SDK: It allows writing logs directly to SLS in applications or dependent frameworks.
- Language Coverage: SDK is provided for multiple language versions, including .NET Java Python PHP and the C language.
- Scenario Coverage: It contains server application, mobile terminal, and H5.
Collected by Logtail as an agent: The application is only responsible for sending logs to the local, and log is used as an agent to collect logs to the SLS side.
- Scene Coverage: Linux, Windows, and Docker
- Ability Support: It supports the collection of multiple text types (such as regular and JSON.) Logtail provides simple data processing capabilities (such as field fetch and filtering) and desensitization capabilities for security data collection. Please refer to sensitive_keys configuration items for details.

It also provides docking of standard protocols (such as HTTP/HTTPS and Syslog) and third-party open-source collection software (such as Logstash and Fluentd.) SLS provides 20+ access points worldwide to meet the demands of quick access of service data and provides a global accelerator scheme for the unstable cross-domain collection network.

SLS is also deeply integrated with multiple cloud products of Alibaba Cloud, providing convenient access to cloud product logs. Users can directly collect the audit data of each cloud product to the SLS side for subsequent analysis and processing.

The list of cloud products that support access continues to grow. For the latest data, please refer to:

Data Processing

Data processing is a managed, highly available, and scalable data processing service launched by SLS. It mainly performs real-time ETL processing on structured or unstructured logs. This function currently includes 200+ operators and is widely used in scenarios, such as data regulation, data aggregation, enrichment, distribution, and aggregation. The data desensitization operators provided by data processing can reduce the exposure of sensitive data when users process, ship, and use the data. Therefore, it can mitigate the risk of data leakage. Common desensitization scenarios include desensitization of sensitive information, such as mobile phone numbers, bank card numbers, email addresses, IP addresses, AccessKey pairs, ID numbers, URLs, order numbers, and strings.

In SIEM scenarios, it is often necessary to collect data from multiple data sources since the format may be messy. At the same time, there are also long-term and massive data analysis requirements. Data processing is structured in advance through the Schema on Write method, which can be convenient for subsequent analysis and processing.

Log Audit

Log Audit is an audit service based on the capabilities of the SLS platform. It supports the automatic and centralized collection of cloud product logs on Alibaba Cloud in multi-account scenarios in real-time. It has the following advantages:

It supports a wide range of cloud products. Currently, it covers basic (ActionTrail and Container Service for Kubernetes), storage (OSS and NAS), network (SLB and API Gateway), databases (ApsaraDB RDS, cloud-native distributed database PolarDB-X 1.0, and cloud-native database PolarDB), and security (WAF, Anti-DDoS, Cloud Firewall, and Security Center.)
Cross-Account: Integrated with Alibaba Cloud Resource Management, users can collect logs from multiple primary accounts to a central primary account.
One-Click Collection: After users configure collection rules at one time, they can discover new resources and collect logs in real-time automatically.
Centralized Storage: Logs are collected and stored in a centralized project in a certain region. This helps users query, analyze, and visualize the collected logs more efficiently. Users can also configure alerts for the logs and perform secondary development.
Rich built-in alarm rules to create a secure network with one click
Eco-Open Docking: Users can connect with open-source software, Alibaba Cloud big data products, and third-party SOC software to extract more value from data.

Query Analysis Capability

Query Analysis Engine

SLS uses SQL as the query and analysis framework and integrates PromQL syntax and machine learning functions into the framework.

SLS SQL = Search + SQL92 (Agg, WIndow, GroupBy...) + PromQL + ...

In the following example:

First, call the PromQL operator to get the monitoring value of the host per minute.
Downsample the original data through the window function. For example, transform the data into a value per second.
The query results are predicted by the prediction function of the outer layer.

Machine Learning Enhancement

SLS has a large number of AI-based inspection, prediction, clustering, root cause analysis, and other algorithms, which are provided to users in the form of SQL/DSL functions and can be used in manual analysis and automatic inspection alarms.

Interactive Visualization

The visualization function provided by SLS is based on the unified query and analysis engine of SLS, which presents the query and analysis results in the form of charts, and shows the global situation clearly. At the same time, it can also interface with third-party visualization tools.

Threat Detection and Response

Management and Maintenance System of Intelligent Alerting

SLS alert is an all-in-one intelligent management and maintenance alert system provided on the SLS cloud-native observability platform. It provides alert monitoring for logs, time series, and other data. It can also accept tripartite sides alerts. It performs noise reduction, event management, and notification management for alerts. It adds 40+ functional scenarios, fully considering the management and maintenance of alert monitoring needs for research and development, management and maintenance, security, and operational staff.

Users can configure alarm monitoring rules to check, evaluate, and query the source logs and timing storage periodically and trigger alarms or recovery notifications according to the evaluation results of the monitoring orchestration logic.

SLS alert provides hundreds of built-in alert rules, which are available out of the box and are continuously increasing. These rule bases cover best practices for CIS (including account security and database security) and security scenarios. Users can enjoy all-time security by enabling the corresponding rules.

When an alert rule detects an exception, it is necessary to notify the corresponding developers of the threat event as soon as possible. SLS provides various notification channels to facilitate all-around access to threat events.

When developers receive an alert event notification, they need to take a series of event management actions, such as event confirmation, assignment of handlers, and recording of processing actions.

Threat Intelligence

Gartner views threat intelligence as some evidence-based knowledge, including context, mechanism, label, meaning, and proposals that can be implemented. Threat intelligence describes existing or imminent threats or danger to an asset. It can also be used to inform the entities to take a certain response to the threats or danger. Most of the so-called threat intelligence in the industry is in a narrow sense. Its main contents are the fall identification used to identify and detect threats, such as file HASH, IP, domain name, program running path, registry key, and some related attribution tags.

Threat intelligence service is an intelligence security service provided by Alibaba Cloud. It combines threat intelligence data to collect, analyze, classify, and associate threat sources automatically in real-time to evaluate threats in enterprise assets and provide suggestions for improving security. Alibaba Cloud threat intelligence service can display the threat statistics of all online users and enterprises worldwide over the last 30 days. Currently, it supports providing threat intelligence for IP addresses, domain names, and files.

SLS Log Audit is deeply integrated with threat intelligence service. With the global threat intelligence assessment capability provided by the threat intelligence service, it performs threat intelligence detection on a wide range of cloud product logs (ActionTrail, SLB, OSS, and SAS) that access the SLS, identifying potential threats that exist during the use of cloud products effectively. It also supports timely notification of detected exceptions to relevant security personnel in the form of alerts, improving the efficiency and response speed of threat inspection.

The following figure shows how to use the SLS log audit service to detect and respond to threat intelligence of cloud products.

Integration with Third-Party SIEM/SOC Platforms

SLS provides a way to interface with third-party SIEM solutions (such as Splunk) to ensure that all regulations, audits, and other related logs on Alibaba Cloud can be imported into the user's security management and maintenance center (SOC).

SLS provides Alibaba Cloud Log Service Add-on for Splunk for Splunk access scenarios, which is convenient for users to deploy.

References

Gartner SIEM 2021 - SIEM Gartner Magic Quadrant

Community