Simple Log Service:Overview of Log Audit Service (new version)

Background information

In recent years, with the widespread application of cloud computing and the deep popularization of enterprises migrating to the cloud, many enterprises or individual users host various logs on the cloud for querying, auditing, and other operations.

• Log Audit is a mandatory requirement of laws and regulations in many countries. Enterprises or organizations need to continuously implement and comply with various laws and regulations to ensure the security of network data, protect user information rights, and maintain the security of enterprise infrastructure.

• Data compliance management is also a professional and complex issue that many enterprises or organizations need to face. In today's globalized world, enterprise data may be stored in various regions or organizations worldwide. On the one hand, data management needs to be based on legal and regulatory requirements to ensure data security, protect personal information rights, and maintain national security and social public interests. On the other hand, for production practice needs, data needs to flow freely in an orderly manner to maximize its value.

• In this context, we have launched the new version of Log Audit Service, which enhances data access capabilities for both cloud product logs and runtime logs. With the multi-log project center, users can centralize, query, count, and analyze logs while ensuring compliance with regional data regulations. This improves the management of lawful and orderly data flow.

Basic features

Simple Log Service platform supports automated log management and has capabilities such as log query, analysis, storage, transformation, delivery, consumption, alerting, and visualization. Based on the Simple Log Service platform, Log Audit Service also supports the following features.

Collect cloud product logs

• Cloud product coverage: Log Audit Service supports a variety of mainstream Alibaba Cloud products and their log types, including Storage (Simple Log Service, Object Storage Service), Networking (Classic Load Balancer, Application Load Balancer, Virtual Private Cloud, Alibaba Cloud DNS), Databases (ApsaraDB RDS, PolarDB), Security (Security Center, Cloud Firewall, Web Application Firewall, Anti-DDoS), Audit (ActionTrail, Cloud Config), and related log types.

• Automated log collection: Log Audit Service supports automated collection of user logs. On the one hand, once a cloud product is connected to the new version of Log Audit Service, new logs are automatically written to the target Logstore; on the other hand, if there are new instances or attribute changes under the cloud account, the logs of the cloud product are automatically collected and written according to the user's configured collection rules.

• Configurable collection rules: Log Audit Service supports three resource matching modes to filter cloud product resources: all resources, filtering resources by instance mode, and filtering resources by attribute mode. Users can choose according to their actual needs.

• Cross-region data aggregation: Log Audit Service achieves cross-region aggregation of log data through the data transformation feature. Simple Log Service automatically creates data transformation tasks, which are created based on the delivery relationship between the default destination Logstore and the centralized destination Logstore configured by the collection rules.

• Support for multiple log centers: Logs only need to be ingested once and can be aggregated into various log projects or Logstores according to the configured rules through data transformation. For example, users can aggregate logs from the China (Beijing) and China (Hangzhou) regions to the China (Shanghai) region and logs from Malaysia (Kuala Lumpur) and Indonesia (Jakarta) regions to the Singapore region.

• Support for multi-account feature of Resource Directory: Administrators or delegated administrators can aggregate all logs of member accounts through multi-account collection configuration.

Collect runtime logs

• Runtime log collection: Log Audit Service supports collecting runtime logs captured by open-source agents (Tetragon, Falco) into the Logstore by using Logtail.

• Collection of systemd journal logs: Log Audit Service supports the use of Logtail to collect systemd journal logs, and can gather journal logs from the hosts in a containerized manner. This approach is suitable for Docker and Kubernetes environments.

Working principles

Collect cloud product logs

1. The logs of cloud products are first stored in their respective default Logstores. These logs include the operation information and running status of the cloud products.

   • For the log types and corresponding Logstore names of cloud products, see Usage notes of cloud service log collection.

   • For each cloud product instance, if multiple collection rules are hit, as long as one collection rule matches, the instance logs will be collected into the default Logstore.

2. Log Audit Service automatically creates data transformation tasks to aggregate the logs of the default Logstore into the user's associated project.

   • The log types of cloud products are determined by Log Audit collection rules configured by the user.

   • Log Audit Service automatically detects new or changed cloud product instances in each region under the user's cloud account and synchronizes them to Log Audit collection rules.

3. If the administrator account or delegated administrator account of the Resource Directory configures multi-account collection rules, the cloud product logs are first collected into the default delivery destination Logstore of the member account's cloud product, and then aggregated into the central Logstore of the administrator account or delegated administrator account through automatically created data transformation tasks.

   • Collection rule 1: Collect logs based on region attribute, delivering the cloud product logs of China (Hangzhou) and China (Shenzhen) regions to the centralized Logstore xxx_log_center, which belongs to the centralized project center-A-cn-shanghai.

   • Collection rule 2: Collect logs based on region attribute, delivering the cloud product logs of the Singapore region to the centralized Logstore xxx_log_center, which belongs to the centralized project center-A-ap-southeast-1.

Collect runtime logs

In a Docker and Kubernetes environment, use Tetragon and Falco to collect container runtime logs to a directory file and standard output. Then, use Logtail to deliver these runtime logs to the Logstore.

Usage limits

• The new version of Log Audit Service is in the public preview stage. If you encounter any issues during use, submit a ticket.

Billing

While Log Audit Service itself is free, activating it incurs charges for log storage and log traffic, including the following fee types:

Ingestion types of cloud products

The current cloud product ingestion for Log Audit is mainly divided into the following types. For specific restrictions, see Usage notes of cloud service log collection.