Simple Log Service provides a real-time consumption feature that lets you use software development kits (SDKs) to consume data in real time. This topic describes the concept, benefits, scenarios, billing rules, and consumers of this feature.
Real-time consumption
Real-time consumption allows third-party software, multi-language applications, cloud products, and stream processing frameworks to use SDKs to consume data from Simple Log Service in real time. This feature sequentially reads and writes full data, similar to the Kafka middleware. You can use Structured Process Language (SPL) statements for data processing, and the server-side returns the results. For more information, see Consume logs based on rules.
Both real-time consumption and query and analysis are used to read data. For more information about the differences between them, see Differences between log consumption and queries.
Scenarios
Real-time consumption is suitable for scenarios such as stream processing and real-time computing. This feature offers low latency, and data is typically processed within seconds. You can customize the data retention period.
Benefits
Real-time consumption provides the following benefits:
Centralized data storage
Simple Log Service centralizes data from different machines. You can simply use an SDK to consume the collected data in real time.
Data classification
You can use the data classification feature of Simple Log Service. This allows different applications and products to consume different types of data from different projects in real time.
Billing rules
If the Logstore uses the pay-by-ingested-data billing mode, you are not charged for real-time consumption. However, if you pull data from a Simple Log Service public endpoint, you are charged for outbound internet traffic. The traffic is calculated based on the compressed data volume. For more information, see Billable items in pay-by-ingested-data mode. To view the billing mode of a Logstore, see Manage a Logstore.
If the Logstore uses the pay-by-feature billing mode, real-time consumption incurs charges for multiple billable items, including read and write traffic and requests. For more information, see Billable items in pay-by-feature mode.
Consumption target
The following tables describe the consumers that are supported for real-time consumption in Simple Log Service.
Type | Objective | Description |
Third-party software | Flume | You can use Flume to consume data from Simple Log Service in real time. For more information, see Consume data using Flume. |
Logstash | You can use Logstash to consume data from Simple Log Service in real time. For more information, see Consume data using Logstash. | |
QRadar | Security information and event management (SIEM) systems, such as QRadar, can consume data from Simple Log Service in real time over the HTTPS or Syslog protocol. For more information, see Deliver logs to a SIEM system and Deliver logs to a SIEM system over Syslog. | |
Multi-language applications | Multi-language applications | Applications developed in languages such as Java, Python, and Go can consume data from Simple Log Service as consumers or consumer groups. For more information, see Consume data using an API and Consume logs using a consumer group. |
Stream processing | Flink | You can use the Flink stream processing framework to consume data from Simple Log Service in real time. For more information, see Consume data using Flink. |
Spark | You can use the Spark stream processing framework to consume data from Simple Log Service in real time. For more information, see Consume data using Spark Streaming. | |
Cloud products | Function Compute | You can use Function Compute to consume data from Simple Log Service in real time. For more information, see Consume log data using Function Compute. |
Blink | You can use real-time computing to consume data from Simple Log Service in real time. For more information, see Consume data using Realtime Compute for Apache Flink. |