All Products
Search
Document Center

Simple Log Service:Data processing

Last Updated:Dec 17, 2024

To process data before writing it into a Logstore, such as data filtering, field extraction, field extension, and data masking, you can use an ingest processor.

Working principle

image

Scenarios

  • Field extraction: Extract new fields from the original log fields using parsing methods such as regular expressions, key-value format, and JSON.

  • Field extension: Add new fields to the original logs.

  • Field dropping: Delete some fields from the original logs.

  • Data masking: Mask sensitive information in the original logs.

  • Data filtering: Discard some data from the original logs.

Limits

For quota limits on the number of ingest processors and SPL statement length, see Ingest processor.

Billing

The ingest processor is billed based on the amount of computational resources consumed during data processing. The billing unit is Observability Capacity Unit (OCU).

OCU is a billing unit proposed by Managed Service for Grafana. The OCU usage is automatically calculated based on the hourly resource usage.

The computational functions of Simple Log Service are gradually being billed based on OCU, with the actual consumption of computing resources by the user serving as the measurement dimension. The performance of one OCU is approximately equivalent to 0.5 Core CPU, 2 GB of memory, and 3000 IOPS. When calculating the total number of OCUs, the system separately calculates the OCU quantities based on the CPU cores, memory size, and IOPS consumed. The maximum value among these three quantities is taken as the final OCU value for billing purposes.

For more information about OCU, see Billable items of pay-by-ingested-data and Billable items of pay-by-feature.