To process data before writing it into a Logstore, such as data filtering, field extraction, field extension, and data masking, you can use an ingest processor.
Working principle
Log data is collected through Logtail, Simple Log Service SDK, or Web tracking. The data first goes through the ingest processor and then is written into the Logstore. The data processing is completed in Simple Log Service and does not occupy client resources.
Ingest processor and Data Transformation (new version) support Simple Log Service Processing Language (SPL) syntax.
Scenarios
Field extraction: Extract new fields from the original log fields using parsing methods such as regular expressions, key-value format, and JSON.
Field extension: Add new fields to the original logs.
Field dropping: Delete some fields from the original logs.
Data masking: Mask sensitive information in the original logs.
Data filtering: Discard some data from the original logs.
Limits
For quota limits on the number of ingest processors and SPL statement length, see Ingest processor.
Billing
The ingest processor is billed based on the amount of computational resources consumed during data processing. The billing unit is Observability Capacity Unit (OCU).
OCU is a billing unit proposed by Managed Service for Grafana. The OCU usage is automatically calculated based on the hourly resource usage.
The computational functions of Simple Log Service are gradually being billed based on OCU, with the actual consumption of computing resources by the user serving as the measurement dimension. The performance of one OCU is approximately equivalent to 0.5 Core CPU, 2 GB of memory, and 3000 IOPS. When calculating the total number of OCUs, the system separately calculates the OCU quantities based on the CPU cores, memory size, and IOPS consumed. The maximum value among these three quantities is taken as the final OCU value for billing purposes.
For more information about OCU, see Billable items of pay-by-ingested-data and Billable items of pay-by-feature.