Simple Log Service provides the text analysis feature to detect text content across large volumes of logs. This topic covers the feature's scenarios, terms, scheduling and execution use cases, and usage notes.
The Intelligent Anomaly Analysis application in Simple Log Service is being phased out and will no longer be available on July 15, 2025 (UTC+8).
Impact scope
Intelligent inspection, text analysis, and time series forecasting will no longer be available.
Feature replacement
The preceding features can be fully replaced by the machine learning syntax, scheduled SQL and dashboard features of Simple Log Service. Documentation will be provided to help you configure feature-related settings.
Background information
When a service runs, it creates many logs, including system and business logs, used for monitoring and troubleshooting. Traditional log analysis involves assessing risk levels and matching keywords such as Error, Failed, and Unsuccessfully. In distributed environments with microservices, analyzing these logs presents the following challenges:
Terabytes or even petabytes of logs are generated daily, making manual analysis labor-intensive.
In a distributed environment with microservices deployment, Warning logs or Error logs do not necessarily indicate system exceptions. These logs may be generated due to system scaling, updates, or iterations. Professional knowledge is required during manual analysis to identify anomalies in logs.
To address these challenges, automated and intelligent analysis of logs and troubleshooting are required. By solving these challenges, the potential of logs can be fully utilized, and the labor cost of log anomaly analysis can be reduced. Automated and intelligent analysis of logs has the following characteristics:
Processes large volumes of logs efficiently.
Identifies anomalies in logs or narrows down the scope of logs used for troubleshooting.
Allows you to customize parameters for text analysis.
To address these needs, Simple Log Service offers a text analysis feature for streamlined log integration and analysis. Implementing this feature requires configuring specific monitored objects and few algorithm parameters. The algorithm then automatically detects anomalies in logs, enabling you to concentrate on the important content.
Feature introduction
Text analysis supports pulling text content from logs through consumer groups without needing to configure indexes. Text analysis jobs retrieve data and input it into the text analysis model based on a scheduled rule. The model then writes the analysis results to the target logstore (internal-ml-log) and visualizes the results on a dashboard, helping you quickly understand the analysis outcomes.
Configure monitoring objects: Set up the log fields to be analyzed (where the field values are text content), then configure algorithm parameters as needed and initiate the task. Using a consumer group to configure log fields does not require enabling indexes.
Scheduled data analysis: The algorithms in text analysis process data using time windows.
Result output: The analysis results are output to the target logstore and a corresponding dashboard is generated to visualize the analysis results.
Terms
Term | Description |
Job | A text analysis job includes data features and algorithm model parameters. |
Instance | A text analysis job creates a text analysis instance based on the configuration of the job. The instance pulls data at regular intervals, runs the algorithm model, and then distributes the analysis result based on the configuration of the job.
For information about how different operations affect the scheduling and running of instances, see Scheduling and running. |
Instance ID | Each instance is identified by a unique ID. |
Creation time | Each instance is created at a specific point in time. In most cases, an instance is created for a text analysis job based on the scheduling rules of the job. If historical data needs to be processed or the delay caused by the timeout of the previous instance is offset, an instance is immediately created. |
Running time | Each instance starts to run at a specific point in time. If the job to which an instance belongs is retried, the start time is the most recent time at which the instance starts to run. |
End time | Each instance stops running at a specific point in time. If the job to which an instance belongs is retried, the end time is the most recent time at which the instance stops running. |
Status | Each instance is in a specific state at a specific point in time. Valid values:
|
Scheduling and execution use cases
Each job can create one or more instances. Only one instance can run in a job at a time regardless of whether the instance is run on schedule or is retried due to an anomaly. You cannot concurrently run multiple instances in a single job. The following list describes common scheduling and execution use cases:
Immediate start: Starting a text analysis job immediately means the algorithm model can't access historical data. It trains on the data from the configured initialization time windows, suppressing anomalies, and then dynamically updates with new data.
Modified schedule parameters: When you change a job's scheduling rules, a new instance is generated based on the updated settings. The model continues from the last analyzed point, handling new data.
Retry on failure: If an instance fails due to issues like permission errors, unavailable log stores, or configuration problems, Simple Log Service auto-retries. If an instance remains in the STARTING state, it indicates a configuration failure, and an error log is created. Check the configuration and retry. The instance status updates to SUCCEEDED or FAILED based on the retry outcome.
Usage notes
To improve the efficiency of text analysis:
Specify the text fields that you want to analyze in logs. If you specify many redundant fields, the analysis effectiveness may be compromised, and the analysis speed may decrease.
Obtain the changes in the time series data of monitored objects to check the stability and periodicity of the data and predict potential anomalies. This approach helps you appropriately configure parameters for the algorithm.