This topic describes how to use Simple Log Service to query and analyze application logs in different scenarios, such as log query, association analysis, and statistical analysis.
Background information
Application logs include important statistical information about application operations and maintenance. Application logs have the following characteristics:
Inconsistent log styles. Application developers have different styles when they write code. Therefore, logs that are generated by different applications have inconsistent styles.
Large data size. The size of application logs is one order of magnitude larger than the size of access logs.
Distribution on multiple servers. Most applications are stateless and run on different frameworks, such as cloud servers and container services. These applications may be deployed on a few to thousands of instances. Therefore, a cross-server solution for log collection is required.
Complex runtime environments. Applications are running in different environments, and application logs are stored in different environments. For example, application-related logs are stored in containers, API-related logs are stored in Function Compute, old system logs are stored in data centers, mobile app logs are stored in mobile terminals, and website logs are stored in browsers.
Before you can obtain full logs, make sure that all application logs are stored in the same environment. Simple Log Service provides multiple log collection methods and the log analysis feature. You can analyze logs from end to end in real time by using the query statements and SQL-92 syntax. You can also visualize the query results in charts. The query and analysis solution that is provided by Simple Log Service in this topic costs only 25% of an open source solution.
Query application logs
For example, an app encounters an order error or request latency. You can use a search statement to query logs that contain terabytes of data and locate the issue within 1 second. You can also specify a query time range and keywords in the search statement to obtain more precise results.
You can execute the following search statement to query the logs of POST requests whose latency exceeds 1 second:
Latency > 1000000 and Method=Post*You can execute the following search statement to query the logs that include the error keyword and exclude the merge keyword:
error not merge
Perform association analysis on application logs
Association analysis supports intra-process association analysis and cross-process association analysis. The two types have the following differences:
Intra-process association analysis: The logs of a process are stored in the same log file. In a multi-threaded process, you can filter logs by thread ID.
Cross-process association analysis: By default, the cross-process association is unclear if multiple processes are used to handle a request. In this case, you can initiate a remote procedure call (RPC), which carries the TracerId parameter. You can use this parameter to associate the processes.
Intra-process association analysis
You can use contextual query to view the associated logs. For example, you can specify a keyword to find an error log. Then, click Context View to view N logs before and after the log. For more information, see Contextual query.
The following figure shows the contextual query results.
Cross-process association analysis
Cross-process association analysis is also referred to as tracing. Common tracing tools include EagleEye, Dapper, Stackdriver Trace, Zipkin, Appdash, and X-Ray.
In this example, tracing is implemented based on Simple Log Service. You can configure settings to ensure that the logs of different modules contain fields that can be used to associate the module logs. The fields include Request_id and OrderId. You can query related logs in different Logstores.
You can use SDKs to query logs that are collected from different modules, such as frontend servers, backend servers, payment systems, and ordering systems. After you obtain the query results, you can create a frontend page to associate the results.
Perform statistical analysis on application logs
After you query and obtain logs, you can perform statistical analysis on the logs.
You can execute the following query statement to analyze the types of all errors and the distribution of all errors by location:
__level__:error | select __file__, __line__, count(*) as c group by __file__, __line__ order by c desc
What to do next
Back up logs
You can back up the obtained logs to other cloud services, such as Object Storage Service (OSS) and MaxCompute.
Configure alerts
Grant permissions
You can grant permissions to a RAM user to isolate the development environment and the production environment.