Text logs are a basic log type. This topic describes how to use Logtail to collect logs, and query and analyze the collected logs. In the following example, Logtail is used to collect the text logs of an Alibaba Cloud Elastic Compute Service (ECS) instance.
Prerequisites
An ECS instance is available. For more information, see Create and manage an ECS instance in the console (express version).
Simple Log Service is activated. For more information, see Activate Simple Log Service.
Background information
In the following example, Logtail is used to collect logs from the /var/log/nginx/access.log file.
The following sample code provides sample logs in the /var/log/nginx/access.log file:
127.0.*.1 - - [20/Mar/2023:12:00:01 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36"
192.168.*.1 - - [20/Mar/2023:12:00:02 +0000] "GET /about.html HTTP/1.1" 200 1435 "http://example.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 15_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Mobile/15E148 Safari/604.1"
10.0.*.1 - - [20/Mar/2023:12:00:03 +0000] "POST /login HTTP/1.1" 302 0 "http://example.com/login.html" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.88 Safari/537.36"
127.0.*.1 - - [20/Mar/2023:12:00:04 +0000] "GET /images/logo.png HTTP/1.1" 200 45678 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
192.168.*.2 - - [20/Mar/2023:12:00:05 +0000] "GET /contact.html HTTP/1.1" 404 155 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15"
10.0.*.2 - - [20/Mar/2023:12:00:06 +0000] "GET /style.css HTTP/1.1" 200 12345 "http://example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.82 Safari/537.36"
127.0.*.1 - - [20/Mar/2023:12:00:07 +0000] "GET /favicon.ico HTTP/1.1" 404 554 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36"
192.168.*.3 - - [20/Mar/2023:12:00:08 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.79.1"
10.0.*.3 - - [20/Mar/2023:12:00:09 +0000] "POST /api/data HTTP/1.1" 201 123 "http://example.com/dashboard" "PostmanRuntime/7.For more information, see Collect text logs from servers.
Procedure
To collect the text logs from a specified file of a server, perform the following steps:
Create a project and a Logstore: A project in Simple Log Service is used to isolate resources and control access to specific resources. A Logstore in Simple Log Service is used to collect, store, and query logs.
Create a Logtail configuration: You can create a Logtail configuration to install Logtail on an ECS instance and use Logtail to collect logs. Logs are collected to a specified Logstore in Simple Log Service.
Query and analyze logs: You can query and analyze the collected logs in the Simple Log Service console. For example, you can retrieve logs, aggregate logs, and collect statistics on logs.
You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Search syntax.
Logtail collects only incremental logs. If a log file on your server is not updated after the related Logtail configuration is applied to the server, Logtail does not collect logs from the file. To update the log file, you can add a log to the log file. For more information, see Log collection process of Logtail.
Step 1: Create a project and a Logstore
1.1 Create a project
Log on to the Simple Log Service console. In the Projects section, click Create Project. In the Create Project panel, select a region from the Region drop-down list and specify a name for the Project Name parameter. Retain the default values for other parameters. The region that you select must be the same as the region where the ECS instance resides. For more information, see Create a project.
1.2 Create a Logstore
After you create the project, you are prompted to create a Logstore.
In the Create Logstore panel, specify a name for the Logstore Name parameter. Retain the default values for other parameters. For more information, see Create a Logstore.
Step 2: Create a Logtail configuration
You can create a Logtail configuration to install Logtail on the ECS instance and use Logtail to collect logs. Logs are collected to the specified Logstore in Simple Log Service.
2.1 Select a log collection method
In the Created message box, click OK.
On the Self-managed Open Source/Commercial Software tab of the Quick Data Import dialog box, find Single Line - Text Logs and click Integrate Now.
2.2 Create a machine group
In the Machine Group Configurations step of the Logtail configuration creation wizard, set Scenario to Servers and set Installation Environment to ECS. Then, click Create Machine Group. In the Create Machine Group panel, select the ECS instance and click Install and Create Machine Group. If the region of the project is different from the region where the ECS instance resides, see Install Logtail on a Linux server.
If Success is displayed in the Logtail Installation Status column, Logtail is installed on the ECS instance. If the installation fails, click Recreate Installation Task. Reselect the ECS instance and install Logtail again.
After Logtail is installed, specify a name for the Name parameter. You do not need to specify an IP address in the IP Address field because the system automatically fills the IP address of the ECS instance on which Logtail is installed. For more information, see Create an IP address-based machine group.
Check the heartbeat status of the machine group. Approximately 2 minutes are required to create a machine group. If the machine group is not created, the heartbeat status of the machine group is FAIL After 2 minutes, click Automatic Retry to refresh the heartbeat status of the machine group until the heartbeat status becomes OK. For more information about how to troubleshoot the issue that a machine group does not have a heartbeat, see How do I troubleshoot an error that is related to a Logtail machine group in a host environment?
2.3 Create a Logtail configuration
Create a Logtail configuration based on the following figure. Specify a name for the Configuration Name parameter and a file path for the File Path parameter, and add a sample log that is provided in the Background information section of this topic. For more information, see Global Configurations.
2.4 Configure settings for data query and analysis
Approximately 3 minutes are required to create a Logtail configuration. Click Refresh. If you can preview data, the Logtail configuration is created. After the Logtail configuration is created, click Next. The Logtail configuration-related settings are complete.
Step 3: Query and analyze logs
In the End step of the Logtail configuration creation wizard, click Query Log. Then, you are navigated to the query and analysis page of the specified Logstore. You can view the logs that are collected from the /var/log/nginx/access.log file. In the search box, enter an arbitrary character, such as HTTP, specify a query time range, and then click Search & Analyze. You can obtain the logs that contain the arbitrary character.
References
For more information about how to collect text logs from servers, see Collect text logs from servers.
For more information about how to collect Kubernetes logs, see Collect text logs from Kubernetes containers in DaemonSet mode and Collect text logs from Kubernetes containers in Sidecar mode.
For more information about how to collect logs of Alibaba Cloud services, see Alibaba Cloud service logs.
You can save important query and analysis results as charts to a dashboard. For more information, see Create a dashboard.
For more information about how to configure alerts for logs, see Configure an alert rule in Simple Log Service.
For more information about how to download logs, see Download logs.
FAQ
Am I charged if I only create projects and Logstores?
By default, shard resources are reserved when you create a Logstore. You are charged for active shards. For more information, see Why am I charged for active shards?
What do I do if logs fail to be collected?
When you use Logtail to collect logs, a failure may occur due to Logtail heartbeat failures, collection errors, or invalid Logtail configurations. For more information, see What do I do if errors occur when I use Logtail to collect logs?
What do I do if I can query logs but cannot analyze logs on the query and analysis page of a Logstore?
If you want to analyze logs, you must configure indexes for log fields and turn on Enable Analytics for the fields. For more information, see Create indexes.