Method and procedure of downloading logs - Simple Log Service

Simple Log Service allows you to download logs or query and analysis results to your on-premises machine. This topic describes how to download logs or query and analysis results.

Download methods

You can download logs or query and analysis results by using the Simple Log Service console, Cloud Shell, Simple Log Service CLI, or Simple Log Service SDK.

Important

If you download data by using Simple Log Service CLI or Simple Log Service SDK, the amount of data that you can download is unlimited. However, interruptions may occur due to network errors.
If you want to download a compressed package of logs, you can use only the Simple Log Service console.
If your Logstore uses the pay-by-ingested-data billing mode, you are not charged when you download the query and analysis results of a query statement. For more information, see Pay-by-ingested-data.
If your Logstore uses the pay-by-feature billing mode, you are charged when you download the query and analysis results of a query statement because the Dedicated SQL feature is used during the download. You are charged for the Dedicated SQL feature. For more information, see Billable items of pay-by-feature.

Item	Download data by using the Simple Log Service console (recommended)	Download data by using Simple Log Service CLI	Download data by using Cloud Shell	Download data by using Simple Log Service SDK
Limit on data amounts	Query only: You can download up to 20 million lines of log data from a Logstore that resides in the China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), and Singapore regions. You can download up to 1 million lines of log data from a Logstore that resides in other regions. The total amount of the log data cannot exceed 20 GB. SQL analysis: You can download up to 100,000 lines of log data. The total amount of the log data cannot exceed 2 GB.	Query only: unlimited. SQL analysis: You can download up to 1 million lines of log data. The total amount of the log data cannot exceed 2 GB.	Query only: unlimited. SQL analysis: You can download up to 1 million lines of log data. The total amount of the log data cannot exceed 2 GB.	Query only: unlimited. SQL analysis: You can download up to 1 million lines of log data. The total amount of the log data cannot exceed 2 GB.
Required deployment	None.	Simple Log Service CLI is manually installed.	Cloud Shell is automatically deployed. The first time you run Cloud Shell, you need to wait until Cloud Shell is initialized.	Simple Log Service SDK is manually installed, and you need to write code based on your business requirements.
Authorization	The permissions to query data in the Simple Log Service console are used. No additional permissions are required.	Manual configuration.	Automatic configuration.	Manual configuration.
Dedicated SQL	When you download the query and analysis results of a query statement, the Dedicated SQL feature is used.	Not used.	Not used.	You can configure Dedicated SQL-related parameters based on your business requirements.
Read traffic over the Internet	None.	If Simple Log Service CLI is deployed on an Elastic Compute Service (ECS) instance that resides in the region of the required Simple Log Service project and an internal Simple Log Service endpoint is used, no fees are generated for read traffic over the Internet.	If your Simple Log Service project resides in the China (Shanghai) region, no fees are generated for read traffic over the Internet.	If Simple Log Service SDK is deployed on an ECS instance that resides in the region of the required Simple Log Service project and an internal Simple Log Service endpoint is used, no fees are generated for read traffic over the Internet.
Integration with File Storage NAS	None.	Manual configuration is required if necessary.	Automatic configuration.	Manual configuration is required if necessary.

You can ship logs to Object Storage Service (OSS) and download the logs in OSS. For more information, see Create an OSS data shipping job (new version).

Procedure

Download data by using the Simple Log Service console

You can download logs or query and analysis results to your on-premises machine by using the Simple Log Service console. The operations that are used to download logs are similar to the operations that are used to download query and analysis results. In this topic, logs are used to show the download procedure. If you want to download query and analysis results, you can click Download Log on the Graph tab after you execute a query statement.

Important

The number of log lines and the amount of log data that you can download at a time cannot exceed the upper limits. If the number of log lines and the amount of log data that you want to download exceed the upper limits, the excess logs cannot be downloaded. To download the full logs in this scenario, you can shorten your query time range and repeat the download operations until the full logs are downloaded.
You can view the download tasks in the Log Export History dialog box.
Simple Log Service allows for up to three concurrent download operations within each Alibaba Cloud account. Simple Log Service does not limit the total number of download operations. If more than three download operations are concurrently performed or multiple Resource Access Management (RAM) users are used to download logs at the same time, an error may be reported. In this case, you must wait until other download operations are complete and try again.
Simple Log Service stores only the export records that are generated within the previous 24 hours and deletes the export records 24 hours after the records are generated.
If a download task fails because network errors occur or query results are inaccurate, Simple Log Service automatically retries the task. If the download task fails after three retries, the download task enters the Failed state.

Log on to the Simple Log Service console.
In the Projects section, click the project that you want to manage.
On the Log Storage > Logstores tab, click the Logstore from which you want to download logs.
Enter a query statement and specify a query time range.
For more information, see Query and analyze logs.
On the Raw Logs tab, choose > Download Log.
Note
If no logs are available, Download Log is unavailable.

In the Log Download dialog box, configure the following parameters and click OK.

Parameter	Description
Task Name	The name of the download task.
Log Quantity	The number of logs that you want to download.
Data Format	The data format of the file to which you want to download logs. The CSV and JSON formats are supported. If you select CSV, the column names in the file are generated based on the fields of the first 100 logs. If the subsequent logs contain new fields, the new fields are stored in the last column of the file in the JSON format. The name of the last column is left empty. If you select JSON, each log is converted to the JSON format and written to the file as a single line.
Quote	The quote that is used to enclose special characters in logs. The enclosed special characters are not escaped.
Download Inaccurate Results	Specifies whether to download inaccurate results. If you select No, the download task fails when the query results are inaccurate.
Compression Method	The method that is used to compress logs. Valid values: gzip, lz4, zstd, and Uncompressed. If you want to download a large number of logs, we recommend that you select a compression method. This way, the log amount can be significantly reduced, and the logs can be quickly downloaded.
Sorting Rule	The rule that is used to sort logs.

Wait until the status of the download task in the Download Tasks dialog box changes to Successful. Then, click Download in the Actions column to download the logs to your on-premises machine.
You can also choose > Download Tasks on the Raw Logs tab to open the Download Tasks dialog box and view the download task.

Download data by using Cloud Shell

The Cloud Shell server resides in the China (Shanghai) region. If you download logs from a Logstore that does not reside in the China (Shanghai) region, you are charged for read traffic over the Internet when you download logs. For more information, see Simple Log Service Pricing and Use Cloud Shell to download logs.

Download data by using Simple Log Service CLI

If you want to download a large number of logs, you can use Simple Log Service CLI.

Install Simple Log Service CLI. For more information, see Install Simple Log Service CLI.
Obtain the AccessKey pair of the current account. For more information, see AccessKey pair.

Obtain the command that is used to download logs.

For example, run the download command in Simple Log Service CLI, and automatically download logs to downloaded_data.txt in the current directory.

aliyunlog log get_log_all --project="aliyun-test-project" --logstore="aliyun-test-logstore" --from_time="2024-07-01 15:33:00+8:00" --to_time="2024-07-09 15:23:00+8:00"  --query="status:200|select request_method as method,COUNT(*) as pv group by method order by pv" --region-endpoint="cn-hangzhou.log.aliyuncs.com" --format-output=json --access-id="LT***CyGg" --access-key="8P***zi" >> ./downloaded_data.txt

For more information, see Overview of Simple Log Service CLI.

Download data by using Simple Log Service SDK

If you want to download a large number of logs, you can use Simple Log Service SDK.

Note

You can call the same API operation that is encapsulated in the SDK to query and download logs.

The following sample code provides an example on how to download logs by using Simple Log Service SDK for Python:

import os
import time

from aliyun.log import LogClient
from aliyun.log import GetLogsRequest

# The Simple Log Service endpoint. 
endpoint = 'cn-qingdao.log.aliyuncs.com'
# In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
accessKeyId = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
accessKey = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')
# The name of the project. 
project = 'Project name'
# The name of the Logstore. 
logstore = 'Logstore name'
client = LogClient(endpoint, accessKeyId, accessKey)
request = GetLogsRequest("project1", "logstore1", fromTime=int(time()-3600), toTime=int(time()), topic='', query="*", line=100, offset=0, reverse=False)
# or
# request = GetLogsRequest("project1", "logstore1", fromTime="2018-1-1 10:10:10", toTime="2018-1-1 10:20:10", topic='', query="*", line=100, offset=0, reverse=False)


res = client.get_logs(request)
res.log_print()

For more information, see Overview of Simple Log Service SDK.