Collect SAE file logs to Simple Log Service for management and analysis - Serverless App Engine

Serverless App Engine (SAE) provides the basic real-time log collection feature, which allows you to view up to 500 lines of logs. SAE also provides the file log collection feature to meet higher query requirements. You can specify stdout.log or the log path in a container as the log source. Then, SAE collects standard output (stdout) logs or business file logs from the container to Simple Log Service. You can collect, aggregate, and analyze unlimited lines of logs. After you enable log collection to Simple Log Service for an application, SAE automatically connects to Simple Log Service when you deploy or scale out the application. Then, you can query logs by using keywords in the Simple Log Service console.

Prerequisites

Simple Log Service is activated, and a project and a Logstore are created. For more information, go to the product page of Simple Log Service.
At least 0.25 CPU core and 250 MB of memory are reserved for each instance in your application.

Usage notes

You can create up to 50 projects for an Alibaba Cloud account. You can create up to 200 Logstores for a project.
Warning
The limits on the basic resources of Simple Log Service may affect the log collection results of SAE applications. For example, the following issues may occur: The log retention period becomes shorter than expected and a log collection failure occurs. For more information about the limits, see Basic resources.
If you configure log collection to Simple Log Service when you create an application, the system automatically checks whether Simple Log Service is activated and whether the built-in resources of Simple Log Service are sufficient after the application is created.
- If Simple Log Service is not activated, activate the service as prompted.
- If the built-in resources are insufficient, join the DingTalk group 32874633 to apply to increase quotas.
You are charged additional fees when you use Simple Log Service. You are charged for the amount of resources that you consume when you use the file log collection feature. For more information, see Simple Log Service billing.

Configure file log collection

Configure log collection when you create an application

Log on to the SAE console.
In the left-side navigation pane, click Applications. In the top navigation bar, select a region. Then, click Create Application.
In the Basic Information step, configure the parameters and click Next: Application Deployment Configurations.
In the Deployment Configurations step, configure the Technology Stack Programming Language and Application Deployment Method parameters and the corresponding settings.

In the Log Collection Service section, turn on Enable Log Collection to Log Service on the Log Collection to Log Service tab, and then configure the parameters.

Select Create Log Service Resources or Use Existing Log Service Resources based on your business requirements, click + Add, and then configure the parameters. The following table describes the parameters.

Parameter	Description
The following parameters are available when you select Create Log Service Resources or Use Existing Log Service Resources:
Log Type	Select a log type. Valid values: File Logs (Log Path in Container): the file logs. This is the default value. You can add multiple collection rules to collect file logs from different log sources. Container Standard Output Logs: the container stdout logs. You can configure only one collection rule to collect stdout logs. This log type is available in the Log Type drop-down list only when your vSwitch resides in one of the recommended zones. For more information, see Change the security group and vSwitch of an application.
Log Source	Enter the directory in which the log file is stored. The directory must contain the name of the log file. Example: /tmp0/cjsc.log. If you select Container Standard Output Logs, you do not need to configure the Log Source parameter. You can specify a regular expression to match the file name and path. If a directory contains a large number of log files that have the same format, you can specify a log source in the /xxx/xxx/xxx/.log format. Important* Do not save important files of other types in the log source. Otherwise, the files are overwritten by log files.
The following parameters are available only when you select Use Existing Log Service Resources:
Use Existing Log Service Project	Select a project in which you want to store logs.
logstore	Select a Logstore.
logtail	Select a Logtail configuration. For more information, see Logtail.

Click Next: Confirm Specifications.
In the Specification Confirmation step, view the details of the application and the fee for the selected specifications. Then, click Confirm.
The Creation Completed step appears. You can click Application Details to go to the Basic Information page of the application.
Verify the result.
After you deploy an application, SAE collects logs to the specified Logstore based on the configured log collection rule.
In the left-side navigation pane of the application details page, choose Log Management > Persistent Logs. On the Persistent Logs page, view the information about the configured log collection rule.
If the log collection rule that you configured is displayed, it has taken effect. You can analyze the logs based on your business requirements.

Configure log collection when you deploy an application

Warning

After you redeploy an application, the application is restarted. To prevent unpredictable errors such as business interruptions, we recommend that you deploy applications during off-peak hours.

The procedure that can be performed to update an application varies based on the number of instances in the application. This section provides an example on how to configure the required features for an application in which the number of instances is greater than or equal to 1. For information about how to update an application in which the number of instances is 0, see Update an application.

Log on to the SAE console.
In the left-side navigation pane, click Applications. In the top navigation bar, select a region. Then, click the name of an application.
In the upper-right corner of the Basic Information page, click Deploy Application.

In the Log Collection Service section of the Deploy Application page, turn on Enable Log Collection to Log Service on the Log Collection to Log Service tab, and then configure the parameters.

Parameter	Description
The following parameters are available when you select Create Log Service Resources or Use Existing Log Service Resources:
Log Type	Select a log type. Valid values: File Logs (Log Path in Container): the file logs. This is the default value. You can add multiple collection rules to collect file logs from different log sources. Container Standard Output Logs: the container stdout logs. You can configure only one collection rule to collect stdout logs. This log type is available in the Log Type drop-down list only when your vSwitch resides in one of the recommended zones. For more information, see Change the security group and vSwitch of an application.
Log Source	Enter the directory in which the log file is stored. The directory must contain the name of the log file. Example: /tmp0/cjsc.log. If you select Container Standard Output Logs, you do not need to configure the Log Source parameter. You can specify a regular expression to match the file name and path. If a directory contains a large number of log files that have the same format, you can specify a log source in the /xxx/xxx/xxx/.log format. Important* Do not save important files of other types in the log source. Otherwise, the files are overwritten by log files.
The following parameters are available only when you select Use Existing Log Service Resources:
Use Existing Log Service Project	Select a project in which you want to store logs.
logstore	Select a Logstore.
logtail	Select a Logtail configuration. For more information, see Logtail.

After you configure the settings, click Confirm.
Note
If specific conditions are met when you deploy an application, logs are still collected to the original project and Logstore before all batches of application instances are deployed. The conditions include that you use a phased release policy or a canary release policy to deploy an application, the log source that you configured during application creation remains unchanged, and you change only the project or Logstore to which logs are collected. In Simple Log Service, you cannot collect logs from the same file to different projects or Logstores at the same time.
Verify the result.
After you deploy an application, SAE collects logs to the specified Logstore based on the configured log collection rule.
In the left-side navigation pane of the application details page, choose Log Management > Persistent Logs. On the Persistent Logs page, view the information about the configured log collection rule.
If the log collection rule that you configured is displayed, it has taken effect. You can analyze the logs based on your business requirements.

Configure environment variables to improve the collection performance of Logtail

You can configure the startup parameters of Logtail by configuring environment variables in the SAE console. After you configure the environment variables and enable the log collection feature, the collection performance of Logtail is improved. This helps you effectively eliminate log collection bottlenecks, such as the large memory usage of log files, large volume of log data traffic, and high data sending rate of Logtail.

In the Environment Variable Settings section, select Custom or Reference ConfigMap, and configure the Variable Name and Variable Value/Variable Reference parameters. For more information, see Configure environment variables.

Variable setting rules
Add the sls_ prefix before the Logtail environment variable that you want to configure. The following figure shows a sample configuration. For example, if you want to add the max_read_buffer_size parameter of Logtail, set the Variable Name parameter to sls_max_read_buffer_size. The configurations displayed in the figure indicate that the maximum size of a log that Logtail can read is 524,288 bytes. For more information, see Configure the startup parameters of Logtail.
Mappings between environment variables and the startup parameters of Logtail
For more information, see Environment variables.

Remove prefixes from stdout logs in Simple Log Service

In Simple Log Service, if you want to configure additional custom settings to extract fields from original logs, such as removing prefixes from stdout logs, you can select Full Regex Mode in the corresponding Logtail configuration. The following figure shows the configurations of the required parameters.

sc_sls_collect_logs_in_full_regex_mode

Regex to Match First Line
```
\d+-\d+-\d+T\d+:\d+:\d+.\d+\+\d+.*
```

RegEx

\d+-\d+-\d+T\d+:\d+:\d+.\d+\+\d+:\d+\s\w+\s\w\s(.*)

If you want to expand JSON fields, you can turn on Enable Plug-in Processing and configure the Plug-in Config parameter.

sc_sls_plugin_configuration

{
  "global": {
    "DefaultLogQueueSize": 10
  },
  "processors": [
    {
      "type": "processor_split_log_regex",
      "detail": {
        "PreserveOthers": true,
        "SplitKey": "content",
        "SplitRegex": "\\d+-\\d+-\\d+T\\d+:\\d+:\\d+.\\d+\\+\\d+.*"
      }
    },
    {
      "type": "processor_regex",
      "detail": {
        "SourceKey": "content",
        "Regex": "\\d+-\\d+-\\d+T\\d+:\\d+:\\d+.\\d+\\+\\d+:\\d+\\s\\w+\\s\\w\\s(.*)",
        "Keys": [
          "newline"
        ]
      }
    },
    {
      "detail": {
        "SourceKey": "newline",
        "KeepSource": true,
        "ExpandDepth": 0,
        "NoKeyError": true,
        "ExpandConnector": "-"
      },
      "type": "processor_json"
    }
  ]
}

For more information, see Collect logs in full regex mode and Manage Logtail configurations for log collection.

FAQ

Does the logging feature of SAE support log rotation?
Yes, the logging feature of SAE supports log rotation. You can rotate logs by adding wildcard characters to the name of a log file.
You can use only the asterisk (*) and question mark (?) wildcard characters.
What do I do if no data is displayed in the Simple Log Service console when I query file logs?
By default, Simple Log Service queries the data of the previous 15 minutes. If no data is displayed in the Simple Log Service console, we recommend that you use a webshell to check whether logs are generated for your application. For more information, see Use the webshell feature to check the health status of applications.
- If logs are generated for your application, join the DingTalk group 32874633 for technical support.
- If no logs are generated, check your application.