Configure application log collection by using pod environment variables - Container Compute Service

Container Compute Service (ACS) is integrated with Simple Log Service. When you create an ACS cluster, you can enable Simple Log Service. Simple Log Service can collect log data from containers in the ACS cluster. The log data includes the stout and text files. This topic describes how to collect application logs by using pod environment variables in an ACS cluster.

Step 1: Install the alibaba-log-controller component

When you create an ACS cluster, you can select Enable Log Service to install the alibaba-log-controller component. If this check box is not selected during creation, you can also perform the following steps to install the alibaba-log-controller component after the cluster is created.

Log on to the ACS console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its ID. In the left-side navigation pane of the cluster details page, choose Operations > Add-ons.
Find alibaba-log-controller in the Logs and Monitoring section. Click Install. In the Install alibaba-log-controller dialog box, click OK.

Important

If an earlier version of the alibaba-log-controller component is installed, you can update it. After the alibaba-log-controller component is updated, the component parameters are reset. The settings and environment variables of alibaba-log-controller will be overwritten. If you have customized the settings and environment variables, you need to reconfigure them.

Step 2: Configure Simple Log Service when you create an application

You can configure Simple Log Service to collect log data from containers when you create an application. To do this, you can use the ACS console or use a YAML file.

Use the wizard in the ACS console to create an application and configure Simple Log Service

Log on to the ACS console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its ID. In the left-side navigation pane of the cluster details page, choose Workloads > Deployments.
On the Deployments page, select a namespace from the Namespace drop-down list. Then, click Create from Image in the upper-right corner of the page.
Note
In this example, a stateless application is used. The configuration method is the same as other types of workloads.
On the Basic Information wizard page, specify Name, Replicas, and Type. Then, click Next to go to the Container wizard page.
Note
Only parameters related to Simple Log Service are described in the following section. For more information about other application parameters, see Create a stateless application by using a Deployment.
In the Log section, configure log collection parameters.
1. Configure Collection Configuration.
  Click the plus sign (+) to add a configuration entry. Each configuration entry consists of Logstore and Log Path in Container (can be set to stdout).
  - Logstore: Specify the name of the Logstore that is used to store the collected log data. If the Logstore does not exist, ACS automatically creates a Logstore in the Simple Log Service project that is associated with your ACS cluster.
    Note
    The default log retention period of Logstores is 180 days.
  - Log Path in Container (can be set to stdout): the path from which you want to collect log data. A value of /usr/local/tomcat/logs/catalina.*.log indicates that the log files of a Tomcat application are collected.
    Note
    When you set the value to stdout, stdout and stderr are collected.
    All settings are added as configuration entries to the corresponding Logstore. By default, logs are collected in simple mode (by row).
2. Set Custom Tag.
  Click the plus sign (+) to add custom tags. Each tag is a key-value pair that is appended to the collected log data. You can use custom tags to mark log data. For example, you can use a tag to denote the application version.
After you set other parameters, click Next to set advanced settings.
For more information about the subsequent steps, see Create a stateless application by using a Deployment.

Create an application by using a YAML template

Log on to the ACS console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, choose Workloads > Deployments.
On the Deployments page, select a namespace from the Namespace drop-down list. Then, click Create from YAML in the upper-right corner of the page.
Note
In this example, a stateless application is used. The configuration method is the same for other types of workloads.
Configure the YAML template.
YAML templates comply with the Kubernetes syntax. You can use env to define the Collection Configuration and Custom Tag parameters. The following code is an example:
```
apiVersion: v1
kind: Pod
metadata:
  name: my-demo
spec:
  containers:
  - name: my-demo-app
    image: 'registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest'
    env:   # Configure environment variables
    - name: aliyun_logs_log-stdout
      value: stdout
    - name: aliyun_logs_log-varlog
      value: /var/log/*.log
    - name: aliyun_logs_mytag1_tags
      value: tag1=v1
    command: ["sh", "-c"]
    args: ["echo 'Starting my demo app'; sleep 3600"]
```
Perform the following steps in sequence based on your requirements:
Note
If you have other log collection requirements, see Step 3: Advanced parameter settings for log collection.
1. Add log collection configurations and custom tags by using environment variables. All environment variables related to log collection must use aliyun_logs_ as the prefix.
  - Add log collection configurations in the following format:
```
apiVersion: v1
kind: Pod
metadata:
  name: my-demo
spec:
  containers:
  - name: my-demo-app
    image: 'registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest'
    env:
    - name: aliyun_logs_log-stdout
      value: stdout
    - name: aliyun_logs_log-varlog
      value: /var/log/*.log    
    command: ["sh", "-c"]
    args: ["echo 'Starting my demo app'; sleep 3600"]                    
```
    In the preceding example, two environment variables in the following format are added to the log collection configuration: aliyun_logs_{key}. The {keys} of the environment variables are log-stdout and log-varlog.
    - Environment variable aliyun_logs_log-stdout indicates that a Logstore named log-stdout is created to store the stdout collected from containers. The name of the collection configuration is log-stdout. This way, the stdout of containers is collected to the Logstore named log-stdout.
    - The aliyun_logs_log-varlog environment variable indicates that a Logstore named log-varlog is created to store the /var/log/*.log files collected from containers. The name of the collection configuration is log-varlog. This way, the /var/log/*.log files are collected to the Logstore named log-varlog.
  - Add custom tags in the following format:
```
apiVersion: v1
kind: Pod
metadata:
  name: my-demo
spec:
  containers:
  - name: my-demo-app
    image: 'registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest'
    env:
    - name: aliyun_logs_mytag1_tags
      value: tag1=v1           
    command: ["sh", "-c"]
    args: ["echo 'Starting my demo app'; sleep 3600"]            
```
    After a tag is added, the tag is automatically appended to the log data that is collected from the container. mytag1 specifies the tag name without underscores (_).
2. If you specify a log path to collect log files other than stdout, you must set the volumeMounts parameter.
  In the preceding YAML template, the mountPath field in volumeMounts is set to /var/log. This allows Logtail to collect log data from the /var/log/*.log file.
After you modify the YAML template, click Create to submit the configurations to the ACS cluster.

Step 3: Advanced parameters for log collection

You can configure container environment variables to customize log collection. You can use environment variables to configure advanced settings to meet your log collection requirement.

Environment Variable Name	Description	Example	Note
aliyun_logs_{key}	Required. {key} can contain only lowercase letters, digits, and hyphens (-). If the specified aliyun_logs_{key}_logstore does not exist, a Logstore named {key} is created. To collect the stdout of a container, set the value to stdout. You can also set the value to a path inside the container to collect the log files.	`- name: aliyun_logs_catalina value: stdout` `- name: aliyun_logs_access-log value: /var/log/nginx/access.log`	The default log collection mode is simple mode. If you want to parse log data, we recommend that you use the Simple Log Service console. {key} specifies the name of the log collection configuration. The configuration name must be unique in the ACS cluster.
aliyun_logs_{key}_tags	Optional. This variable is used to add tags to log data. The value must be in the following format: {tag-key}={tag-value}.	`- name: aliyun_logs_catalina_tags value: app=catalina`	N/A
aliyun_logs_{key}_project	Optional. The variable specifies a project in Simple Log Service. By default, the project that you specified when you created the cluster is used.	`- name: aliyun_logs_catalina_project value: my-k8s-project`	The project must be deployed in the same region as the alibaba-log-controller component.
aliyun_logs_{key}_logstore	Optional. The variable specifies a Logstore in Simple Log Service. By default, the Logstore is named {key}.	`- name: aliyun_logs_catalina_logstore value: my-logstore`	N/A
aliyun_logs_{key}_shard	Optional. The variable specifies the number of shards of the Logstore. Valid values: 1 to 10. Default value: 2. Note If the Logstore that you specify already exists, this variable does not take effect.	`- name: aliyun_logs_catalina_shard value: 4`	N/A
aliyun_logs_{key}_ttl	Optional. The variable specifies the log retention period. Valid values: 1 to 3650. To retain log data permanently, set the value to 3650. The default retention period is 90 days. Note If the Logstore that you specify already exists, this variable does not take effect.	`- name: aliyun_logs_catalina_ttl value: 3650`	N/A
aliyun_logs_{key}_machinegroup	Optional. This variable specifies the node group in which the application is deployed. The default node group is the one in which the alibaba-log-controller component is deployed. For more information about how to use the variable, see Special Scenario 2: Collect data from different applications into different projects.	`- name: aliyun_logs_catalina_machinegroup value: my-machine-group`	N/A
aliyun_logs_{key}_logstoremode	Optional. This variable specifies the type of Logstore. Default value: standard. Valid values: Note If the Logstore that you specify already exists, this variable does not take effect. standard: standard Logstore. This type of Logstore supports the log analysis feature and is suitable for scenarios such as real-time monitoring and interactive analysis. You can also use this type of Logstore to build a comprehensive observability system. query: query Logstore. This type of Logstore supports high-performance queries. The index traffic fee of a query Logstore is approximately half that of a standard Logstore. Query Logstores do not support SQL analysis. Query Logstores are suitable for scenarios in which the amount of data is large, the log retention period is long, or log analysis is not required. If logs are stored for weeks or months, the log retention period is considered long.	`- name: aliyun_logs_catalina_logstoremode value: standard` `- name: aliyun_logs_catalina_logstoremode value: query`	N/A

Special Scenario 1: Collect data from multiple applications to the same Logstore

In this scenario, set the aliyun_logs_{key}_logstore variable. The following example shows how to collect stdout from two applications and store the output in stdout-logstore.

The {key} of Application 1 is set to app1-stdout. The {key} of Application 2 is set to app2-stdout.

Configure the following environment variables for Application 1:

apiVersion: v1
kind: Pod
metadata:
  name: my-demo
spec:
  containers:
  - name: my-demo-app
    image: 'registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest'
    env:    # Configure environment variables
    - name: aliyun_logs_app1-stdout
      value: stdout
    - name: aliyun_logs_app1-stdout_logstore
      value: stdout-logstore
    command: ["sh", "-c"]
    args: ["echo 'Starting my demo app'; sleep 3600"]

Configure the following environment variables for Application 2:

apiVersion: v1
kind: Pod
metadata:
  name: my-demo
spec:
  containers:
  - name: my-demo-app
    image: 'registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest'
    env: # Configure environment variables
    - name: aliyun_logs_app2-stdout
      value: stdout
    - name: aliyun_logs_app2-stdout_logstore
      value: stdout-logstore
    command: ["sh", "-c"]
    args: ["echo 'Starting my demo app'; sleep 3600"]

Special Scenario 2: Collect data from different applications to different projects

In this scenario, perform the following steps:

Create a machine group in each project and set the custom identifier of the machine group in the following format: k8s-group-{cluster-id}, where {cluster-id} is the ID of the cluster. You can specify a custom machine group name.

Specify the project, Logstore, and the machine group in the environment variables for each application. The name of the machine group is the same as that of the machine group that you created in the previous step.

In the following example, the {key} of Application 1 is set to app1-stdout. The {key} of Application 2 is set to app2-stdout. If the two applications are deployed in the same ACS cluster, you can use the same machine group for the applications.

Configure the following environment variables for Application 1:

apiVersion: v1
kind: Pod
metadata:
  name: my-demo
spec:
  containers:
  - name: my-demo-app
    image: 'registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest'
    env: # Configure environment variables
    - name: aliyun_logs_app1-stdout
      value: stdout
    - name: aliyun_logs_app1-stdout_project
      value: app1-project
    - name: aliyun_logs_app1-stdout_logstore
      value: app1-logstore
    - name: aliyun_logs_app1-stdout_machinegroup
      value: app1-machine-group
    command: ["sh", "-c"]
    args: ["echo 'Starting my demo app'; sleep 3600"]

Configure the following environment variables for Application 2:

apiVersion: v1
kind: Pod
metadata:
  name: my-demo
spec:
  containers:
  - name: my-demo-app
    image: 'registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest'
    env: # Configure environment variables for Application 2
    - name: aliyun_logs_app2-stdout
      value: stdout
    - name: aliyun_logs_app2-stdout_project
      value: app2-project
    - name: aliyun_logs_app2-stdout_logstore
      value: app2-logstore
    - name: aliyun_logs_app2-stdout_machinegroup
      value: app1-machine-group
    command: ["sh", "-c"]
    args: ["echo 'Starting my demo app'; sleep 3600"]

Step 4: View logs by using the Simple Log Service console

This example shows how to view the logs of applications created in the console. After configuration, application logs are collected and stored in Simple Log Service. You can view the container logs in the Simple Log Service console.

Log on to the Simple Log Service console.
In the Projects section, select the project corresponding to the ACS cluster (default: k8s-log-{ACS cluster ID}), and click and go to the Logstore tab.
In the Logstore list, find the Logstore that is specified when you configure log collection. Move the pointer over the Logstore name and click the icon. Then, click Search & Analysis.
In this example, you can view the standard output logs and text logs inside the container of the application on the log query page, and you can find custom tags appended to the log fields.

References

You can view the logs collected from containers in ACS clusters in the Simple Log Service console.
For more information about how to troubleshoot log collection errors, see What do I do if errors occur when I use Logtail to collect logs?