Use AliyunPipelineConfig to manage a Logtail configuration - Simple Log Service

Simple Log Service defines a CustomResourceDefinition (CRD) named AliyunPipelineConfig. You can configure a YAML file to create a CustomResource (CR) from the CRD to manage a Logtail configuration. This topic describes how to use AliyunPipelineConfig to manage a Logtail configuration.

Prerequisites

The Logtail components V0.5.1 or later are installed. For more information, see Install Logtail components in an ACK cluster and Install Logtail components in a self-managed Kubernetes cluster.
The container from which you want to collect logs continuously generates logs.
Important
Logtail collects only incremental logs. If a log file on a server is not updated after the applied Logtail configuration is delivered to the server, Logtail does not collect logs from the file. For more information, see Read log files.

Procedure

Important

You can create a CR from the AliyunPipelineConfig CRD to create a Logtail configuration. The alibaba-log-controller component periodically synchronizes this type of Logtail configuration to Simple Log Service. If you want to modify this type of Logtail configuration, you must modify the related CR. The Logtail configuration created from the Simple Log Service console will be overwritten by the CR. We recommend that you always use CRs to manage Logtail configurations.

Create a Logtail configuration

You must create a CR from the AliyunPipelineConfig CRD. Then, the alibaba-log-controller component automatically creates a Logtail configuration based on the CR. After the Logtail configuration is created, it is automatically applied, and Logtail starts to collect logs to Simple Log Service based on the Logtail configuration.

Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.
Run the following command to create a YAML file named cube.yaml. You can specify a different file name based on your business requirements.

vim cube.yaml

Enter the following script in the YAML file and configure the parameters based on your business requirements. For more information about CR parameters, see CR parameters.

apiVersion: telemetry.alibabacloud.com/v1alpha1 # The default value is used. You do not need to modify this parameter. 
kind: ClusterAliyunPipelineConfig               # The default value is used. You do not need to modify this parameter. 
metadata:
  name: test-config                             # The name of the resource. The name must be unique in the current Kubernetes cluster. 
spec:
  project:                                      # The name of the Logstore. 
    name: k8s-your-project                      
  config:                                       # The Logtail configuration. 
    inputs:                                     # The input plug-ins of the Logtail configuration.
      ...
    flushers:                                   # The output plug-ins of the Logtail configuration.
      ...

Run the following command to create a CR from the AliyunPipelineConfig CRD. The alibaba-log-controller component creates a Logtail configuration based on the CR. Then, the Logtail configuration is automatically applied. After the Logtail configuration is applied, Logtail starts to collect logs to Simple Log Service. In the following command, cube.yaml is a sample file name. You can specify a different file name based on your business requirements.

kubectl apply -f cube.yaml

View a Logtail configuration

View all Logtail configurations that are created in the current Kubernetes cluster based on the CRs of the AliyunPipelineConfig CRD.
You can run the following command to view the Logtail configurations:
```
kubectl get clusteraliyunpipelineconfigs
```

View the detailed settings and status of a Logtail configuration that is created based on a CR of the AliyunPipelineConfig CRD.

You can run the following command to view the Logtail configuration. In this command, <config_name> is the name of the required CR. You can specify a different name based on your business requirements.

kubectl get clusteraliyunpipelineconfigs <config_name> -o yaml

Information that is similar to the following code is returned. You can check whether the Logtail configuration is applied based on the status parameter.

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  creationTimestamp: null
  finalizers:
    - finalizer.pipeline.alibabacloud.com
  name: example-config
# The expected configuration.
spec:
  config:
    flushers:
      - Endpoint: cn-hangzhou-intranet.log.aliyuncs.com
        Logstore: example-config-logstore
        Region: cn-hangzhou
        TelemetryType: logs
        Type: flusher_sls
    inputs:
      - FilePaths:
          - /var/log/*.log
        Type: input_file
    processors:
      - SourceKey: content
        Type: processor_parse_json_native
  logstores:
    - name: example-config-logstore
  project:
    name: k8s-log-clusterid
# The application status.
status:
  # Whether the Logtail configuration is applied.
  success: true
  # The status message.
  message: success
  # The update time of the current status.
  lastUpdateTime: "2024-06-13T12:02:19.133422+08:00"
  # The Logtail configuration that was previously applied. Default values are used in the Logtail configuration.
  lastAppliedConfig:
    # The point in time at which the Logtail configuration was previously applied.
    appliedTime: "2024-06-13T12:02:19.133422+08:00"
    # The detailed settings of the Logtail configuration that was previously applied.
    config:
      configTags:
        sls.crd.cluster: clusterid
        sls.crd.kind: ClusterAliyunPipelineConfig
        sls.logtail.channel: CRD
      flushers:
        - Endpoint: cn-hangzhou-intranet.log.aliyuncs.com
          Logstore: example-config-logstore
          Region: cn-hangzhou
          TelemetryType: logs
          Type: flusher_sls
      global: null
      inputs:
        - FilePaths:
            - /var/log/*.log
          Type: input_file
      name: example-config
      processors:
        - SourceKey: content
          Type: processor_parse_json_native
    logstores:
      - appendMeta: true
        autoSplit: true
        encryptConf: {}
        maxSplitShard: 64
        name: example-config-logstore
        shardCount: 2
        ttl: 30
    machineGroups:
      - name: k8s-group-clusterid
    project:
      description: k8s log project, created by alibaba cloud log controller
      endpoint: cn-hangzhou.log.aliyuncs.com
      name: k8s-log-clusterid

Update a Logtail configuration

To update a Logtail configuration, you need to only modify the YAML file that is used to create the related CR and apply the new file.

Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.
Modify the YAML file that is used to create your CR. In the following command, cube.yaml is a sample file name. You can specify a different file name based on your business requirements.

vim cube.yaml

Modify parameters in the YAML file.
Run the following command to apply the new file. In the following command, cube.yaml is a sample file name. You can specify a different file name based on your business requirements.

kubectl apply -f cube.yaml

Delete a Logtail configuration

To delete a Logtail configuration, you need to only delete the CR based on which the Logtail configuration is created. In the following command, <config_name> is the name of the required CR. You can specify a different name based on your business requirements.

kubectl delete clusteraliyunpipelineconfigs <config_name>

Important

After a Logtail configuration is deleted, the related CR is released.
If a Logtail configuration fails to be deleted, the related CR is not released. You can view the error message in the status parameter of the CR. If you want to ignore the error message and directly release the CR, you can manually delete the metadata.finalizers field from the CR.

CR parameters

apiVersion
The value is fixed as telemetry.alibabacloud.com/v1alpha1.
kind
The value is fixed as ClusterAliyunPipelineConfig.

metadata

Field

Type

Required

Description

name

string

Yes

The unique identifier of the CR. The value is the same as the name of the Logtail configuration.

Note:

The value of this field is used as the name of the Logtail configuration. The name must be unique in the project to which the Logtail configuration belongs. After the Logtail configuration is created, you cannot change the name of the configuration. The value must meet the following requirements:

The value can contain only lowercase letters, digits, hyphens (-), and underscores (_).

The value must start and end with a lowercase letter or a digit.

The value must be 2 to 128 characters in length.

Spec

Field	Type	Required	Description
project	PipelineProject	Yes	The information about the project. Note After a CR is created, you cannot change the value of the project field. To use another project, you must create a different CR.
config	PiplineConfig	Yes	The detailed settings of the Logtail configuration.
logstores	PipelineLogstore	No	The Logstores. You can create multiple Logstores. Note All fields, except the metadata.name field, take effect only when you create a Logstore. The value of this field does not specify the Logstore to which logs are collected based on the Logtail configuration. The value specifies the Logstores that are created. The value of the config.flushers field specifies the Logstore to which logs are collected based on the Logtail configuration. If the specified Logstore exists, you can leave this field empty. You can configure this field only to add or delete Logstores. You cannot modify existing Logstores.
machineGroups	PipelineMachineGroup	No	The machine groups that are associated with the Logtail configuration. When you install the `logtail-ds` component, Simple Log Service automatically creates a machine group named `k8s-group-${your_k8s_cluster_id}`. By default, the k8s-group-${your_k8s_cluster_id} machine group is used. Note The alibaba-log-controller component ensures that the Logtail configuration is associated with only the machine group specified by the machineGroups field. Other machine groups are disassociated from the Logtail configuration.
enableUpgradeOverride	bool	No	Default value: false. A CR of the AliyunLogConfig CRD and a CR of the AliyunPipelineConfig CRD can point to the same Logtail configuration in the same cluster. This field is designed for this scenario. Valid values: true: The alibaba-log-controller component overwrites and updates the Logtail configuration created based on the CR of the AliyunLogConfig CRD. false: A Logtail configuration conflict occurs, and the Logtail configuration created based on the CR of the AliyunPipelineConfig CRD fails to be applied. Note If the following conditions are met, the system considers that the CRs point to the same Logtail configuration: The same project to which logs are collected is used. For the AliyunLogConfig CRD, the project in the CR is specified by the spec.project field. For the AliyunPipelineConfig CRD, the project in the CR is specified by the spec.project.name field. The same Logtail configuration name is used. For the AliyunLogConfig CRD, the name in the CR is specified by the spec.logtailConfig.configName field. For the AliyunPipelineConfig CRD, the name in the CR is specified by the metadata.name field. The component overwrites and updates the Logtail configuration in the following process: The component attempts to apply the Logtail configuration created based on the CR of the AliyunPipelineConfig CRD. After the Logtail configuration is applied, the component deletes the CR of the AliyunLogConfig CRD from the cluster. The process is complete.

PipelineProject

Note

After a CR is created, you cannot change the value of the project field. To use another project, you must create a different CR.

Field	Type	Required	Description
name	string	Yes	The name of the project. Note If the specified project does not exist, Simple Log Service automatically creates a project.
description	string	No	The description of the project. Note This field takes effect only when you create a project.
endpoint	string	No	The endpoint of the project. For more information, see Endpoints. By default, the endpoint for the region where the current cluster resides is used. If the project and the cluster reside in different regions, you must configure this field. Note This field specifies the endpoint for the region where the Logtail configuration resides. To collect logs to the project based on the endpoint by using Logtail, you must also configure the config_server_address and data_server_list fields for your Logtail. For more information, see Logtail configuration files and record files.
uid	string	No	The ID of the account. By default, the account to which the current cluster belongs is used. If you want to collect logs across accounts, you must configure this field and include `ALICLOUD_LOG_ACCOUNT_INFOS={"<uid>":{"accessKeyID":"<your_access_key_id>","accessKeySecret":"<your_access_key_secret>"}}` in the environment variable settings for the alibaba-log-controller component. Note This field specifies the account to which the project belongs. To collect logs to the account by using Logtail, you must configure a user identifier for your Logtail. For more information, see Configure a user identifier.

PiplineConfig

You must configure the following fields based on the instructions provided in CreateLogtailPipelineConfig. The configName field uses the value of the metadata.name field. You cannot separately configure the configName field.

Field	Type	Required	Description
sample	string	No	The sample log. You can specify multiple sample logs. Make sure that the value does not exceed 1,500 bytes.
global	object	No	The global settings. For more information, see Request parameters.
inputs	object list	Yes	The input plug-ins. For more information about the details and parameters of the plug-ins, see Request parameters. Note You can configure only one input plug-in.
processors	object list	No	The processing plug-ins. For more information about the details and parameters of the plug-ins, see Request parameters.
aggregators	object list	No	The aggregation plug-ins. For more information about the details and parameters of the plug-ins, see Request parameters.
flushers	object list	Yes	The output plug-ins. For more information about the details and parameters of the plug-ins, see Request parameters. Note You can configure only one output plug-in.
configTags	map	No	The custom tags that are added to identify the Logtail configuration.

PipelineLogstore

Note

All fields, except the metadata.name field, take effect only when you create a Logstore.

Field	Type	Required	Description
name	string	Yes	The name of the Logstore.
queryMode	string	No	The type of the Logstore. For more information, see Logstore types. Default value: standard. Valid values: query: Query Logstore standard: Standard Logstore
ttl	int	No	The retention period of logs in the Logstore. Unit: days. For more information, see Data Retention Period. Default value: 30. Valid values: 1 to 3650. If you set this field to 3650, logs are permanently stored.
hotTtl	int	No	The retention period of logs in the hot storage tier of the Logstore. Unit: days. For more information, see Intelligent Tiered Storage. Default value: 0. The value of this field must be less than the value of the ttl field and greater than or equal to 7.
shardCount	int	No	The number of shards in the Logstore. For more information, see Manage a logstore. Default value: 2. Valid values: 1 to 100.
maxSplitShard	int	No	The maximum number of shards into which existing shards in the Logstore can be automatically split. For more information, see Automatic Sharding. Default value: 64. Valid values: 1 to 256.
autoSplit	bool	No	Specifies whether to enable automatic sharding for the Logstore. For more information, see Automatic Sharding. Default value: true.
telemetryType	string	No	The type of observable data in the Logstore. For more information, see telemetryType. Default value: None. Valid values: None: log data Metrics: metric data
appendMeta	bool	No	Specifies whether to record public IP addresses for the Logstore. For more information, see Log Public IP. Default value: true.
enableTracking	bool	No	Specifies whether to enable the web tracking feature for the Logstore. For more information, see WebTracking. Default value: false.
encryptConf	object	No	The encryption settings of the Logstore. For more information, see Common data structures. This field is empty by default.
meteringMode	string	No	The billing mode of the Logstore. For more information, see Billing Mode. This field is empty by default. Valid values: ChargeByFunction: pay-by-feature billing mode ChargeByDataIngest: pay-by-ingested-data billing mode Note If you set the queryMode field to query, you must set this field to ChargeByFunction. If the pay-by-ingested-data billing mode is disabled for your account, you can set this field only to ChargeByFunction.

PipelineMachineGroup

Field	Type	Required	Description
name	string	Yes	The name of the machine group that is associated with the Logtail configuration. Note If no machine group exists, the alibaba-log-controller component automatically creates an identifier-based machine group. The machine group is named after the identifier. For more information, see Create a custom identifier-based machine group. This operation takes effect only when you create a machine group.