Use AliyunPipelineConfig to manage a Logtail configuration - Simple Log Service

AliyunPipelineConfig is a CustomResourceDefinition (CRD) provided by Simple Log Service. Create a CustomResource (CR) from a YAML file to manage a Logtail collection configuration in your Kubernetes cluster.

Prerequisites

Logtail components of version 0.5.1 or later are installed. For more information, see Install Logtail components in an ACK cluster and Install Logtail components in a self-managed Kubernetes cluster.
The target container continuously generates logs.
Important
Logtail collects only incremental logs. If a log file is not updated after the Logtail configuration is delivered, Logtail does not collect logs from that file. For more information, see Read logs.

Procedure

Important

The alibaba-log-controller component periodically synchronizes Logtail collection configurations that are defined by AliyunPipelineConfig to the server. To modify a Logtail collection configuration that is created from a CR, you must update the corresponding CR. Logtail collection configurations that are created in the Simple Log Service console are overwritten by CRs. Therefore, you must use only one method to manage Logtail collection configurations.

Create an iLogtail collection configuration

Create a CR of AliyunPipelineConfig. The alibaba-log-controller component then automatically creates a Logtail collection configuration based on the CR. After the configuration is created, the system automatically applies it.

Procedure:

Obtain the cluster kubeconfig and connect to the cluster using kubectl.
Run the following command to create a cube.yaml file. You can replace the file name as required.

vim cube.yaml

Add the following content to the YAML file and configure the parameters as required. For more information about the parameters in the CR structure, see CR parameters.

apiVersion: telemetry.alibabacloud.com/v1alpha1 # Fixed value. Do not modify.
kind: ClusterAliyunPipelineConfig               # Fixed value. Do not modify.
metadata:
  name: test-config                             # Specify a resource name. The name must be unique within the current Kubernetes cluster.
spec:
  project:                                      # Set the destination project name.
    name: k8s-your-project                      
  config:                                       # Define the iLogtail collection configuration.
    inputs:                                     # Specify input plug-ins.
      ...
    flushers:                                   # Specify output plug-ins.
      ...

Run the following command to create a CR of AliyunPipelineConfig. The alibaba-log-controller component automatically creates a Logtail collection configuration based on the CR, and the configuration then takes effect. After the Logtail collection configuration takes effect, Logtail starts to collect data and sends the data to Simple Log Service. Replace cube.yaml with your actual file name.

kubectl apply -f cube.yaml

View an iLogtail collection configuration

View all Logtail collection configurations that are created using AliyunPipelineConfig in the current Kubernetes cluster.
Run the following command to view all Logtail collection configurations created by AliyunPipelineConfig.
```
kubectl get clusteraliyunpipelineconfigs
```

View the details and status of a Logtail collection configuration that is created using AliyunPipelineConfig.

Run the following command to view the configuration. Replace <config_name> with the name of the AliyunPipelineConfig.

kubectl get clusteraliyunpipelineconfigs <config_name> -o yaml

The output is similar to the following example. Check the status field to verify whether the Logtail collection configuration is applied.

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  creationTimestamp: null
  finalizers:
    - finalizer.pipeline.alibabacloud.com
  name: example-config
# Expected configuration
spec:
  config:
    flushers:
      - Endpoint: cn-hangzhou-intranet.log.aliyuncs.com
        Logstore: example-config-logstore
        Region: cn-hangzhou
        TelemetryType: logs
        Type: flusher_sls
    inputs:
      - FilePaths:
          - /var/log/*.log
        Type: input_file
    processors:
      - SourceKey: content
        Type: processor_parse_json_native
  logstores:
    - name: example-config-logstore
  project:
    name: k8s-log-clusterid
# Application status
status:
  # Whether the CR is successfully applied
  success: true
  # Current status message of the CR
  message: success
  # Update time of the current status
  lastUpdateTime: "2024-06-13T12:02:19.133422+08:00"
  # Configuration information from the last successful application. This configuration includes filled default values and is the one currently in effect.
  lastAppliedConfig:
    # Time of the last successful application
    appliedTime: "2024-06-13T12:02:19.133422+08:00"
    # Details of the last successfully applied configuration
    config:
      configTags:
        sls.crd.cluster: clusterid
        sls.crd.kind: ClusterAliyunPipelineConfig
        sls.logtail.channel: CRD
      flushers:
        - Endpoint: cn-hangzhou-intranet.log.aliyuncs.com
          Logstore: example-config-logstore
          Region: cn-hangzhou
          TelemetryType: logs
          Type: flusher_sls
      global: null
      inputs:
        - FilePaths:
            - /var/log/*.log
          Type: input_file
      name: example-config
      processors:
        - SourceKey: content
          Type: processor_parse_json_native
    logstores:
      - appendMeta: true
        autoSplit: true
        encryptConf: {}
        maxSplitShard: 64
        name: example-config-logstore
        shardCount: 2
        ttl: 30
    machineGroups:
      - name: k8s-group-clusterid
    project:
      description: k8s log project, created by alibaba cloud log controller
      endpoint: cn-hangzhou.log.aliyuncs.com
      name: k8s-log-clusterid

Update an iLogtail collection configuration

To update a Logtail collection configuration, you can modify the existing AliyunPipelineConfig YAML file and reapply it.

Obtain the cluster kubeconfig and connect to the cluster using kubectl.
Run the following command to edit the existing AliyunPipelineConfig file. Replace cube.yaml with your actual file name.

vim cube.yaml

Modify the configuration parameters in the YAML file.
Run the following command to apply the updated Logtail collection configuration. Replace cube.yaml with your actual file name.

kubectl apply -f cube.yaml

Delete an iLogtail collection configuration

To delete a Logtail collection configuration, you must delete its corresponding CR. Replace <config_name> with the name of the AliyunPipelineConfig.

kubectl delete clusteraliyunpipelineconfigs <config_name>

Important

After the Logtail collection configuration is deleted, the CR is released.
If the Logtail collection configuration fails to be deleted, the CR is not released. You can check the error message in the status field of the CR. To ignore the error and directly delete the CR, you can manually remove the finalizers field from the metadata of the CR.

CR parameters

apiVersion
The value is fixed as telemetry.alibabacloud.com/v1alpha1.
kind
The value is fixed as ClusterAliyunPipelineConfig.

metadata

Parameter

Data type

Required

Description

name

string

Yes

The unique identifier of the CR. This value is also the name of the created iLogtail collection configuration.

Note:

This parameter serves as the iLogtail collection configuration name. It must be unique within the project and cannot be changed after creation. The value must meet the following requirements:

It can contain only lowercase letters, digits, hyphens (-), and underscores (_).

It must start and end with a lowercase letter or digit.

Its length must be between 2 and 128 characters.

spec

Parameter	Data type	Required	Description
project	PipelineProject	Yes	Information about the destination project. Note The project field cannot be changed after the CR is created. To switch projects, create a new CR.
config	PiplineConfig	Yes	Defines details of the iLogtail collection configuration.
logstores	PipelineLogstore	No	A list of Logstores to create. You can create multiple Logstores. Note All parameters except name take effect only when creating a Logstore. This list does not specify the destination Logstore for the iLogtail collection configuration. It only creates Logstores as needed. The destination Logstore is determined by the flusher plug-in in the config field. If the destination Logstore specified in the iLogtail collection configuration already exists, you can leave this field empty. You can only add or delete Logstore definitions in this list. You cannot modify existing Logstores.
machineGroups	PipelineMachineGroup	No	A list of machine groups to associate with the iLogtail collection configuration. When you install the `logtail-ds` component, Simple Log Service automatically creates a machine group named `k8s-group-${your_k8s_cluster_id}`. By default, the list contains only this machine group. Note The alibaba-log-controller ensures that the iLogtail collection configuration is associated only with the machine groups specified in machineGroups. Machine groups not listed are dissociated.
enableUpgradeOverride	bool	No	Default value: false. When an AliyunLogConfig-defined collection configuration exists in the cluster and points to the same iLogtail collection configuration as the current AliyunPipelineConfig, you can: Enable this option: alibaba-log-controller performs an override upgrade on the existing AliyunLogConfig-defined collection configuration. Disable this option: A configuration conflict occurs, and the AliyunPipelineConfig application fails. Note Definitions that point to the same iLogtail collection configuration: The source and target projects are the same. For AliyunLogConfig, the project is the cluster default project or spec.project. For AliyunPipelineConfig, the project is spec.project.name. Same iLogtail collection configuration name. In AliyunLogConfig, the parameter is spec.logtailConfig.configName. In AliyunPipelineConfig, the parameter is metadata.Name. Definition of an overwrite upgrade: First, AliyunPipelineConfig takes effect and updates the iLogtail collection configuration. Then, if AliyunPipelineConfig succeeds, the system deletes the AliyunLogConfig in the cluster that points to the same iLogtail collection configuration. Finally, the override upgrade completes.

PipelineProject

Note

The project field cannot be changed after the CR is created. To switch to a different project, you must create a new CR.

Parameter	Data type	Required	Description
name	string	Yes	The name of the destination project. Note If the specified project does not exist, Simple Log Service automatically creates it.
description	string	No	Description of the target project. Note This parameter takes effect only when creating a project.
endpoint	string	No	The service endpoint of the destination project. For more information, see Endpoints. By default, the endpoint for the region where the current cluster resides is used. If the destination project and the cluster are in different regions, configure this parameter. Note This parameter controls only the region of the collection configuration. To deliver iLogtail data to this region, separately configure iLogtail's config_server_address and data_server_list. For more information, see Logtail configuration files and record files.
uid	string	No	The UID of the destination account. By default, the account to which the current cluster belongs is used. To deliver logs across accounts, configure this parameter and set the following environment variable for alibaba-log-controller: `ALICLOUD_LOG_ACCOUNT_INFOS={"<uid>":{"accessKeyID":"<your_access_key_id>","accessKeySecret":"<your_access_key_secret>"}}`. Note This parameter controls only the account that owns the project. To deliver iLogtail data to this account, separately configure the iLogtail user identity. For more information, see Configure a user identity.

PipelineConfig

Configure the parameters based on the requirements of the CreateLogtailPipelineConfig API. The configName parameter uses the `metadata.name` of the CR and cannot be configured separately.

Parameter	Data type	Required	Description
sample	string	No	A log sample. Multiple logs are supported, with a total length not exceeding 1,500 bytes.
global	object	No	Global configuration. For more information, see Request parameters.
inputs	object list	Yes	List of input plug-ins. For details and plug-in parameters, see Request parameters. Note Only one input plug-in is allowed.
processors	object list	No	List of processing plug-ins. For details and plug-in parameters, see Request parameters.
aggregators	object list	No	List of aggregation plug-ins. For details and plug-in parameters, see Request parameters.
flushers	object list	Yes	List of output plug-ins. For details and plug-in parameters, see Request parameters. Note Only one flusher_sls plug-in is allowed.
configTags	map	No	Custom tags used to mark the iLogtail collection configuration.

PipelineLogstore

Note

All parameters except `name` take effect only when you create a Logstore.

Parameter	Data type	Required	Description
name	string	Yes	The name of the Logstore to create.
queryMode	string	No	The type of the destination Logstore. For more information, see Logstore types. Default value: standard. Valid values: query: Query Logstore standard: Standard Logstore
ttl	int	No	The data retention period of the destination Logstore, in days. For more information, see Manage a Logstore. Default value: 30. Valid values: 1 to 3650. A value of 3650 indicates permanent retention.
hotTtl	int	No	The retention period for hot data storage in the target Logstore. Unit: days. Default value: 0. The value of this field must be less than the value of the ttl field and greater than or equal to 7.
infrequentAccessTTL	int	No	The IA storage period for the destination Logstore, in days. The default value is 0. The `hotTtl` parameter must be specified. Its value must be less than `ttl` and greater than or equal to 30. If `hotTtl+infrequentAccessTTL` is not equal to `ttl`, the value of `ttl-(hotTtl+infrequentAccessTTL)` must also be greater than or equal to 60. This parameter takes effect only when the version of the `loongcollector-operator` component is 1.0.6 or later.
shardCount	int	No	The number of shards in the destination Logstore. For more information, see Manage a Logstore. Default value: 2. Valid values: 1 to 100.
maxSplitShard	int	No	The maximum number of shards for automatic sharding in the destination Logstore. For more information, see Manage a Logstore. Default value: 64. Valid values: 1 to 256.
autoSplit	bool	No	Specifies whether to enable automatic shard splitting for the destination Logstore. For more information, see Manage a Logstore. Default value: true.
telemetryType	string	No	The observable data type of the destination Logstore. For more information, see Create a Logstore. Default value: None. Valid values: None: log type. Metrics: The type of metric.
appendMeta	bool	No	Specifies whether to enable public IP address recording for the destination Logstore. For more information, see Manage a Logstore. Default value: true.
enableTracking	bool	No	Specifies whether to enable Web Tracking for the destination Logstore. For more information, see Manage a Logstore. Default value: false.
encryptConf	object	No	Encryption configuration information for the destination Logstore. For more information, see EncryptConf. Default value: empty.
meteringMode	string	No	The billing mode of the destination Logstore. For more information, see Manage a Logstore. Default value: empty. Valid values: ChargeByFunction: pay-by-function ChargeByDataIngest: pay-by-ingested-data Note If the Logstore's queryMode is query, only ChargeByFunction is supported. If pay-by-ingested-data billing is not enabled for your account, you cannot set this parameter to ChargeByDataIngest.
index	object	No	Specifies the index (effective only during Logstore creation). For the format, see the common data format index. This parameter is effective only when the loongcollector-operator component version is 1.0.6 or later.

PipelineMachineGroup

Parameter	Data type	Required	Description
name	string	Yes	The name of the machine group to associate with the iLogtail collection configuration. Note If the machine group does not exist, alibaba-log-controller automatically creates an identifier-based machine group with the same name as the identifier. For more information, see Create a custom identifier-based machine group. This behavior takes effect only during machine group creation.