Enable Simple Log Service for a workflow cluster and access Simple Log Service from a workflow cluster - Container Service for Kubernetes

In most cases, you need to configure reclaim policies for workflows and pods. This prevents linear increases of resources used by the control planes and workflow controllers in the cluster after workflows are completed. Access to workflow logs is required for process analysis and workflow troubleshooting. After a pod is deleted, open source Kubernetes does not retain the logs of the pod or the logs of the workflow for which the pod is created. To retain the logs, you can use workflow clusters. Workflow clusters are integrated with Simple Log Service, which allows you to collect the logs and store them in Simple Log Service. You can use the Argo CLI or Argo UI to view the collected logs.

Usage notes

After you enable log collection for a workflow, you can use the Argo CLI or Argo UI to view the collected logs before the workflow is deleted, regardless of whether the pod of the workflow is deleted.
If a workflow is persisted to a database, you can use the Argo CLI to download the ZIP file of the workflow logs or view the workflow logs in the Simple Log Service console after the workflow is deleted.
For more information about how to persist a workflow to a database, see Persist workflows.

Precautions

If oss-artifact-repository is enabled and archiveLogs: true is specified for a workflow cluster, Simple Log Service does not collect pod logs from the cluster. In this case, the pod logs are collected to oss-artifact-repository.
Compared with oss-artifact-repository, Simple Log Service provides automated log lifecycle management and powerful log query features. Simple Log Service allows you to specify the retention period of logs. To enable Simple Log Service in this case, delete the archiveLogs: true setting.

Prerequisites

If you choose to use the Argo CLI to enable Simple Log Service for log collection, you must first perform the following operations:

Download and install Alibaba Cloud Argo CLI 3.4.12. For more information, see Alibaba Cloud Argo CLI.
If you use a Resource Access Management (RAM) user, you must grant the RAM user read-only access to Simple Log Service by attaching the AliyunLogReadOnlyAccess policy to the RAM user. For more information, see Create a RAM user and authorize the RAM user to access Simple Log Service.

Enable Simple Log Service

After you create a workflow cluster, a Simple Log Service project named k8s-log-<clusterid> is automatically created by the system to collect workflow logs from the cluster. If the k8s-log-<clusterid> project is not created, you can manually create the project. For more information about how to create a Simple Log Service project, see Create a Simple Log Service project.

You can enable Simple Log Service by using the Alibaba Cloud Argo CLI or creating an AliyunLogConfig.

Enable Simple Log Service by using the Alibaba Cloud Argo CLI

The Alibaba Cloud Argo CLI is fully compatible with the open source Argo CLI and provides improved logging capabilities. The Alibaba Cloud Argo CLI can collect the logs of deleted pods.

Run the following command to configure Simple Log Service:

argo config sls
Please input log retention days. Default is 7 days.
10

Expected output:

Start to config SLS for your cluster.
Created AliyunLogConfig CR workflow-sls-config in default namespace.
Created SLS logstore workflow-logstore in SLS project k8s-log-<clusterid>, log retention days is 10 days

The output shows that Simple Log Service is configured. The logs of all workflows in the cluster are collected to the workflow-logstore Logstore.

Log on to the Simple Log Service console, find the k8s-log-<clusterid> project, and then click workflow-logstore to view logs.

Enable Simple Log Service by creating an AliyunLogConfig

Run the following command to create an AliyunLogConfig. The Log Service controller automatically creates a project named k8s-log-<clusterid> and a Logstore named workflow-logstore.

cat << EOF | kubectl apply -f -
apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  name: workflow-sls-config
  namespace: default
spec:
  # log will store for 5 days
  lifeCycle: 5
  logstore: workflow-logstore
  logtailConfig:
    inputType: plugin
    configName: workflow-sls-config
    inputDetail:
      plugin:
        inputs:
        - detail:
            Stderr: true
            Stdout: true
          type: service_docker_stdout
EOF

Access Simple Log Service by using the Argo CLI

View workflow logs in the cluster

You can use the Argo CLI to view workflows logs, including the logs of existing pods and the logs of deleted pods.

You can run the kubectl logs <pod-name> command to view the logs of an existing pod. By default, the command returns up to 2,000 rows of log data. If you want to view more log data or view the logs of a deleted pod, you can select one of the following methods:

Use the Argo CLI to access Simple Log Service (recommended)

Run the following command to grant permissions to the Argo CLI:

argo config init # Specify information such as the AccessKey ID and AccessKey secret based on the on-screen instructions.

Run the following command to query the logs of a pod:

argo logs <workflow-name> <pod-name> --sls # Query the logs of the specified pod. 
argo logs <workflow-name> --sls # Download the ZIP file of the logs of the specified workflow.

Use Argo Server to access Simple Log Service

If you want to view the logs of deleted pods by using the Argo CLI, you must first enable Argo Server and configure the required parameters.

Note

To query the workflow logs collected by Simple Log Service, you must specify <pod-name>.

To access workflows or pods in another namespace, you must obtain the KUBE_TOKEN for the namespace.

Run the following commands to enable Argo Server and configure parameters:

export ARGO_SERVER=argo.<cluster id>.<region>.alicontainer.com:2746
export KUBE_TOKEN=$(k create token default -n default --duration 24h)
export ARGO_TOKEN="Bearer $KUBE_TOKEN"
export ARGO_INSECURE_SKIP_VERIFY=true

Run the following command to query the log of a workflow pod:
```
argo logs <workflow-name> <pod-name>
```

View the logs of workflows that are persisted to databases

You can persist workflows to databases. This way, you can use the Argo CLI to download the workflow logs after the workflows are deleted.

Run the following command to grant permissions to the Argo CLI:

argo config init # Specify information such as the AccessKey ID and AccessKey secret based on the on-screen instructions.

Run the following command to query the user ID (UID) of a workflow:

# argo archive list
NAMESPACE   NAME                                    STATUS      AGE   DURATION   PRIORITY   MESSAGE   P/R/C   PARAMETERS   UID
default     hello-world-l6c2r                       Succeeded   3d    1m         0                    0/0/0                179eaef0-fde3-496f-946d-549e8f******

Download the workflow logs:

# argo archive logs 179eaef0-fde3-496f-946d-549e8f******
# ls
hello-world-l6c2r.zip

Use the Argo UI to access Simple Log Service

You can view the logs of workflow pods by using the Argo UI, regardless of whether the pods are deleted. If you query the logs of a deleted pod, the logs are displayed in the Argo UI, as shown in the following figure.

访问日志

Disable Simple Log Service

Run the following command to delete the AliyunLogConfig:

kubectl delete aliyunlogconfigs.log.alibabacloud.com workflow-sls-config -n default

Log on to the Simple Log Service console and delete the workflow-logstore Logstore.