E-MapReduce:Use Simple Log Service to collect Spark job logs

Step 1: Enable the Logtail component

Enable the Logtail component for Simple Log Service in your ACK cluster. See Collect container logs from ACK clusters.

Step 2: Open the SLS project console

1. Log on to the Container Service for Kubernetes console.

2. On the Clusters page, click the name of your cluster, or click Details in the Actions column.

3. On the Basic Information page, go to the Cluster Resources area and click the link in the Log Service Project row.

The SLS project console opens.

Step 3: Create two Logstores

On the Logstores tab, create two Logstores:

• spark-driver-log — collects Spark driver logs

• spark-executor-log — collects Spark executor logs

For instructions, see Create a project and a Logstore.

Step 4: Configure the spark-driver-log Logstore

1. In the spark-driver-log Logstore, create a Logtail configuration with Kubernetes - Standard Output as the data source. Select an existing Kubernetes machine group.

2. Under Data Collection > Logtail Configurations, select the Kubernetes machine group.

3. Switch to editor mode and enter the following configuration:

   Field	Value	Required	Description
   SPARKLOGENV	spark-driver	Yes	Environment variable filter — collects logs only from Spark driver containers
   Stdout	true	Yes	Collect standard output
   Stderr	true	Yes	Collect standard error
   BeginLineCheckLength	10	Yes	Number of characters checked to identify the start of a new log line
   BeginLineRegex	\\d+/\\d+/\\d+.*	Yes	Regex pattern that marks the beginning of a log entry (matches Spark's timestamp format)
   type	service_docker_stdout	Yes	Input plugin for collecting container stdout/stderr

   {
       "inputs": [
           {
               "detail": {
                   "IncludeEnv": {
                       "SPARKLOGENV": "spark-driver"
                   },
                   "Stderr": true,
                   "Stdout": true,
                   "BeginLineCheckLength": 10,
                   "BeginLineRegex": "\\d+/\\d+/\\d+.*"
               },
               "type": "service_docker_stdout"
           }
       ]
   }

   Key fields:

Step 5: Configure the spark-executor-log Logstore

Repeat Step 4 for the spark-executor-log Logstore, but change SPARKLOGENV to spark-executor:

{
    "inputs": [
        {
            "detail": {
                "IncludeEnv": {
                    "SPARKLOGENV": "spark-executor"
                },
                "Stderr": true,
                "Stdout": true,
                "BeginLineCheckLength": 10,
                "BeginLineRegex": "\\d+/\\d+/\\d+.*"
            },
            "type": "service_docker_stdout"
        }
    ]
}

All other fields are identical to the driver configuration.

Step 6: Enable indexing

Enable indexing for both Logstores so you can run queries. See Create indexes.