Apache Spark web UI - Data Lake Analytics - Deprecated - Alibaba Cloud Documentation Center

This topic describes how to view the information about a Spark job on Apache Spark web UI when the job is running or completed in Data Lake Analytics (DLA).

Procedure

Log on to the DLA console.
In the top navigation bar, select the region where DLA resides.
In the left-side navigation pane, choose Serverless Spark > Submit job.
On the Parameter Configuration page, click the Job list tab, find your Spark job, and then select SparkUI from the Operation drop-down list.
Use a browser to access Apache Spark web UI. The access method is the same as that provided by the Apache Spark community.

Usage notes

1. Log directory

You can access Apache Spark web UI after your job is completed. The historical logs of the job are automatically uploaded to the OSS directory oss://aliyun-oa-query-results-<user-parent-id>-oss-<region-id>/Spark_Tmp/. If you want to customize the directory, you can use the following parameter to change the directory.

spark.dla.job.log.oss.uri":"oss://{bucket_name}/{dir_name}

Note

bucket_name: The name of the OSS bucket that is created and deployed in the same region as the serverless Spark engine of DLA.
dir_name: the name of the directory. Multi-level directories are supported.

Sample job configurations:

{
    "name": "SparkPi",
    "file": "local:///tmp/spark-examples.jar",
    "className": "org.apache.spark.examples.SparkPi",
    "conf": {
        "spark.dla.job.log.oss.uri": "oss://your-bucket/your-logs-dir"
    }
}

2. Validity period of Apache Spark web UI

Apache Spark web UI occupies the network resources of the service platform. You can access Apache Spark web UI to view the logs of the historical jobs that are completed within the last three days. After the validity period expires, Apache Spark web UI cannot be accessed. However, log data on OSS is not deleted. You can query or delete log data on the OSS client based on your business requirements.