This topic describes how to configure and view the job priorities of job resource groups in AnalyticDB for MySQL Data Lakehouse Edition (V3.0).
Prerequisites
An AnalyticDB for MySQL Data Lakehouse Edition (V3.0) cluster of V3.1.6.3 or later is created.
A job resource group is created. A job is submitted to run in the job resource group.
Usage notes
You cannot modify the priorities of submitted jobs regardless of whether the jobs are running.
Overview
Job priorities refer to the priorities at which jobs in a queue are allocated resources. If a large number of jobs are concurrently run in a job resource group of an AnalyticDB for MySQL Data Lakehouse Edition (V3.0) cluster, the jobs may require more resources than the capacity of the resource group. AnalyticDB for MySQL Data Lakehouse Edition (V3.0) supports the job queue feature. If resources are insufficient, new jobs are queued and remain in the submitted state.
For example, if a key metric depends on the intermediate result of an upstream job, you must configure a higher priority for the upstream job to provide sufficient resources for the job. AnalyticDB for MySQL Data Lakehouse Edition (V3.0) provides the priority queue feature for jobs in job resource groups. You can configure job priorities to allow jobs to enter the following priority queues: LOWEST, LOW, NORMAL, and HIGH. Jobs that have higher priorities are preferentially run.
Configure job priorities
You can configure the following priorities for jobs: HIGH, NORMAL, LOW, and LOWEST. If you do not configure a priority for a job, the job is assigned the NORMAL priority.
Extract-transform-load (ETL) queries and SELECT queries are of the NORMAL priority.
The configuration methods of job priorities vary based on the job type. The following section describes how to configure a job priority:
Configure a priority for XIHE BSP jobs
Add a hint: /*+ query_priority= <priority level>*/
Example: Set the priority of a XIHE bulk synchronous parallel (BSP) job to HIGH to allow the job to enter the HIGH queue.
/*+ query_priority=HIGH*/ SELECT * FROM test_table;Configure a priority for Spark SQL jobs
Execute the SET statement: SET spark.adb.priority = <priority level>
Example: Set the priority of a Spark SQL job to LOW to allow the job to enter the LOW queue.
SET spark.adb.priority = LOW;
SELECT * FROM test_table;Configure a priority for Spark batch jobs
Configure the conf parameter: "conf": { "spark.adb.priority": "<priority level>"}
Example: Set the priority of a Spark batch job to HIGH to allow the job to enter the HIGH queue.
{
"comments": [
"-- Here is just an example of SparkPi. Modify the content and run your spark program."
],
"args": [
"1000"
],
"file": "local:///tmp/spark-examples.jar",
"name": "SparkPi",
"className": "org.apache.spark.examples.SparkPi",
"conf": {
"spark.driver.resourceSpec": "medium",
"spark.executor.instances": 2,
"spark.executor.resourceSpec": "medium",
"spark.adb.priority": "HIGH"
}
}View job priorities
View the priorities of XIHE BSP and Spark SQL jobs
Log on to the AnalyticDB for MySQL console. In the left-side navigation pane of a cluster, choose Job Development > SQL Development. On the Execution Records tab, view the priorities of XIHE BSP and Spark SQL jobs.
View the priorities of Spark batch jobs
Log on to the AnalyticDB for MySQL console. In the left-side navigation pane of a cluster, choose Job Development > Spark JAR Development. On the Applications tab, view the priorities of Spark batch jobs.
