:Spark application configuration guide

Precautions

Data Lakehouse Edition (V3.0) is now in canary release. To request a trial for Data Lakehouse Edition (V3.0), Submit a ticket.

Batch applications

Spark supports batch, streaming, and SQL applications. In particular, executing a batch application requires the JAR packages or Python files that contain the entry class. This may also change to include the JAR packages required for applications, the Python sandboxes, and the entry class parameters based on the actual requirements of your business.

You can write a Spark batch application in AnalyticDB for MySQL by using command-line parameters similar to those of the spark-submit tool.

Example of batch application configuration

 {
  "args": ["oss://${testBucketName}/data/test/test.csv"],
  "name": "spark-oss-test",
  "file": "oss://${testBucketName/jars/test/spark-examples-0.0.1-SNAPSHOT.jar",
  "className": "com.aliyun.spark.oss.SparkReadOss",
  "conf": {
    "spark.driver.resourceSpec": "medium",
    "spark.executor.resourceSpec": "medium",
    "spark.executor.instances": 2,
    "spark.adb.connectors": "oss"
  }
}

SQL applications

AnalyticDB for MySQL allows you to submit Spark SQL applications directly in the console without the need to package statements into a JAR file or write Python code. This helps data developers analyze data by using Spark. When you submit a Spark SQL application, you must set the application type to SQL.

Example of SQL application configuration

-- Here is just an example of SparkSQL. Modify the content and run your spark program.

conf spark.driver.resourceSpec=medium;
conf spark.executor.instances=2;
conf spark.executor.resourceSpec=medium;
conf spark.app.name=Spark SQL Test;
conf spark.adb.connectors=oss;

-- Here are your sql statements
show databases;

Statement types supported by Spark SQL

You can edit SQL statements in the Spark editor. Each individual statement must be separated by a semicolon (;).

Switch the metadata service

Description of conf parameters

Spark configuration parameters of AnalyticDB for MySQL are similar to those of Apache Spark. The following tables describe the configuration parameters of AnalyticDB for MySQL inconsistent with those of Apache Spark and the configuration parameters specific to AnalyticDB for MySQL.

• Specified driver and executor resources
  Important The usage of the parameters described in the following table is different from that of Apache Spark.
  The spark.driver.resourceSpec and spark.executor.resourceSpec parameters are set to the same value.

  Parameter	Description	Corresponding parameter in Apache Spark
  spark.driver.resourceSpec	The resource specifications of the Spark driver. Each type corresponds to a distinct specification. For more information, see the Type column in the "Spark resource specifications" section of this topic. Example: CONF spark.driver.resourceSpec = c.small;. In this example, the Spark driver occupies resource specifications of 1 core and 2 GB memory.	spark.driver.cores and spark.driver.memory
  spark.executor.resourceSpec	The resource specifications of each Spark executor. Each type corresponds to a distinct specification. For more information, see the Type column in the "Spark resource specifications" section of this topic. Example: CONF spark.executor.resourceSpec = c.small;. In this example, each Spark executor occupies resource specifications of 1 core and 2 GB memory.	spark.executor.cores and spark.executor.memory

• Spark UI

  Parameter	Default value	Description
  spark.app.log.rootPath	None	The directory in which the Spark UI event logs and the logs generated by AnalyticDB for MySQL Spark applications are stored. You must manually specify an OSS path for this parameter. Otherwise, you cannot access the Spark UI or view application logs.

• RAM user O&M

  Parameter	Default value	Description
  spark.adb.roleArn	None	The Alibaba Cloud Resource Name (ARN) of the Resource Access Management (RAM) role that is granted permissions to submit Spark applications in the RAM console. For more information, see RAM role overview. This parameter is required only when you want to submit Spark applications as a RAM user.

• Built-in data source connectors in Spark of AnalyticDB for MySQL

  Parameter	Default value	Description
  spark.adb.connectors	None	The names of the built-in connectors in Spark of AnalyticDB for MySQL. Separate the connector names with commas (,). Valid values: OSS, hbase1.x, and tablestore.
  spark.hadoop.io.compression.codec.snappy.native	false	Specifies whether a Snappy file is in the standard Snappy format. By default, Hadoop recognizes the Snappy files edited in Hadoop. If you set this parameter to true, the standard Snappy library is used for decompression. Otherwise, the default Snappy library of Hadoop is used for decompression.

• VPC access and data source connection

  Parameter	Default value	Description
  spark.adb.eni.vswitchId	None	The ID of the vSwitch that is associated with an elastic network interface (ENI). This ID is used to access a virtual private cloud (VPC). If your Elastic Compute Service (ECS) instance can access a destination data source, you can set this parameter to the ID of the vSwitch to which your ECS instance is connected.
  spark.adb.eni.securityGroupId	None	The ID of the security group that is associated with an ENI. This ID is used to access a VPC. If your ECS instance can access a destination data source, you can set this parameter to the ID of the security group to which your ECS instance belongs.
  spark.adb.eni.extraHosts	None	The mappings between IP addresses and hostnames. This parameter allows Spark to parse the hostnames of data sources. This parameter is required if you want to connect to a self-managed Hive data source. Note Separate IP addresses and hostnames with spaces. Separate multiple groups of IP addresses and hostnames with commas (,). Example: "ip0 master0, ip1 master1".

• Connection from Spark SQL to the metadata of AnalyticDB for MySQL

  Parameter	Default value	Description
  spark.sql.hive.metastore.version	None	The version of the Hive metastore. If you set this parameter to ADB, you can access the metadata of AnalyticDB for MySQL and read table data from AnalyticDB for MySQL.

• Application reties

  Parameter	Default value	Description
  spark.adb.maxAttempts	1	The maximum number of retries. The default value is 1, which indicates that the spark application is not retried when it fails. If you set this parameter to 3, the Spark application is retried up to three times within a sliding window.
  spark.adb.attemptFailuresValidityInterval	Long.MAX	The duration of the sliding window. Unit: milliseconds. For example, if you set this parameter to 6000, the system counts the number of retries within the last 6,000 milliseconds after a retry fails. If the number of retries is less than the value of spark.adb.maxAttempts, the system continues to retry the application.

• Source configuration

  Parameter	Default value	Description
  spark.adb.driver.cpu-vcores-ratio	1	The ratio of vCPUs to actual CPU cores used by the driver. For example, if the driver uses the medium resource specifications of 2 cores and 8 GB memory and you set this parameter to 2, the driver can run 4 vCPUs in parallel. You can also set spark.driver.cores to 4 to achieve the same performance.
  spark.adb.executor.cpu-vcores-ratio	1	The ratio of vCPUs to actual CPU cores used by an executor. If the CPU utilization of a task is low, you can configure this parameter to increase the CPU utilization. For example, if an executor uses the medium resource specifications of 2 cores and 8 GB memory and you set this parameter to 2, the executor can run 4 vCPUs in parallel. This means that 4 tasks are scheduled in parallel. You can also set spark.executor.cores to 4 to achieve the same performance.

Spark resource specifications