DataWorks:Develop ODPS Spark jobs

Limitations

If submission fails for an ODPS Spark Node that uses Spark 3.x, you must purchase and use a Serverless Resource Group. For more information, seeUse serverless resource groups.

Prerequisites

ODPS Spark Nodes support running Spark on MaxCompute Offline Jobs using Java/Scala or Python. The development process and configuration options vary by language.

Parameters

DataWorks runs Spark on MaxCompute Offline Jobs in Cluster Mode. In this mode, you must specify a custom program entry point, main. The Spark Job terminates when the main method completes, returning a Success or Fail status. Additionally, configurations from spark-defaults.conf must be added one by one to the Configuration Item section of the ODPS Spark Node. Examples include the number of executor instances, Memory, and the spark.hadoop.odps.runtime.end.point configuration.

Simple example

This section demonstrates how to use an ODPS Spark Node with a simple example: checking if a string is numeric.

1. Create a resource.

   1. On the Data Development page, create a new Python Resource and name it spark_is_number.py. For more information, see Create and use MaxCompute resources.

      # -*- coding: utf-8 -*-
      import sys
      from pyspark.sql import SparkSession
      
      try:
          # for python 2
          reload(sys)
          sys.setdefaultencoding('utf8')
      except:
          # python 3 not needed
          pass
      
      if __name__ == '__main__':
          spark = SparkSession.builder\
              .appName("spark sql")\
              .config("spark.sql.broadcastTimeout", 20 * 60)\
              .config("spark.sql.crossJoin.enabled", True)\
              .config("odps.exec.dynamic.partition.mode", "nonstrict")\
              .config("spark.sql.catalogImplementation", "odps")\
              .getOrCreate()
      
      def is_number(s):
          try:
              float(s)
              return True
          except ValueError:
              pass
      
          try:
              import unicodedata
              unicodedata.numeric(s)
              return True
          except (TypeError, ValueError):
              pass
      
          return False
      
      print(is_number('foo'))
      print(is_number('1'))
      print(is_number('1.3'))
      print(is_number('-1.37'))
      print(is_number('1e3'))

   2. Save and commit the Resource.

2. In the ODPS Spark Node you created, configure the node Parameters and scheduling Parameters as described in Parameters, then save and commit the Node.

   Parameter	Description
   Spark version	Spark 2.x
   Language	Python
   Select main Python resource	From the drop-down list, select the Python Resource you created: spark_is_number.py.

3. Go to the Operation Center for the development environment and run a Data Backfill Job. For detailed instructions, see Data backfill instance O&M.

   Note

   Because ODPS Spark Nodes in Data Development cannot be run directly, you must execute the Job from the Operation Center in the development environment.

4. View the results.

   After the Data Backfill Instance runs successfully, go to its tracking URL in the Run Log to view the results:

   False
   True
   True
   True
   True

Advanced examples

For more examples of developing Spark on MaxCompute Jobs for different use cases, see the following topics:

• Java/Scala example: Spark 1.x

• Java/Scala example: Spark 2.x

• Python example: PySpark development

• Use case: Access a VPC instance from Spark

• Use case: Access OSS from Spark