DataWorks:Develop a MaxCompute Spark task

Limits

If an error is reported when you commit an ODPS Spark node that uses the Spark 3.X version, purchase a serverless resource group. For more information, see Create and use a serverless resource group.

Preparations

ODPS Spark nodes allow you to use Java, Scala, or Python to develop and run offline Spark on MaxCompute tasks. The operations and parameters that are required for developing the offline Spark on MaxCompute tasks vary based on the programming language that you use. You can select a programming language based on your business requirements.

Java/Scala

Before you run Java or Scala code in an ODPS Spark node, you must complete the development of code for a Spark on MaxCompute task on your on-premises machine and upload the code to DataWorks as a MaxCompute resource. You must perform the following steps:

1. Prepare a development environment.

   You must prepare the development environment in which you want to run a Spark on MaxCompute task based on the operating system that you use. For more information, see Set up a Linux development environment or Set up a Windows development environment.

2. Develop Java or Scala code.

   Before you run Java or Scala code in an ODPS Spark node, you must complete the development of code for a Spark on MaxCompute task on your on-premises machine or in the prepared development environment. We recommend that you use the sample project template provided by Spark on MaxCompute.

3. Package the developed code and upload the code to DataWorks.

   After the code is developed, you must package the code and upload the package to DataWorks as a MaxCompute resource. For more information, see Create and use MaxCompute resources.

Programming language: Python (Use the default Python environment)

Programming language: Python (Use a custom Python environment)

If the default Python environment cannot meet your business requirements, you can perform the following steps to prepare a custom Python environment to run your Spark on MaxCompute task.

1. Prepare a Python environment on your on-premises machine.

   You can refer to PySpark Python versions and supported dependencies to configure a Python environment based on your business requirements.

2. Package the code for the Python environment and upload the package to DataWorks.

   You must package the code for the Python environment in the ZIP format and upload the package to DataWorks as a MaxCompute resource. This way, you can run the Spark on MaxCompute task in the environment. For more information, see Create and use MaxCompute resources.

Descriptions of parameters

You can run offline Spark on MaxCompute tasks in cluster mode in DataWorks. In this mode, you must specify the Main method as the entry point of a custom application. A Spark task ends when Main enters the Success or Fail state. You must add the configuration items in the spark-defaults.conf file to the configurations of the ODPS Spark node. For example, you must add the configuration items such as the number of executors, the memory size, and spark.hadoop.odps.runtime.end.point.

Simple code editing example

This section provides a simple example to show how to use an ODPS Spark node to develop a Spark on MaxCompute task. In this example, a Spark on MaxCompute task is developed to determine whether a string can be converted into digits.

1. Create a resource.

   1. On the DataStudio page of the DataWorks console, create a Python resource named spark_is_number.py. For more information, see Create and use MaxCompute resources. Sample code:

      # -*- coding: utf-8 -*-
      import sys
      from pyspark.sql import SparkSession
      
      try:
          # for python 2
          reload(sys)
          sys.setdefaultencoding('utf8')
      except:
          # python 3 not needed
          pass
      
      if __name__ == '__main__':
          spark = SparkSession.builder\
              .appName("spark sql")\
              .config("spark.sql.broadcastTimeout", 20 * 60)\
              .config("spark.sql.crossJoin.enabled", True)\
              .config("odps.exec.dynamic.partition.mode", "nonstrict")\
              .config("spark.sql.catalogImplementation", "odps")\
              .getOrCreate()
      
      def is_number(s):
          try:
              float(s)
              return True
          except ValueError:
              pass
      
          try:
              import unicodedata
              unicodedata.numeric(s)
              return True
          except (TypeError, ValueError):
              pass
      
          return False
      
      print(is_number('foo'))
      print(is_number('1'))
      print(is_number('1.3'))
      print(is_number('-1.37'))
      print(is_number('1e3'))

   2. Save and commit the resource.

2. In the created ODPS Spark node, configure parameters and scheduling properties for the MaxCompute Spark task by referring to the Descriptions of parameters section in this topic, and save and commit the node.

   Parameter	Description
   Spark Version	Select Spark2.x.
   Language	Select Python.
   Main Python Resource	The Python resource spark_is_number.py that you created.

3. Go to Operation Center in the development environment to backfill data for the ODPS Spark node. For more information, see Backfill data and view data backfill instances (new version).

   Note

   DataWorks does not provide entry points for you to run ODPS Spark nodes in DataStudio. You must run ODPS Spark nodes in Operation Center in the development environment.

4. View the result.

   After the data backfill instance is successfully run, click tracking URL in the run logs that are generated to view the result. The following information is returned:

   False
   True
   True
   True
   True

Advanced code editing examples

For more information about the development of Spark on MaxCompute tasks in other scenarios, see the following topics:

• Spark 1.x examples for Java or Scala

• Spark 2.x examples for Java or Scala

• Develop a Spark on MaxCompute application by using PySpark

• Access instances in a VPC from Spark on MaxCompute

• Access OSS from Spark on MaxCompute