Alibaba Cloud EMR Serverless Spark is a serverless big data processing service built on the Apache Spark engine. Spark applications run on a standard Java Virtual Machine (JVM), and all Spark tasks are executed through Java or Scala code.
Engine Version Numbers and Their Meanings
The engine version format is esr-** (Spark **, Scala **).
You can use vector-based technologies and native libraries available in the Alibaba Cloud Fusion engine's runtime environment to improve the execution performance of SQL applications. For more information, see Fusion engine.
Content | Meaning |
esr-** | The EMR Serverless Spark version number. For example, esr-2.1. |
Spark ** | The Apache Spark engine version number followed. For example, Spark 3.3.1, which means it is built on Apache Spark 3.3.1. |
Scala ** | The Scala programming language version supported by this engine version. For example, Scala 2.12. |
Engine Versions and Lake Table Format Mappings
In the table below, - indicates that this engine version does not support this component.
5.x Series
Engine Version | Spark Version | Scala Version | Delta Version | Hudi Version | Iceberg Version | Paimon Version | JDK Version |
esr-5.1.0 | 4.0.1 | 2.13 | - | - | - | 1.3 | JDK 17 |
esr-5.0.0 | 4.0.1 | 2.13 | - | - | - | 1.3 | JDK 17 |
4.x Series
Engine Version | Spark Version | Scala Version | Delta Version | Hudi Version | Iceberg Version | Paimon Version | JDK Version |
esr-4.7.0 | 3.5.2 | 2.12 | 3.2.1 | 0.15 | 1.9.2 | 1.3 | JDK 17 |
esr-4.6.0 | 3.5.2 | 2.12 | 3.2.1 | 0.15 | 1.9.2 | 1.3 | JDK 17 |
esr-4.5.1 esr-4.5.0 | 3.5.2 | 2.12 | 3.2.1 | 0.15 | 1.9.2 | 1.2 | JDK 17 |
esr-4.4.1 esr-4.4.0 | 3.5.2 | 2.12 | 3.2.1 | 0.15 | 1.5.0 | 1.2 | JDK 8 |
esr-4.3.0 | 3.5.2 | 2.12 | 3.2.1 | 0.15 | 1.5.0 | 1.1 | JDK 8 |
esr-4.2.0 | 3.5.2 | 2.12 | 3.2.1 | 0.15 | 1.5.0 | 1.0 | JDK 8 |
esr-4.1.1 esr-4.1.0 | 3.5.2 | 2.12 | 3.2.1 | 0.15 | 1.5.0 | 1.0 | JDK 8 |
esr-4.0.0 | 3.5.2 | 2.12 | 3.2.1 | 0.15 | 1.5.0 | 1.0 | JDK 8 |
3.x Series
Engine Version | Spark Version | Scala Version | Delta Version | Hudi Version | Iceberg Version | Paimon Version | JDK Version |
esr-3.6.0 | 3.4.4 | 2.12 | 3.0.0 | 0.15 | 1.5.0 | 1.3 | JDK 8 |
esr-3.5.0 | 3.4.4 | 2.12 | 3.0.0 | 0.15 | 1.5.0 | 1.3 | JDK 8 |
esr-3.4.1 esr-3.4.0 | 3.4.4 | 2.12 | 3.0.0 | 0.15 | 1.5.0 | 1.2 | JDK 8 |
esr-3.3.1 esr-3.3.0 | 3.4.4 | 2.12 | 3.0.0 | 0.15 | 1.5.0 | 1.1 | JDK 8 |
esr-3.2.0 | 3.4.4 | 2.12 | 3.0.0 | 0.15 | 1.5.0 | 1.0 | JDK 8 |
esr-3.1.1 esr-3.1.0 | 3.4.4 | 2.12 | 3.0.0 | 0.15 | 1.5.0 | 1.0 | JDK 8 |
esr-3.0.1 esr-3.0.0 | 3.4.3 | 2.12 | 3.0.0 | 0.15 | 1.5.0 | 1.0 | JDK 8 |
2.x Series
Engine Version | Spark Version | Scala Version | Delta Version | Hudi Version | Iceberg Version | Paimon Version | JDK Version |
esr-2.9.0 | 3.3.1 | 2.12 | 2.2.0 | 0.15 | 1.5.0 | 1.3 | JDK 8 |
esr-2.8.1 esr-2.8.0 | 3.3.1 | 2.12 | 2.2.0 | 0.15 | 1.5.0 | 1.2 | JDK 8 |
esr-2.7.1 esr-2.7.0 | 3.3.1 | 2.12 | 2.2.0 | 0.15 | 1.1.0 | 1.1 | JDK 8 |
esr-2.6.0 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 1.0 | JDK 8 |
esr-2.5.1 esr-2.5.0 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 1.0 | JDK 8 |
esr-2.4.1 esr-2.4.0 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 1.0 | JDK 8 |
esr-2.2.3 esr-2.2.2 esr-2.2 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 0.9 | JDK 8 |
esr-2.1 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 0.9 | JDK 8 |
Default Parameter Adjustments That May Affect Job Behavior
To improve performance and stability, EMR Serverless Spark optimizes some Spark default configurations in its engine versions. These adjustments may cause job behavior to differ from open-source Spark. Before use, evaluate if these changes affect your business logic.
ESR Version | Parameter | Serverless Spark Default Value | Open-source Spark Default Value |
|
|
|
|
|
|
|