All Products
Search

This Product

    E-MapReduce:Manage default configurations

    Document Center

    E-MapReduce:Manage default configurations

    Last Updated:Nov 25, 2024

    Alibaba Cloud E-MapReduce (EMR) Serverless Spark provides default global Spark configurations to run and manage various types of jobs. The default configurations contain all the configuration information that is required to run specific jobs. This ensures that the configurations and the runtime environment used to submit and run jobs are the same.

    Prerequisites

    A workspace is created. For more information, see Manage workspaces.

    Configure parameters

    In the left-side navigation pane of the EMR Serverless Spark page, click Configurations to view or modify the related parameters.

    Parameter

    Description

    Engine Version

    The version of the engine that is used by the compute. For more information about engine versions, see Engine versions.

    spark.driver.cores

    The number of CPU cores that are used by the driver of the Spark application.

    spark.driver.memory

    The size of memory that is available to the driver of the Spark application.

    spark.executor.cores

    The number of CPU cores that can be used by each executor.

    spark.executor.memory

    The size of memory that is available to each executor.

    spark.executor.instances

    The number of executors that are allocated to the Spark application.

    Dynamic Resource Allocation

    By default, this feature is disabled. After you enable this feature, you must configure the following parameters:

    • Minimum Number of Executors: Default value: 2.

    • Maximum Number of Executors: If you do not configure spark.executor.instances, the default value 10 is used.

    More Memory Configurations

    • spark.driver.memoryOverhead: the size of non-heap memory that is available to each driver. If you leave this parameter empty, Spark automatically assigns a value to this parameter based on the following formula: max(384 MB, 10% × spark.driver.memory).

    • spark.executor.memoryOverhead: the size of non-heap memory that is available to each executor. If you leave this parameter empty, Spark automatically assigns a value to this parameter based on the following formula: max(384 MB, 10% × spark.executor.memory).

    • spark.memory.offHeap.size: the size of off-heap memory that is available to the Spark application. Default value: 1 GB.

      This parameter is valid only if you set the spark.memory.offHeap.enabled parameter to true. By default, if you use the Fusion engine, the spark.memory.offHeap.enabled parameter is set to true and the spark.memory.offHeap.size parameter is set to 1 GB.

    Spark Configuration

    The Spark configurations. Separate the configurations with spaces, such as spark.sql.catalog.paimon.metastore dlf.