Data Lake Analytics - Deprecated:Overview

Apache Spark is a prevailing engine in the big data field. It applies to data lake scenarios and uses built-in connectors to access data sources. These connectors allow you to extend API operations with ease. Apache Spark supports SQL and allows you to write DataFrame in multiple programming languages. This makes Apache Spark easy to use and flexible. Apache Spark serves as an end-to-end engine to support features, such as SQL, streaming, machine learning, and graph computing.

Before you use Apache Spark, you must deploy a set of open source and basic big data components. These components include Yarn, Hadoop Distributed File System (HDFS), and ZooKeeper. After you deploy these components, the following issues may occur:

• Complex development and O&M operations: To complete the development and perform O&M operations, developers must be familiar with a variety of big data components. If they encounter issues, they must conduct in-depth research on the source code provided by the Apache Spark community.

• High O&M costs: Enterprises require an O&M team to maintain open source components. The O&M team needs to configure resource nodes, configure and deploy open source software, monitor and update open source components, and scale clusters. Customized development is also required to meet enterprise-level requirements, such as permission isolation and monitoring and alerting.

• High resource costs: Loads of Spark jobs significantly fluctuate over time. During off-peak hours, large amounts of idle resources exist in Apache Spark clusters. Cluster management and control components still consume resources during off-peak hours but do not bring business value to customers. These components include master nodes, ZooKeeper, and Hadoop.

• Lack of elasticity: During peak hours, enterprises need to accurately estimate resource requirements and add machines at the earliest opportunity. If you add a large number of machines, some machines may not be used. If you add only a small number of machines, your business may be affected due to insufficient resources. In addition, the cluster scale-out process is complex and time-consuming, and resources may become insufficient.

The serverless Spark engine of DLA is a big data analysis and computing service. This engine is developed based on Apache Spark and uses a service-oriented architecture (SOA). The following figure shows the architecture of the serverless Spark engine of DLA.

DLA deeply integrates this engine with Spark, serverless, and cloud-native technologies. Compared with Apache Spark, the serverless Spark engine of DLA provides the following benefits:

• Easy to use: provides simple API operations and scripts without requiring developers to understand basic components at the underlying layer. In addition, the serverless Spark engine provides an easy way to perform operations in the DLA console. It allows developers with only a basic knowledge of Apache Spark to develop big data services.

• Zero O&M: provides product interfaces for you to manage Spark jobs. You do not need to configure servers or Hadoop clusters, or perform O&M operations such as scaling.

• Job-based scalability: allows you to create resources based on the driver and executors. Compared with compute units (CUs) in Apache Spark clusters, the serverless Spark engine of DLA reduces the probability of insufficient resources. This engine allows you to start up to 500 to 1,000 CUs in a minute. This meets business resource requirements.

• Low costs: uses the pay-as-you-go billing method. You are charged only for the jobs that you used. You are not charged for resource management and control. In addition, you do not need to pay for idle computing resources during off-peak hours.

• Superior performance: improves performance threefold to fivefold in typical scenarios when Alibaba Cloud services, such as Object Storage Service (OSS), are deployed. To achieve this purpose, the development team of DLA customizes and optimizes the serverless Spark engine based on Apache Spark.

  Note

  For more information about performance comparison results, see Test results.

• Enterprise-level capability: shares metadata with the serverless Presto engine of DLA. You can execute the GRANT and REVOKE statements to manage permissions granted to Resource Access Management (RAM) users. The serverless Spark engine provides a user-friendly web UI. Compared with the Apache Spark history server, the serverless Spark engine takes only a few seconds to open the web UI, no matter how complex a job is.