Use Spark on an EMR cluster to process data stored in OSS-HDFS - Object Storage Service

This topic describes how to use Spark on an E-MapReduce (EMR) cluster to process data stored in OSS-HDFS.

Prerequisites

A cluster of EMR V3.42.0 or later, or EMR V5.8.0 or later is created. For more information, see Create a cluster.
OSS-HDFS is enabled for a bucket and access permissions on OSS-HDFS are granted. For more information about how to enable OSS-HDFS, see Enable OSS-HDFS and grant access permissions.

Log on to the EMR cluster.
1. Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.
2. Click the EMR cluster that you created.
3. Click the Nodes tab, and then click on the left side of the node group.
4. Click the ID of the ECS instance. On the Instances page, click Connect next to the instance ID.
For more information about how to log on to a cluster in Windows or Linux by using an SSH key pair or SSH password, see Log on to a cluster.
Run the following command on the terminal to start Spark Shell:
```
spark-shell
```

Use Spark to access OSS-HDFS.

Create a table.

spark.sql("CREATE TABLE test_oss (`c1` string) OPTIONS (PATH 'oss://examplebucket.cn-hangzhou.oss-dls.aliyuncs.com/dir')")

Insert data into the table.

spark.sql("INSERT INTO TABLE test_oss SELECT 'testdata' AS c1")