E-MapReduce:Use JindoFS in EMR V3.22.0 to V3.26.3

Overview

JindoFS supports the block storage mode and cache mode.

JindoFS adopts a heterogeneous multi-backup mechanism. Storage Service provides the data storage capability. Data is stored in OSS to ensure high reliability. Redundant backups are stored in the local cluster to accelerate read operations. Namespace Service manages metadata of JindoFS. In this case, metadata is queried from Namespace Service instead of OSS, which improves query performance. This query method of JindoFS is similar to that of Hadoop Distributed File System (HDFS).

Prepare the environment

• Create an EMR cluster

  Select EMR V3.22.0 or a later version. Select SmartData from the optional services. For more information about how to create an EMR cluster, see Create a cluster.

  
• Configure JindoFS
  JindoFS provided by SmartData uses OSS as the storage backend. Therefore, you must configure parameters related to OSS before you use JindoFS. EMR provides the following two configuration methods: 1. Modify parameters related to Bigboot after you create an EMR cluster, and then restart SmartData for the configurations to take effect. 2. Add custom configurations when you create an EMR cluster. In this case, the related services are restarted based on custom parameters after the EMR cluster is created.

  • Initialize parameters after the EMR cluster is created

    You can configure all parameters related to JindoFS in Bigboot, as shown in the following figures.

    1. In the Service Configuration section, click the bigboot tab.

       
    2. Click Custom Configuration in the upper-right corner.
    Note

    • The parameters framed in red in the preceding figures are required.
    • JindoFS supports multiple namespaces. A namespace named test is used in this topic.

    Parameter	Description	Example
    jfs.namespaces	The namespace supported by JindoFS. Separate multiple namespaces with commas (,).	test
    jfs.namespaces.test.uri	The storage backend of the test namespace.	oss://oss-bucket/oss-dir Note You can set the value to a directory in an OSS bucket. In this case, this directory serves as the root directory, in which the test namespace reads and writes data.
    jfs.namespaces.test.mode	The storage mode of the test namespace.	block Note JindoFS supports the block storage mode and cache mode.
    jfs.namespaces.test.oss.access.key	The AccessKey ID used to access the OSS bucket that serves as the storage backend.	xxxx Note We recommend that you store data in an OSS bucket that is in the same region and under the same account as your EMR cluster. This ensures high performance and stability. In this case, you do not need to configure the AccessKey ID and AccessKey secret because the OSS bucket allows password-free access from the EMR cluster.
    jfs.namespaces.test.oss.access.secret	The AccessKey secret used to access the OSS bucket that serves as the storage backend.

    Save and deploy the JindoFS configuration. Restart all components in SmartData to use JindoFS.

    
  • Add custom configurations when you create an EMR cluster
    You can add custom configurations when you create an EMR cluster. Assume that you want to create an EMR cluster in the same region as an OSS bucket to access OSS without using an AccessKey pair. Turn on Custom Software Settings as shown in the following figure. Add the following configuration to the field in the Advanced Settings section to customize parameters for the test namespace:

    [
    {
    "ServiceName":"BIGBOOT",
    "FileName":"bigboot",
    "ConfigKey":"jfs.namespaces","ConfigValue":"test"
    },{
    "ServiceName":"BIGBOOT",
    "FileName":"bigboot",
    "ConfigKey":"jfs.namespaces.test.uri",
    "ConfigValue":"oss://oss-bucket/oss-dir"
    },{
    "ServiceName":"BIGBOOT",
    "FileName":"bigboot",
    "ConfigKey":"jfs.namespaces.test.mode",
    "ConfigValue":"block"
    }
    ]

    

Use JindoFS

The use of JindoFS is similar to that of HDFS. JindoFS also provides a prefix. To use JindoFS, you only need to replace the hdfs prefix with the jfs prefix.

JindoFS supports most of the computing components in the EMR cluster, including Hadoop, Hive, Spark, Flink, Presto, and Impala.

Examples:

• Run shell commands

  hadoop fs -ls jfs://your-namespace/ 
  hadoop fs -mkdir jfs://your-namespace/test-dir
  hadoop fs -put test.log jfs://your-namespace/test-dir/
  hadoop fs -get jfs://your-namespace/test-dir/test.log . /

• Run a MapReduce job

  hadoop jar /usr/lib/hadoop-current/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar teragen -Dmapred.map.tasks=1000 10737418240 jfs://your-namespace/terasort/input
  hadoop jar /usr/lib/hadoop-current/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar terasort -Dmapred.reduce.tasks=1000 jfs://your-namespace/terasort/input jfs://your-namespace/terasort/output

• Run a Spark SQL test

  CREATE EXTERNAL TABLE IF NOT EXISTS src_jfs (key INT, value STRING) location 'jfs://your-namespace/Spark_sql_test/';

Control disk space usage

JindoFS uses OSS as the data storage backend, which allows you to store large volumes of data. However, the capacity of local disks is limited. JindoFS automatically deletes cold data in local disks. Alibaba Cloud uses the node.data-dirs.watermark.high.ratio and node.data-dirs.watermark.low.ratio parameters to adjust the space usage of local disks. The values of both parameters are in the range of 0 to 1 to indicate the percentage of space usage. JindoFS uses the total storage capacity of all data disks by default. The node.data-dirs.watermark.high.ratio parameter specifies the upper limit of space usage on each disk. Less frequently accessed data stored on a disk is released if the space used by JindoFS reaches the upper limit. The node.data-dirs.watermark.low.ratio parameter specifies the lower limit of space usage on each disk. After the space usage of a disk reaches the upper limit, less frequently accessed data is released until the space usage of the disk reaches the lower limit. You can configure the upper limit and lower limit to adjust and assign disk space to JindoFS. Make sure that the upper limit is greater than the lower limit.

Configure the storage policy

JindoFS provides multiple storage policies to meet different storage needs. The following table lists four available storage policies for a directory.

JindoFS provides a command-line tool Admin to configure the storage policy of a directory. The default storage policy is WARM. New files are stored based on the storage policy configured for the parent directory. Run the following command to configure the storage policy:

jindo dfsadmin -R -setStoragePolicy [path] [policy]

Run the following command to obtain the storage policy configured for a directory:

jindo dfsadmin -getStoragePolicy [path]

Use the Admin tool