E-MapReduce:Use JindoFS in E-MapReduce 3.22.0 or later

Overview

JindoFS provides the block storage mode and the cache mode.

JindoFS uses a heterogeneous multi-backup mechanism that leverages both local storage and OSS. The Storage Service uses OSS as the storage backend to ensure high data reliability and uses local storage for redundant backups to accelerate data reads. In addition, the local Namespace Service manages JindoFS metadata, providing metadata operation performance comparable to that of HDFS.

Prepare the environment

• Create a cluster

  Select E-MapReduce 3.22.0 or a later version, and select SmartData in the Optional Services section. For more information, see Create a cluster.

  
• Configure the cluster
  The JindoFS file system provided by SmartData uses OSS as its storage backend. Therefore, you must configure OSS-related parameters before you use JindoFS. You can configure the parameters in two ways. You can modify the Bigboot parameters after you create a cluster and then restart the SmartData service for the changes to take effect. Alternatively, you can add custom configurations during cluster creation. After the cluster is created, the related services start based on your custom parameters.

  • Initialize parameters after the cluster is created

    All JindoFS-related configurations are located in the Bigboot component.

    1. On the Service Configuration page, click the bigboot tab.

       
    2. Click Custom Configuration.
    Note

    • The configuration items in the red boxes are required.
    • JindoFS supports multiple namespaces. This topic uses a namespace named test as an example.

    Parameter	Description	Example
    jfs.namespaces	The namespaces that JindoFS supports. Separate multiple namespaces with commas (,).	test
    jfs.namespaces.test.uri	The storage backend of the test namespace.	oss://oss-bucket/oss-dir Note You can also set this parameter to a specific directory in an OSS bucket. The namespace then uses this directory as the root directory to read and write data.
    jfs.namespaces.test.mode	The storage mode for the test namespace.	block Note JindoFS supports the block and cache storage modes.
    jfs.namespaces.test.oss.access.key	The AccessKey ID for the OSS storage backend.	xxxx Note For better performance and stability, use an OSS bucket in the same region and under the same account as the storage backend. The E-MapReduce cluster can then access OSS without a password. You do not need to configure an AccessKey ID and secret.
    jfs.namespaces.test.oss.access.secret	The AccessKey secret for the OSS storage backend.

    After you complete the configuration, save and deploy it. Then, restart all components in the SmartData service. You can then start using JindoFS.

    
  • Add custom configurations when you create a cluster
    E-MapReduce clusters allow you to add custom configurations during creation. For example, to enable passwordless access to OSS in the same region, select Custom Software Settings as shown in the following figure. Configure the parameters for the test namespace as follows:

    [
    {
    "ServiceName":"BIGBOOT",
    "FileName":"bigboot",
    "ConfigKey":"jfs.namespaces","ConfigValue":"test"
    },{
    "ServiceName":"BIGBOOT",
    "FileName":"bigboot",
    "ConfigKey":"jfs.namespaces.test.uri",
    "ConfigValue":"oss://oss-bucket/oss-dir"
    },{
    "ServiceName":"BIGBOOT",
    "FileName":"bigboot",
    "ConfigKey":"jfs.namespaces.test.mode",
    "ConfigValue":"block"
    }
    ]

    

Use JindoFS

Using JindoFS is similar to using HDFS. It provides the jfs prefix. To use JindoFS, replace the hdfs prefix with jfs.

JindoFS supports most computing components on EMR clusters, such as Hadoop, Hive, Spark, Flink, Presto, and Impala.

Examples:

• Shell commands

  hadoop fs -ls jfs://your-namespace/ 
  hadoop fs -mkdir jfs://your-namespace/test-dir
  hadoop fs -put test.log jfs://your-namespace/test-dir/
  hadoop fs -get jfs://your-namespace/test-dir/test.log ./

• MapReduce job

  hadoop jar /usr/lib/hadoop-current/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar teragen -Dmapred.map.tasks=1000 10737418240 jfs://your-namespace/terasort/input
  hadoop jar /usr/lib/hadoop-current/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar terasort -Dmapred.reduce.tasks=1000 jfs://your-namespace/terasort/input jfs://your-namespace/terasort/output

• Spark SQL

  CREATE EXTERNAL TABLE IF NOT EXISTS src_jfs (key INT, value STRING) location 'jfs://your-namespace/Spark_sql_test/';

Control disk space usage

JindoFS uses OSS as the data storage backend, which lets you store large volumes of data. However, the capacity of local disks is limited. JindoFS automatically deletes cold data in local disks. Alibaba Cloud uses the node.data-dirs.watermark.high.ratio and node.data-dirs.watermark.low.ratio parameters to adjust the space usage of local disks. The values of both parameters are in the range of 0 to 1 to indicate the percentage of space usage. JindoFS uses the total storage capacity of all data disks by default. The node.data-dirs.watermark.high.ratio parameter specifies the upper limit of space usage on each disk. Less frequently accessed data stored on a disk is released if the space used by JindoFS reaches the upper limit. The node.data-dirs.watermark.low.ratio parameter specifies the lower limit of space usage on each disk. After the space usage of a disk reaches the upper limit, less frequently accessed data is released until the space usage of the disk reaches the lower limit. You can configure the upper limit and lower limit to adjust and assign disk space to JindoFS. Make sure that the upper limit is greater than the lower limit.

Storage policies

JindoFS provides a Storage Policy feature that offers flexible policies for different storage needs. You can set one of the following four storage policies for a directory:

JindoFS provides an Admin tool to set the storage policy for a directory. The default policy is WARM. New files inherit the storage policy of their parent directory. The command is used as follows:

jindo dfsadmin -R -setStoragePolicy [path] [policy]

Use the following command to retrieve the storage policy of a directory:

jindo dfsadmin -getStoragePolicy [path]

Admin tool