All Products
Search

This Product

    E-MapReduce:Use the password-free feature of JindoFS SDK

    Document Center

    E-MapReduce:Use the password-free feature of JindoFS SDK

    Last Updated:Nov 14, 2024

    This topic describes how to access JindoFileSystem (JindoFS) in password-free mode from an ECS instance (not an instance in EMR clusters) when you use JindoFS SDK.

    Prerequisites

    1. You have ECS instances other than those for EMR. 2. The Hadoop ecosystem is used. 3. JindoFS SDK for Java is obtained.

    Background information

    Before you can use JindoFS SDK, you must remove packages related to Jindo from your environment, such as jboot.jar and smartdata-aliyun-jfs-*.jar. To use Spark, you must also remove the packages in the /opt/apps/spark-current/jars/ directory.

    Step 1: Create an instance RAM role

    1. Log on to the RAM console with an Alibaba Cloud account.

    2. In the left-side navigation pane, choose Identities > Roles.

    3. On the RAM Roles page, click Create Role. In the Create RAM Role panel, select Alibaba Cloud Service for Trusted entity type.

    4. Click Next.

    5. Enter a role name in the RAM Role Name field and select Elastic Compute Service from the Select Trusted Service drop-down list.

    6. Click OK.

    Step 2: Grant permissions to the RAM role

    1. Log on to the RAM console with an Alibaba Cloud account.

    2. Optional. If you do not use system permissions, you can create custom permissions. For more information, see the "(Optional) Create a custom authorization policy" section in Implement access control by using RAM.

    3. In the left-side navigation pane, choose Identities > Roles.

    4. Find the created RAM role and click Role Name.

    5. Click Precise Permission.

    6. In the panel that appears, select System Policy or Custom Policy for Type.

      For example, AliyunOSSReadOnlyAccess. Please select the appropriate policy based on your specific business needs.

    7. Enter the policy name.

    8. Click OK.

    Step 3: Bind the RAM role to an ECS instance

    1. Log on to the ECS console.

    2. In the left-side navigation pane, choose Instances & Images > Instances.

    3. In the top navigation bar, select a region.

    4. Find the ECS instance to which you want to bind the RAM role and choose image > Instance Settings > Attach/Detach RAM Role in the Actions column.

      image

    5. In the Bind/Unbind RAM Role dialog box, select the RAM role and click OK.

    Step 4: Configure environment variables on ECS

    Run one of the following commands to configure environment variables on ECS:

    export CLASSPATH=/xx/xx/jindofs-2.5.0-sdk.jar

    and

    HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/xx/xx/jindofs-2.5.0-sdk.jar

    Step 5: Access JindoFS in password-free mode

    1. Use Shell to access OSS.

      hdfs dfs -ls/-mkdir/-put/....... oss://<ossPath>

    2. Access OSS by using the FileSystem interface of Hadoop.

      Example: 

      import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.LocatedFileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.RemoteIterator;

import java.net.URI;

public class test {
    public static void main(String[] args) throws Exception {
        FileSystem fs = FileSystem.get(new URI("ossPath"), new Configuration());
        RemoteIterator<LocatedFileStatus> iterator = fs.listFiles(new Path("ossPath"), false);
        while (iterator.hasNext()){
            LocatedFileStatus fileStatus = iterator.next();
            Path fullPath = fileStatus.getPath();
            System.out.println(fullPath);
        }
    }
}