All Products
Search

This Product

    Object Storage Service:Enable OSS-HDFS

    Document Center

    Object Storage Service:Enable OSS-HDFS

    Last Updated:Aug 28, 2024

    You can enable OSS-HDFS for an existing bucket or when you create a bucket.

    Prerequisites

    OSS-HDFS is supported in the following regions: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Ulanqab), China (Shenzhen), China (Guangzhou), China (Zhangjiakou), China (Hong Kong), Japan (Tokyo), Singapore, Germany (Frankfurt), US (Silicon Valley), US (Virginia), Indonesia (Jakarta), and Thailand (Bangkok).

    Billing rules

    • Metadata management fees

      You are charged metadata management fees for objects when you use OSS-HDFS. However, you are not charged for this billable item.

    • Data storage fees

      When you use OSS-HDFS, data blocks are stored in Objects Storage Service (OSS). Therefore, the billing method of OSS applies to data blocks in OSS-HDFS. For more information, see Billing overview.

    Limits

    • You cannot disable OSS-HDFS after you enable OSS-HDFS. Exercise caution when you enable OSS-HDFS.

    • You can access OSS-HDFS only from a virtual private cloud (VPC). The VPC used to access OSS-HDFS must be located in the same region in which the bucket for which you want to enable OSS-HDFS is located.

    • OSS-HDFS cannot be enabled for Archive, Cold Archive, and Deep Cold Archive buckets.

    • You cannot use methods that are not supported by OSS-HDFS to perform write operations on the .dlsdata/ directory in which OSS-HDFS data is stored and on the objects in the directory. For example, you cannot use methods that are not supported by OSS-HDFS to rename or delete the directory or delete the objects in the directory. Otherwise, OSS-HDFS does not work as expected and data may be lost.

    • Before you can delete a bucket for which OSS-HDFS is enabled, you must delete the metadata of objects that are stored in the bucket from OSS-HDFS and then delete the objects in the bucket. Otherwise, you cannot delete the bucket.

    Methods to enable OSS-HDFS

    Warning

    When you enable OSS-HDFS for a bucket, the AliyunOSSDlsDefaultRole role is automatically created and the AliyunOSSDlsRolePolicy policy is attached to the role. To maintain access to the .dlsdata/ directory and objects in the directory, do not disable, modify, or delete the role and the attached policy.

    Enable OSS-HDFS for a bucket when you create the bucket

    The first time you create a bucket by using an Alibaba Cloud account or a RAM user with administrator permissions, you must follow the on-page instructions to grant the role the required permissions to access OSS-HDFS before you enable OSS-HDFS. For more information, see Create a bucket.

    Enable OSS-HDFS for an existing bucket

    To use an Alibaba Cloud account or a RAM user that has administrator permissions to enable OSS-HDFS for an existing bucket, perform the following steps:

    1. Grant the RAM user the required permissions to access OSS-HDFS.

      1. Log on to the OSS console.

      2. In the left-side navigation pane, click Buckets. On the Buckets page, click the name of the bucket for which you want to enable OSS-HDFS.

      3. In the left-side navigation tree, choose Data Lake > OSS-HDFS.

      4. On the OSS-HDFS tab, click Authorize. Then, follow the on-screen instructions to grant the RAM user the permissions to access OSS-HDFS. HDFS1

    2. Enable OSS-HDFS

      1. On the OSS-HDFS page, click Enable OSS HDFS. 2

      2. In the message that appears, click OK.

        After you enable OSS-HDFS, you can still use OSS. If you want to use OSS-HDFS, you must use the OSS-HDFS endpoint. To obtain the OSS-HDFS endpoint, view the Port section of the Overview page of the bucket. Example: cn-hangzhou.oss-dls.aliyuncs.com.

    What to do next

    After you enable OSS-HDFS, you can grant permissions to a RAM user to connect EMR or non-EMR clusters to OSS-HDFS. For more information, see Grant permissions to a RAM user to access OSS-HDFS.