Object Storage Service:How to use automatic storage tiering of OSS-HDFS

Usage notes

• You are charged data retrieval fees when you read IA, Archive, or Cold Archive objects in OSS-HDFS. We recommend that you do not store frequently accessed data as IA, Archive, or Cold Archive objects. For more information about the data retrieval fees, see Data processing fees.

• When you configure a storage policy for data in OSS-HDFS, you must add tags to data blocks. You are charged for the tags based on object tagging billing rules. For more information, see Object tagging fees.

• If the version of JindoSDK is earlier than 6.4.0, you cannot create an object in an IA, Archive, or Cold Archive directory. If you need to create an object in an IA, Archive, or Cold Archive directory. If you need to create an object in an IA, Archive, or Cold Archive directory, you can create an object and save it in a Standard directory. Then, move the object to the IA, Archive, or Cold Archive directory by using the rename operation.

  If you want to directly create an object in the IA, Archive, or Cold Archive directory, you must upgrade JindoSDK to 6.4.0 or later.

• When you convert the storage class of objects to Archive or Cold Archive, additional system overheads are generated and data restoration is slow. Proceed with caution.

• You can convert the storage class of Archive objects to Cold Archive, but you cannot convert the storage class of Cold Archive objects to Archive.

Procedure

1. Configure environment variables.

   1. Connect to an Elastic Compute Service (ECS) instance. For more information, see Connect to an instance.

   2. Go to the bin directory of the installed JindoSDK JAR package.

      cd jindosdk-x.x.x/bin/

      Note

      x.x.x indicates the version number of the JindoSDK JAR package.

   3. Grant read and write permissions to the jindo-util file in the bin directory.

      chmod 700 jindo-util

   4. Rename the jindo-util file to jindo.

      mv jindo-util jindo

   5. Create a configuration file named jindosdk.cfg, and then add the following parameters to the configuration file:

      [common] Retain the following default configurations: 
      logger.dir = /tmp/jindo-util/
      logger.sync = false
      logger.consolelogger = false
      logger.level = 0
      logger.verbose = 0
      logger.cleaner.enable = true
      hadoopConf.enable = false
      
      [jindosdk] Specify the following parameters: 
      <!-- In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint.  -->
      fs.oss.endpoint = cn-hangzhou.oss-dls.aliyuncs.com
      <! -- Configure the AccessKey ID and AccessKey secret that are used to access OSS-HDFS.  -->
      fs.oss.accessKeyId = LTAI********    
      fs.oss.accessKeySecret = KZo1********                                        

   6. Configure environment variables.

      export JINDOSDK_CONF_DIR=<JINDOSDK_CONF_DIR>

      Set <JINDOSDK_CONF_DIR> to the absolute path of the jindosdk.cfg configuration file.

2. Specify a storage policy for the data that is written to OSS-HDFS. The following table describes the storage policy.

   Scenario	Command	Result
   IA	./jindo fs -setStoragePolicy -path oss://examplebucket/dir1 -policy CLOUD_IA	Objects in the dir1/ directory contain a tag whose key is transition-storage-class and whose value is IA.
   Archive	./jindo fs -setStoragePolicy -path oss://examplebucket/dir2 -policy CLOUD_AR	Objects in the dir2/ directory contain a tag whose key is transition-storage-class and whose value is Archive.
   Cold Archive	./jindo fs -setStoragePolicy -path oss://examplebucket/dir3 -policy CLOUD_COLD_AR	Objects in the dir3/ directory contain a tag whose key is transition-storage-class and whose value is ColdArchive.

3. Enable the automatic storage tiering feature.

   1. Log on to the OSS console.

   2. In the left-side navigation pane, click Buckets. On the Buckets page, click the name of the bucket for which you want to enable the automatic storage tiering feature.

   3. In the left-side navigation tree, choose Data Lake > OSS-HDFS.

   4. On the OSS-HDFS tab, click Configure.

   5. In the Basic Settings section of the Automatic Storage Tiering panel, turn on Status.

      

      To prevent the automatic storage tiering feature from failing to run as expected due to incorrect configurations, OSS automatically creates a lifecycle rule to convert the storage class of data in OSS-HDFS that contains a specific tag:

      • The lifecycle rule specifies that the storage class of the data that contains a tag whose key is transition-storage-class and whose value is IA in the .dlsdata/ directory is converted to IA one day after the data is last modified.

      • The lifecycle rule specifies that the storage class of the data that contains a tag whose key is transition-storage-class and whose value is Archive in the .dlsdata/ directory is converted to Archive one day after the data is last modified.

      • The lifecycle rule specifies that the storage class of the data that contains a tag whose key is transition-storage-class and whose value is ColdArchive in the .dlsdata/ directory is changed to Cold Archive one day after the data is last modified.

      Important

      Do not modify the lifecycle rule that is automatically created after the automatic storage tiering feature is enabled. Otherwise, data or OSS-HDFS service exceptions may occur.

   6. Click OK.

      • OSS-HDFS converts the storage class of objects based on the storage policy configured in Step 2.

      • OSS loads a lifecycle rule within 24 hours after the rule is created. After the rule is loaded, OSS starts to execute the rule at 08:00 (UTC+8) every day. The specific execution time varies based on the number of objects. The objects are converted to the specified storage class within at least 48 hours.

Related commands

./jindo fs -setStoragePolicy -path <path> -policy <policy>

./jindo fs -getStoragePolicy -path <path>

./jindo fs -unsetStoragePolicy -path <path>

./jindo fs -checkStoragePolicy -path <path>

./jindofs fs -setStoragePolicy -path <path> -policy <policy> -restoreDays <restoreDays>

FAQ

• What do I do if I accidentally delete a lifecycle rule configured for OSS-HDFS data after the automatic storage tiering feature is enabled?

• Can I change the storage class of objects from Archive or Cold Archive to Standard or IA?

References

• Overview

• Storage fees