Configure storage tiers for storage resources - MaxCompute

In terms of tiered storage, MaxCompute supports three storage tiers: standard storage, infrequent access (IA) storage, and long-term storage. By default, data is stored at the standard storage tier. You can specify IA storage or long-term storage as the storage tier for a specific table or partition based on the data access frequency. This way, cold and hot data are separately stored, and data storage costs are reduced.

Apply for the use of the feature

New projects can use the storage tiers feature by default, including the China (Hong Kong), Japan (Tokyo), Indonesia (Jakarta), Malaysia (Kuala Lumpur), and Germany (Frankfurt) regions.
If you want to use the tiered storage feature for existing projects and other regions, such as Singapore, click the link to apply for the use of the feature. For more information, see Apply for trial use of new features.

Storage tiers

Storage type	Description
Standard storage	The default storage type, which is suitable for scenarios where data is frequently accessed and read and write operations are frequently performed on the data.
IA storage	IA storage and long-term storage are suitable for data that is infrequently accessed. IA storage and long-term storage can significantly reduce storage costs. Note Platform for AI (PAI) and Hologres cannot directly access data at the IA or long-term storage tier. If a large amount of data is accessed or data access is frequent, the fees incurred may be higher than the storage fees of the data at the standard storage tier. The following content is for demonstration purposes: If you access all data in a table or partition at the IA storage tier once in a month, the sum of the data access fees and storage fees are the same as the storage fees of the data at the standard storage tier. If you access all data in a table or partition at the long-term storage tier once in six months, the sum of the data access fees and storage fees are the same as the storage fees of the data at the standard storage tier.
Long-term storage

Tiered storage pricing

Storage type	Billing formula	Price	Description
Standard storage	Daily storage fees = Average daily storage volume × Unit price	If the actual storage volume of a project is 1 GB or less than 1 GB, the data storage is free of charge. If the actual storage volume exceeds 1 GB, you are charged for the extra data volume based on the unit price 0.0006 USD/GB/day.	Average daily storage volume = Sum of the collected storage volume/24
Infrequent access (IA) storage	Daily storage fees = Average daily storage volume × Unit price Data access fees = Amount of accessed data × Unit price	Unit price for IA storage: 0.000412 USD/GB/day. Unit price for access to data of IA storage: 0.006191 USD/GB.	Average daily storage volume = Sum of the collected storage volume/24 If the storage type of a table or partition is changed from IA storage to standard storage or long-term storage, you are charged data access fees for IA storage of the entire table or partition. However, no data access fees are charged when IA storage is automatically converted to long-term storage through the tiered storage lifecycle. If the storage type of a table or partition is changed from long-term storage to standard storage or IA storage, you are charged data access fees for long-term storage of the entire table or partition.
Long-term storage	Daily storage fees = Average daily storage volume × Unit price Data access fees = Amount of accessed data × Unit price	Unit price for long-term storage: 0.00017 USD/GB/day. Unit price for access to data of long-term storage: 0.080793 USD/GB.

For more information about storage pricing, see Storage pricing.
You can log on to the Expenses and Costs console and choose Bills > Bill Details in the left-side navigation pane. On the Bill Details page, view the fees that are incurred for data storage and access at different storage tiers. The billable items that correspond to the standard storage tier, IA storage tier, and long-term storage tier are Storage, Tiered Storage-IA, and Tiered Storage-Long-term. For more information, see View billing details.

Precautions

PAI and Hologres cannot directly access data at the IA or long-term storage tier.
If the storage tier of a table or partition is set to IA or long-term storage, take note of the amount of data and the data access frequency. If a large amount of data is accessed or data access is frequent, the fees incurred may be higher than the storage fees of the data at the standard storage tier.
- If you access all data in a table or partition at the IA storage tier once in a month, the sum of the data access fees and storage fees are the same as the storage fees of the data at the standard storage tier.
- If you access all data in a table or partition at the long-term storage tier once in six months, the sum of the data access fees and storage fees are the same as the storage fees of the data at the standard storage tier.

Specify a storage type

Storage type can be mutually converted. The conversion does not negatively affect data access performance.

The following table shows the type conversion of storage:

Conversion process	Description
Standard storage -> IA storage	Supports both manual and automatic conversion, with no I/O access fees. The fields of the table or the partition, such as 'last data update time' and 'last data access time,' are not updated.
Standard storage -> Long-term storage
IA storage -> Long-term storage	Supports both manual and automatic conversion, manual conversion incurs I/O access fees, whereas automatic conversion does not. The fields of the table or the partition, such as 'last data update time' and 'last data access time,' are not updated.
IA storage -> Standard storage	Only supports manual conversion, which incurs I/O access fees. The fields of the table or the partition, such as 'last data update time' and 'last data access time,' are updated.
Long-term storage -> IA storage
Long-term storage -> Standard storage

Manual configuration

For non-partitioned tables or partitions, you can manually specify the IA or long-term storage tier. The setting takes effect immediately after manual configuration.

Syntax

ALTER TABLE <TABLE_NAME> [PARTITION(<PARTITION_SPEC>)]
        SET <TBLPROPERTIES|PARTITIONPROPERTIES>("storagetier"="standard|lowfrequency|longterm");

Parameter description

TABLE_NAME: the name of a table. This parameter is required regardless of whether you want to specify a storage tier for a table or a partition.
PARTITION_SPEC: required when you want to specify a storage tier for a partition.
TBLPROPERTIES|PARTITIONPROPERTIES: the properties of a table or partition for which you want to specify a storage tier.
- TBLPROPERTIES: the properties of a table.
- PARTITIONPROPERTIES: the properties of a partition.
storagetier: required. The storage tier. Valid values:
- standard: standard storage. You are charged only storage fees.
- lowfrequency: IA storage. You are charged storage fees and the fees incurred for access to data at the IA storage tier.
- longterm: long-term storage. You are charged storage fees and the fees incurred for access to data at the long-term storage tier.

Note

For partitioned tables, storage tiers can be specified only at the partition level and cannot be specified at the table level.

Examples

Example 1: Specify IA storage as the storage tier for a non-partitioned table.

ALTER TABLE tablename
        SET TBLPROPERTIES("storagetier"="lowfrequency");

View the StorageTier field in table properties to check the current storage tier.

-- View table properties.
DESC extended tablename;  

-- The following result is returned:
+-------------------------------------------------------------------+
| Owner:                    ALIYUN$mofan_****@test.aliyunid.com      |
| Project:                  mf_mc_****                                |
| TableComment:                                                     |
+-------------------------------------------------------------------+
| CreateTime:               2021-11-18 15:14:00                     |
| LastDDLTime:              2023-09-11 14:34:55                     |
| LastModifiedTime:         2023-09-13 15:02:28                     |
| LastAccessTime:           2023-09-14 10:50:57                     |
+-------------------------------------------------------------------+
| InternalTable: YES      | Size: 1923683131                        |
+-------------------------------------------------------------------+
| Native Columns:                                                   |
+-------------------------------------------------------------------+
| Field| Type| Label |ExtendedLabel| Nullable| DefaultValue|Comment |
+-------------------------------------------------------------------+
| empno    | bigint |       |               | true     | NULL  |    |
| ename    | string |       |               | true     | NULL  |    |
| job      | string |       |               | true     | NULL  |    |
| mgr      | bigint |       |               | true     | NULL  |    |
| hiredate | datetime |     |               | true     | NULL  |    |
| sal      | bigint |       |               | true     | NULL  |    |
| comm     | bigint |       |               | true     | NULL  |    |
| deptno   | bigint |       |               | true     | NULL  |    |
+-------------------------------------------------------------------+
| Extended Info:                                                    |
+-------------------------------------------------------------------+
| TableID:                  8e0cc78c81ab4ad7af30bff7a8e****         |
| IsArchived:               false                                   |
| PhysicalSize:             5771049393                              |
| FileNum:                  3                                       |
| StoredAs:                 AliOrc                                  |
| CompressionStrategy:      normal                                  |
| odps.timemachine.retention.days: 1                                |
| ColdStorageStatus:        N/A                                     |
| encryption_enable:        false                                   |
| StorageTier:              lowfrequency                            |
| StorageTierLastModifiedTime:  2023-09-11 14:34:55                 |
+-------------------------------------------------------------------+

Example 2: Specify IA storage as the storage tier for a partition in a partitioned table named bank_data_pt.

ALTER TABLE bank_data_pt  PARTITION (credit='yes') SET PARTITIONPROPERTIES ("storagetier" = 'lowfrequency');

View the StorageTier field in partition properties to check the current storage tier.

-- View partition properties.
DESC extended bank_data_pt PARTITION(credit='yes');  

-- The following result is returned:
+------------------------------------------------------------------------------------+
| PartitionSize: 0                                                                   |
+------------------------------------------------------------------------------------+
| CreateTime:               2024-05-10 10:28:16                                      |
| LastDDLTime:              2024-05-10 10:31:01                                      |
| LastModifiedTime:         2024-05-10 10:28:16                                      |
+------------------------------------------------------------------------------------+
| IsExstore:                false                                                    |
| IsArchived:               false                                                    |
| PhysicalSize:             0                                                        |
| FileNum:                  0                                                        |
| ColdStorageStatus:        N/A                                                      |
| StorageTier:              LowFrequency                                             |
| StorageTierLastModifiedTime:  2024-05-10 10:31:01                                  |
+------------------------------------------------------------------------------------+

Automatic configuration based on lifecycle rules

For projects or partitioned tables, you can define rules for the lifecycles of storage tiers. The system can trigger automatic conversion among storage tiers based on the lifecycle rules.

After you define storage tier lifecycle rules for a project, if all the non-partitioned tables or partitions in the project meet the rules, the system automatically performs storage tier conversion for the non-partitioned tables or partitions.
After you define storage tier lifecycle rules for a partitioned table, if all the partitions in the table meet the rules, the system automatically performs storage tier conversion for the partitions.

Usage notes

You cannot separately configure storage tier lifecycle rules for each partition or non-partitioned table.
The storage tier lifecycle rules for a partitioned table have a higher priority than the storage tier lifecycle rules for a project to which the partitioned table belongs.
If the lifecycle of a table or partition meets the rules for both the long-term and IA storage tiers, the storage tier of the table or partition is preferentially converted into long-term storage.
If the lifecycle of a table or partition meets the rule for IA storage, the storage tier of the table or partition is converted into IA storage. After that, if the lifecycle of the table or partition meets the rule for long-term storage, the storage tier of the table or partition is converted into long-term storage. For the conversion from IA storage to long-term storage, fees are incurred for access to data at the IA storage tier. For pricing details, see Tiered storage pricing in this topic.
The platform regularly scans the rules twice a day. Therefore, the conversion may not be immediately performed after the rules are met.

Syntax

Configure lifecycle rules at the project level
```
setproject odps.table.lifecycle.config=<lifecycle_config_json_string>;
```
You can also configure through the MaxCompute console:
1. Log on to the MaxCompute console. In the top navigation bar, select a region.
2. In the left-side navigation pane, choose Workspace > Projects, find the project you want to manage and click Manage in the Actions column.
3. In the Lifecycle Configuration section of the Parameter Configuration tab, configure the parameters for the Last Access Configuration Policy and the Last Modified Configuration Policy.
  - Last access configuration policy: corresponds to the DaysAfterLastAccessGreaterThan parameter.
  - Last modified configuration policy: corresponds to the DaysAfterLastModificationGreaterThan parameter.

Configure lifecycle rules at the partitioned table level

Configure rules during table creation

CREATE [EXTERNAL] TABLE [if NOT EXISTS] <table_name>
 [PRIMARY key (<pk_col_name>, <pk_col_name2>),(<col_name> <data_type> [NOT NULL] [DEFAULT <default_value>] [comment <col_comment>], ...)]
 partitioned BY (<col_name> <data_type> [comment <col_comment>], ...)
tblproperties ('lifecycle_config' = '<lifecycle_config_json_string>')
;

Configure rules by modifying table settings

ALTER TABLE <TABLE_NAME> SET TBLPROPERTIES ('lifecycle_config' = '<lifecycle_config_json_string>');

Parameter description

This section describes the main parameters. For more information about other parameters, see "Parameters" in Table operations.

lifecycle_config_json_string:

The following code shows the definition of a lifecycle rule at the project level.

{
  "TierToLowFrequency": {
    "DaysAfterLastModificationGreaterThan": <days>, // xx days after the last modification time
    "DaysAfterLastAccessGreaterThan": <days>, // xx days after the last access time
  },
  "TierToLongterm": {
    "DaysAfterLastModificationGreaterThan": <days>,
    "DaysAfterLastAccessGreaterThan": <days>
  }
  // Each condition is optional, and multiple conditions have the OR relationship.
}

The following code shows the definition of a lifecycle rule at the partitioned table level.

{
  \"TierToLowFrequency\": {
    \"DaysAfterLastModificationGreaterThan\": <days>, // xx days after the last modification time
    \"DaysAfterLastAccessGreaterThan\": <days>, // xx days after the last access time
  },
  \"TierToLongterm\": {
    \"DaysAfterLastModificationGreaterThan\": <days>,
    \"DaysAfterLastAccessGreaterThan\": <days>
  }
  // Each condition is optional, and multiple conditions have the OR relationship.
}

TierToLowFrequency: the IA storage tier.
TierToLongterm: the long-term storage tier.
DaysAfterLastModificationGreaterThan: Automatic conversion is triggered N days after the last modification time. N is specified by this parameter. This parameter corresponds to LastModifiedTime of the table or partition.
DaysAfterLastAccessGreaterThan: Automatic conversion is triggered N days after the last access time. N is specified by this parameter. If LastAccessTime is left empty for the table or partition, the following principles are applied:
- For tables or partitions that you created before October 1, 2023, 2023.10.01 00:00:00 in UTC+0 is considered as the last access time.
- For tables or partitions that you created on or after October 1, 2023, if no data is accessed, the table or partition creation time is considered as the last access time.

Examples

Example 1: Configure storage tier lifecycle rules at the project level.

setproject odps.table.lifecycle.config={"TierToLongterm":{"DaysAfterLastModificationGreaterThan":180},"TierToLowFrequency":{"DaysAfterLastModificationGreaterThan":120}};

Example 2: Remove the configurations of storage tier lifecycle rules at the project level.
```
setproject odps.table.lifecycle.config=;
```

Example 3: Configure storage tier lifecycle rules at the partitioned table level.

-- Configure rules during partitioned table creation.
CREATE TABLE lifecycle_part_t (key string) 
partitioned BY (ds  STRING)
tblproperties ('lifecycle_config' = '{\"TierToLowFrequency\": {\"DaysAfterLastModificationGreaterThan\": 2,\"DaysAfterLastAccessGreaterThan\": 2},\"TierToLongterm\": {\"DaysAfterLastModificationGreaterThan\": 4,\"DaysAfterLastTierModificationGreaterThan\": 7}}')
;
-- Configure rules by modifying table settings.
ALTER TABLE lifecycle_part_t SET tblproperties ('lifecycle_config'='{\"TierToLowFrequency\": {\"DaysAfterLastModificationGreaterThan\": 90,\"DaysAfterLastAccessGreaterThan\": 30},\"TierToLongterm\": {\"DaysAfterLastModificationGreaterThan\": 180,\"DaysAfterLastTierModificationGreaterThan\": 7}}');