Configure IA storage and Archive storage to reduce log storage costs - Simple Log Service

Simple Log Service provides the intelligent tiered storage feature. You can use the feature to store data in the hot, Infrequent Access (IA), or Archive storage tier based on your business requirements. This helps reduce the costs of long-term storage and ensure that log-related capabilities are not affected. The capabilities include query, analysis, visualization, alerting, shipping, and transformation. This topic describes the lifecycle management of data storage supported by Simple Log Service.

Introduction to intelligent tiered storage

Storage types

Storage type	Description
Hot storage	Hot storage is a scalable and highly available data storage solution. You can store data that is frequently accessed in the hot storage tier. You can access data in real time and query and analyze data with high performance. Hot storage is suitable for scenarios that involve high-frequency data query and analysis.
IA storage	IA storage, formerly known as cold storage, can be used to reduce the costs of long-term storage and ensure that log-related capabilities are not affected. The capabilities include query, analysis, visualization, alerting, shipping, and transformation. IA storage is suitable for scenarios that involve low-frequency data query and analysis and problem backtracking.
Archive storage	Archive storage is a long-term data storage solution that allows you to query and analyze data at a lower cost compared with hot storage and IA storage. Archive storage is suitable for scenarios that involve long-term storage of data for audit purposes.

Data retention periods

Important

The data retention period of a Logstore is the sum of the data retention periods for the hot storage tier, IA storage tier, and Archive storage tier.
If you set the data retention period of a Logstore to 3,650 days, data is permanently stored in the Logstore. The data in the Logstore continues to be stored in the Archive storage tier even after the 3,650 days elapse.

Data retention period less than 3,650 days
Scenario 1: The data retention period of a Logstore is 90 days. The data retention periods for the hot storage tier and Archive storage tier are 30 days and 60 days, respectively. Data in the hot storage tier is automatically moved to the Archive storage tier when the data retention period for the hot storage tier elapses. Data in the Archive storage tier is automatically deleted when the data retention period for the Archive storage tier elapses.
Scenario 2: The data retention period of a Logstore is 97 days. The data retention periods for the hot storage tier, IA storage tier, and Archive storage tier are 7 days, 30 days, and 60 days, respectively. Data in the hot storage tier is automatically moved to the IA storage tier when the data retention period for the hot storage tier elapses. Data in the IA storage tier is automatically moved to the Archive storage tier when the data retention period for the IA storage tier elapses. Data in the Archive storage tier is automatically deleted when the data retention period for the Archive storage tier elapses.
Data retention period of 3,650 days or more
Scenario 3: The data retention period of a Logstore is 3,650 days. The data retention periods for the hot storage tier and Archive storage tier are 30 days and 3,620 days, respectively. Data in the hot storage tier is automatically moved to the Archive storage tier when the data retention period for the hot storage tier elapses. The data retention period of 3,650 days indicates that data is permanently stored in the Logstore. In this case, the data that is stored in the Archive storage tier for 3,620 days continues to be stored in the Archive storage tier even after the data retention period elapses.
Scenario 4: The data retention period of a Logstore is 3,650 days. The data retention periods for the hot storage tier, IA storage tier, and Archive storage tier are 7 days, 30 days, and 3,613 days, respectively. The data retention period of 3,650 days indicates that data is permanently stored in the Logstore. In this case, the data that is stored in the Archive storage tier for 3,613 days continues to be stored in the Archive storage tier even after the data retention period elapses.

Storage type comparison

Item		Hot storage	IA storage	Archive storage
Scenario		High-frequency and high-performance query and analysis	Problem backtracking	Data audit
Fee		USD 0.002875 per GB-day	USD 0.000762 per GB-day	USD 0.000259 per GB-day
Performance	Latency (billions of data records)	Ten to hundred milliseconds	Hundred milliseconds to seconds	Minutes
Performance	Concurrency (projects)	Concurrent query requests: 100 Concurrent analysis requests: 2	Concurrent query requests: 10 Concurrent analysis requests: 2	Concurrent query requests: 1 Concurrent analysis requests: 1
Advantage		High-concurrency query and analysis	Low-frequency query and analysis and high cost-effectiveness	Occasional query and analysis and low costs
Advantage		Data is automatically moved across storage tiers based on data lifecycle management. You can use intelligent tiered storage after simple configuration. You do not need to write scripts or manually move data. Data that is stored in the IA or Archive storage tier can be accessed in real time. You do not need to manually retrieve data or modify applications. You are not charged for data retrieval.
Limit on the data retention period	Pay-by-feature	Data in the hot storage tier must be stored for at least 7 days before it can be moved to the IA storage tier. Data in the hot storage tier must be stored for at least 30 days before it can be moved to the Archive storage tier.	Data in the IA storage tier must be stored for at least 30 days before it can be moved to the Archive storage tier.	The minimum data retention period for the Archive storage tier is 60 days.
Limit on the data retention period	Pay-by-ingested-data	You are not charged for data storage in the hot storage tier for 30 days.

Data movement across storage tiers

Data that is stored in the hot storage tier, IA storage tier, and Archive storage tier can be moved across storage tiers. The following table describes such movement.

No.	Description
1	Data in the hot storage tier must be stored for at least 7 days before it can be moved to the IA storage tier.
2	Data in the hot storage tier must be stored for at least 30 days before it can be moved to the Archive storage tier.
3	Data in the IA storage tier must be stored for at least 30 days before it can be moved to the Archive storage tier.
4	You can change the data retention period for the hot storage tier to move data in the IA storage tier to the hot storage tier.
5	You can change the data retention period for the IA storage tier to move data in the Archive storage tier to the IA storage tier.
6	You can change the data retention period for the hot storage tier to move data in the Archive storage tier to the hot storage tier.

Manage storage tiers

Console

Log on to the Simple Log Service console.
In the Projects section, click the project that you want to manage.
On the Log Storage > Logstores tab, click the icon of the Logstore that you want to manage and click Modify.

On the Logstore Attributes page, click Modify. Turn on Intelligent Tiered Storage. Then, configure the related parameters based on the following figure and click Save.

The following table describes the parameters that are used to enable the intelligent tiered storage feature. For more information about other parameters, see Create a Logstore.

Parameter	Description
Data Retention Period	Specify the data retention period of the Logstore. For more information, see Basic resources. You can set the Data Retention Period parameter to Specified Days or Permanent Storage. The data retention period of a Logstore is the sum of the data retention periods for the hot storage tier, IA storage tier, and Archive storage tier. Important If you shorten the data retention period of a Logstore, Simple Log Service deletes all expired data in approximately 1 hour. For example, if you change the value of the Data Retention Period parameter from 5 to 1, Simple Log Service deletes the data of the previous four days in approximately 1 hour.
Intelligent Tiered Storage	Turn on Intelligent Tiered Storage to enable the intelligent tiered storage feature.
Storage Policy	Specify the data retention period for each tier of the Logstore.
Data retention period for the hot storage tier	Specify the data retention period for the hot storage tier. If you want to move data from the hot storage tier to the IA storage tier, you can set the data retention period for the hot storage tier to a value within the range of 7 to 3650. Unit: days. If you want to move data from the hot storage tier to the Archive storage tier, you can set the data retention period for the hot storage tier to a value within the range of 30 to 3650. Unit: days. Important After you change the data retention period for the hot storage tier, Simple Log Service moves data based on the new setting in approximately 1 hour. For example, if you change the data retention period for the hot storage tier from 30 days to 40 days, Simple Log Service starts to move data that is stored in the IA or Archive storage tier to the hot storage tier in approximately 1 hour. The data retention period for the hot storage tier must be less than the value of the Data Retention Period parameter.
Data retention period for the IA storage tier	Specify the data retention period for the IA storage tier. If you want to move data from the IA storage tier to the Archive storage tier, you can set the data retention period for the IA storage tier to a value within the range of 30 to 3650. Unit: days. Important After you change the data retention period for the IA storage tier, Simple Log Service moves data based on the new setting in approximately 1 hour. For example, if you change the data retention period for the IA storage tier from 30 days to 40 days, Simple Log Service starts to move data that is stored in the Archive storage tier to the IA storage tier in approximately 1 hour. The data retention period for the IA storage tier must be less than the value of the Data Retention Period parameter.
Data retention period for the Archive storage tier	Specify the data retention period for the Archive storage tier. Valid values: 60 to 3650. Unit: days. When the data retention period for the Archive storage tier elapses, data in the Archive storage tier is automatically deleted. Important The data retention period for the Archive storage tier must be less than the value of the Data Retention Period parameter.

API

When you call the CreateLogStore operation to create a Logstore, you can configure the ttl, hot_ttl, and infrequentAccessTTL parameters to specify the retention policy of tiered storage. The ttl, hot_ttl, and infrequentAccessTTL parameters specify the data retention period of the Logstore, data retention period of the hot storage tier, and data retention period of the IA storage tier, respectively. For more information about how to create a Logstore, see CreateLogStore.
For an existing Logstore, you can call the UpdateLogStore operation to modify a Logstore and modify the ttl, hot_ttl, and infrequentAccessTTL parameters to dynamically modify the retention policy of tiered storage. This way, your requirements for data retention and cost control are met. For more information about how to modify a Logstore, see UpdateLogStore.

References

For more information about how to manage a Logstore, see Manage a Logstore.
You are charged for data storage in different tiers based on storage usage. You are not charged for data movement. For more information, see Billing overview.
In pay-by-ingested-data billing mode, you are not charged for data storage within 30 days after the data is written to Simple Log Service. If intelligent tiered storage is enabled, you are charged based on the data retention periods and storage tiers. For more information, see Billing examples.
For more information about the pay-by-feature and pay-by-ingested-data billing modes, see Pay-as-you-go.
Simple Log Service supports locally redundant storage (LRS) and zone-redundant storage (ZRS). For more information, see Storage redundancy.