ApsaraDB RDS:Use the data archiving feature of general ESSDs for an ApsaraDB RDS for PostgreSQL instance

Feature description

RDS instances access OSS buckets by using tablespaces and external file systems. You can execute DDL statements to transfer data between ESSDs and OSS buckets. If you enable the data archiving feature for your RDS instance, a directory named /cold-data is generated, and the tablespace of the new directory is rds_oss. The /data directory is an existing directory. Data in the rds_oss tablespace is stored in an OSS bucket, and data in the pg_default tablespace is stored in ESSDs.

After you enable the data archiving feature for an RDS instance, you can execute the following DDL statements to transfer data between ESSDs and OSS buckets. The data transfer is suitable for standard tables and child partitioned tables. During the data transfer, the system needs to separately process table data and index data.

-- Archive data.
ALTER TABLE $tableName SET tablespace rds_oss;
ALTER INDEX $indexName SET tablespace rds_oss;
-- Retrieve data.
ALTER TABLE $tableName SET tablespace pg_default;
ALTER INDEX $indexName SET tablespace pg_default;

Benefits

After you enable the data archiving feature for an RDS instance, you can store data on your RDS instance in OSS buckets at low costs based on the mass storage, security, cost-effectiveness, and high reliability of OSS. The data archiving feature allows you to access data that is infrequently accessed. After you enable the data archiving feature, the storage cost is 80% lower the storage cost when you use performance level 1 (PL1) ESSDs.

Applicable scope

• The RDS instance runs PostgreSQL.

• The RDS instance runs PostgreSQL 12 or later.

• The RDS instance runs a minor engine version of 20240530 or later. For more information about how to update a minor engine version, see Update the minor engine version.

  Note

  If your RDS instance runs PostgreSQL 15 and you want to use the data archiving feature for the RDS instance, make sure that the minor engine version of the RDS instance is 20240229 or later.

• The RDS instance runs RDS Basic Edition or RDS High-availability Edition.

• The RDS instance uses the standard or YiTian produce type.

• The RDS instance uses general ESSDs.

• The RDS instance uses the subscription or pay-as-you-go billing method.

• The number of CPU cores of the RDS instance is greater than or equal to 8.

• The RDS instance does not reside in the following regions: South Korea (Seoul), Australia (Sydney), UAE (Dubai), or UK (London).

Limits

• After you enable the data archiving feature for an RDS instance, specific features of the RDS instance, such as backup, are affected.

  • Single-digit second backups are not supported.

  • Cross-region backups are not supported.

  • When you restore individual databases and tables, you can restore only the data that is not archived.

  • When you download a backup set, the downloaded backup set contains only data that is not archived.

  Note

  If you enable the data archiving feature, the speed of instance backup or restoration decreases because the system needs to restore the archived data.

• Before the system archives table data and index data to OSS buckets, you must make sure that the data cleanup operation is performed. The storage performance of OSS buckets is significantly different from that of ESSDs, and the efficiency of autovacuum operations on the tablespaces in OSS is low. To avoid negative impacts on the RDS instance due to long-term use of autovacuum, we recommend that all_visible and all_frozen flags be added for all pages transferred to archived tables. This avoids autovacuum operations on archived tablespaces. To clean up archived data, we recommend that you use the following methods:

  VACUUM (FREEZE) table_name;

  You can enable the display of cleanup process details by using the VERBOSE option. Then, you can implement a complete cleanup based on the displayed details.

  • If the RDS instance contains long-running transactions and replication slots, you may need to retain specific transaction IDs because complete cleanups cannot be implemented.

  • You can use different cleanup policies for RDS instances that run different database engine versions. To implement a complete cleanup, you can query the displayed details and specify the required parameters. Examples:

    VACUUM (FREEZE, INDEX_CLEANUP on) table_name;

    • If the RDS instance runs PostgreSQL 14 or later, we recommend that you configure the PROCESS_TOAST parameter to clean up TOAST tables.

    • If the RDS instance runs PostgreSQL 16 or later, we recommend that you configure the PROCESS_MAIN parameter to clean up the primary table.

  For more information about VACUUM, see VACUUM.

Usage notes

• You must execute the VACUUM statement on the required table before you archive the table.

• Due to the access latency of OSS, the access latency of cold tables in OSS buckets reaches 10 ms, which is higher than the access latency of ESSDs.

  Note

  If tables archived in OSS buckets can be accessed in a quick manner, the tables may be cached in the buffer pool of the RDS instance or the page cache of the operating system.

• Reading data from OSS buckets consumes memory resources. In addition, accessing data in OSS buckets consumes a specific percentage of network bandwidth resources of Elastic Compute Service (ECS). You must change the speed and frequency of data transfer and data access.

Billing rules

Free of charge during the public preview

During the public preview, which starts from June 2, 2024 to July 25, 2024, you can use the feature free of charge. After the public preview ends, you are charged based on the volume of data that is archived in OSS buckets.

Enable and disable the data archiving feature

Enable the data archiving feature

• When you create an RDS instance, you can enable the data archiving feature on the ApsaraDB RDS buy page.

• If you want to enable the data archiving feature for an existing RDS instance that uses general ESSDs, enable the feature on the instance details page.

Disable the data archiving feature

1. Go to the Instances page. In the top navigation bar, select the region in which the RDS instance resides. Then, find the RDS instance and click the ID of the instance.

2. On the instance details page, click Configure General ESSD in the Basic Information section. In the Data Archiving card of the dialog box that appears, turn off Data Archiving.

   Important

   • When you disable the data archiving feature, a transient connection that lasts for approximately 30 seconds occurs.

   • If you want to disable the data archiving feature, make sure that the OSS bucket does not contain archived tables. If the OSS bucket contains an archived table, you cannot disable the feature. In this case, you must convert the archived table into an online table or perform the DROP operation on the archived table before you disable the data archiving feature.

View archived data

1. Go to the Instances page. In the top navigation bar, select the region in which the RDS instance resides. Then, find the RDS instance and click the ID of the instance.

2. In the left-side navigation pane, click Cold Storage Management.

3. In the Archive Layer Settings section of the page that appears, view the databases for which the data archiving feature is enabled.