Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Who Said All Redis Data Must Be Stored in Memory? Explore Redis Hybrid Storage Instances

Who Said All Redis Data Must Be Stored in Memory? Explore Redis Hybrid Storage Instances

This article explores how the Redis hybrid storage instance breaks the convention that all Redis data must be stored in the memory.

By Zhang Youdong (Linqing)

The Redis hybrid storage instance is a cloud database product independently developed by Alibaba Cloud and is compatible with Redis protocols and features. The hybrid storage instance breaks the convention that all Redis data must be stored in the memory. Now, use disks to store full data and cache hot data to the memory for balancing access performance with storage costs.

Architecture and Features

1

Command Compatibility

Hybrid storage is compatible with the vast majority of Redis commands. Compared to the native Redis, the following commands are not supported or their functions are limited. These commands are not supported mainly for performance considerations. If your business does need to use these commands, submit a ticket to apply for them.

2

Selection Guide - Scenarios

3

Selection Guide - Specifications

While selecting hybrid storage instances, select an appropriate memory configuration and disk configuration. The disk determines the total amount of data that can be stored, and the memory determines the total amount of hot data that can be cached. When an instance is created, an appropriate CPU resource configuration is selected based on the storage specifications. Currently, it doesn't allow specifying the number of CPU cores.

For example, an instance with 64 GB memory and a 256 GB disk stores up to 256 GB of data (subject to the total size of the physical files of the KV storage engine), 64 GB of which can be cached in the memory.

Memory Selection Recommendation: To ensure maximum compatibility with the Redis native access protocol, Redis hybrid storage requires that all keys reside in the memory, while values are automatically stored in the memory or disk according to their access frequency. Therefore, the memory space must be sufficient to store all keys and the corresponding metadata.

4

Disk Selection Recommendation: Since Redis data is stored in the KV storage engine, each key will have additional metadata, increasing the required storage space. Therefore, reserve a proper margin while selecting the disk space. We recommend choosing a disk capacity of 1.2 to 1.5 times greater than the actual storage needs.

  • Case 1: User A uses a Redis cluster to store 100 GB of data. The total queries per second (QPS) is less than 20,000 and 80% of the data is rarely accessed. User A may use a hybrid storage instance with 32 GB memory and a 128 GB disk, saving nearly 70 GB of memory storage and reducing storage costs by over 50%.
  • Case 2: User B builds a user-created Pika or SSDB instance in the IDC to reduce the storage cost of Redis and store about 400 GB of data, about 10% of which is frequently accessed. However, the cluster O&M workload is heavy, so the user wants to migrate data to a cloud database. In this case, user B may use a hybrid storage instance with 64 GB memory and a 512 GB disk to achieve zero O&M without compromising service quality.

Performance Metrics

The performance of Redis hybrid storage is related to the memory-to-disk ratio and the service access level. According to the selected specification and service access mode, the performance of simple SET and GET methods may fluctuate between several thousand and tens of thousands. In the best case, all access requests hit the memory, and the performance is basically consistent with that of the Redis memory edition. In the worst case, all access requests need to be read from the disk.

Test Scenario: There are 20 million keys with values of 1 KB, and 25% of hot keys are stored in the memory. The data of the GET request test is as follows:

5

Scenarios

Live Streaming

Live streaming generates a large amount of hot data, most of which comes from popular live-streaming rooms. Use ApsaraDB for Redis hybrid storage instances to store data from popular live-streaming rooms in the memory and data from unpopular rooms in disks. In this way, make full use of the limited memory space.

E-commerce

E-commerce applications have a large volume of commodity data, but newly released products are the most frequently accessed, while older products are less frequently accessed. By using Redis hybrid storage instances, easily break through the memory capacity limit by storing a large amount of commodity data on disks. During normal business requests, active commodity data is gradually cached into the memory to meet business needs at the lowest possible cost.

Online Education

In online education scenarios, data such as courses, question banks, and conversation records between teachers and students are frequently accessed only for popular courses and the latest question banks. By using Redis hybrid storage, users store a large amount of course information on disks. Active course and question bank information is transferred to and stored in the memory. This ensures the access performance of frequently accessed data and balances performance and storage costs.

Other Scenarios

Other scenarios with significant access frequency variations and lower performance requirements may also use Redis hybrid storage to reduce storage costs.

Frequently Asked Question

The disk is not full, but the system reports an out of memory (OOM) error when the memory is full.

  • The memory size is too small to accommodate all keys and their metadata. We recommend upgrading the instance type in the console to increase its memory size.
  • The values corresponding to the keys are relatively small. Hybrid storage does not trigger swap-out to the disk for relatively small values (for example, less than 20 bytes). This is because the metadata is still stored in the memory when small values are swapped to the disk. Eventually, the memory space cannot be freed up by swapping out values. To address this problem, the hybrid storage kernel is continuously optimized to adapt to more application scenarios.

ApsaraDB for Redis

ApsaraDB for Redis is a stable and reliable database service with remarkable performance and support for auto-scaling. It is built on Apsara Distributed File System and full SSD high-performance storage and supports the primary-secondary and cluster-based high-availability architectures.

Redis Database Storage Data storage Cloud Database Redis Hybrid Storage Disk
0 0 0
Share on

Read previous post:

Get Started with MongoDB BI Connector in 4 Steps

Read next post:

5 Simple Steps to Migrate Data Warehouses from Redshift to Alibaba Cloud AnalyticDB for PG

ApsaraDB

445 posts | 94 followers

You may also like

Comments

ApsaraDB

445 posts | 94 followers

Related Products

More Posts by ApsaraDB

See All