What's New in Redis 4.0?

As one of the most popular NoSQL databases, Redis earned its name by offering outstanding performance and extensive data structure support. Besides, it never stops evolving and, ever since its birth, it has been continuously launching new features.

Redis 4.0, the latest GA version in the community, has been released for almost a year, and Tair (Redis® OSS-Compatible) 4.0 has been released for half a year. There are a series of articles that analyze Redis 4.0's new features from the perspective of source code implementation. This article will discuss how to quickly understand and use benefits of Redis 4.0 from the perspective of users.

Lazyfree Feature

Many of you must have encountered the problem of deleting large keys. In addition to strings, Redis also supports list, set, hash, sorted set, and other complex data structures. These data structures have enriched Redis usage. However, improper usage may cause problems of over-sized single keys.

To better understand the problem, let's look at a simple example. Let's assume that there is an influential social networking account, "Mr. V", who has millions of subscribers. We can use the set type data structure to store Mr. V's fans' IDs. The key that stores the fans set is called funs; let's take a look at the number of the subscribers:

127.0.0.1:6379> SCARD funs
(integer) 6320505

Mr. V has over 6 million fans! Unfortunately, Mr. V canceled his account one day and we need to delete his information. Let's use the DEL command to delete this key.

127.0.0.1:6379> DEL funs
(integer) 1
(3.11s)
127.0.0.1:6379> slowlog get
1) 1) (integer) 4
   2) (integer) 1528169923
   3) (integer) 3104812
   4) 1) "DEL"
      2) "funs"
   5) "127.0.0.1:48398"
   6) ""

Redis 4.0 expands slowlog's return results and displays the IP:Port of clients that generated slow logs for convenient tracing.

We can see that the deletion operation took 3 seconds. This means that Redis was unable to execute other commands within these 3 seconds. This may cause adverse effect for online businesses. How can we avoid the blockage caused by deleting large keys? Redis 4.0 launched the Lazyfree function. Lazyfree uses the UNLINK command to delete large keys. The main thread is only responsible for removing the key from the database and the actual release operation is done by BIO backend threads. Let's take a look at the result:

127.0.0.1:6379> UNLINK funs
(integer) 1
(3.11s)
127.0.0.1:6379> slowlog get
(empty list or set)

We can see that the UNLINK execution was very fast and no slowlog was generated.

Lazyfree involves three commands:

UNLINK: asynchronously deletes the key.
FLUSHDB ASYNC: asynchronously clears the current DB.
FLUSHALL ASYNC: asynchronously clears all DBs.

And 4 configuration items:

lazyfree-lazy-expire: asynchronously deletes expired keys.
lazyfree-lazy-eviction: asynchronously evicts keys.
lazyfree-lazy-server-del: uses the asynchronous deletion function in the case of implicit deletion. For example, rename b, if b exists, delete b.
slave-lazy-flush: in the case of full data synchronization, the slave asynchronously clears all DBs.

Lua Script Supports Random Operations

Redis is built with a Lua environment to allow users to expand features. However, for the sake of consistency, the scripts must be pure functions. In other words, when a Lua script is given the same parameters, the same results will be returned for repeated executions.

Why? Because Redis not only is a standalone memory database, Redis also supports Master/slave replication and persistence. An executed Lua script will be replicated to the slave and be persisted to disk. If the results for repeated executions are not the same, then data in memory, disk, and slave will be inconsistent. This will cause data disorder after failover and restart and cause adverse effect to the business.

Here is an actual example. Assume that we have a Lua script to simply record the current time:

local now = redis.call('time')[1]

redis.call('set','now',now)

return redis.call('get','now')

It uses Redis' TIME command to obtain the timestamp, and store the timestamp to a key named now. However, an exception was thrown when executing this script:

$redis-cli --eval escript
(error) ERR Error running script (call to f_cfba5ec6a699dad183456f19d1099d8dabfdb80c): 
@user_script:3: @user_script: 3: Write commands not allowed after non deterministic commands. 
Call redis.replicate_commands() at the start of your script in order to switch to single commands replication mode.

The error message is simple: Write commands not allowed after non deterministic commands. Because TIME is random, Redis does not allow executing the write command, in order to ensure data consistency. How can we achieve the random write operation? The answer is provided in the above error message: call redis.replicate_commands(). After executing redis.replicate_commands(), Redis does not synchronize the complete Lua script to Slave or persist it to disk. Instead, it directly replicates the write command to call Redis from the script and then the same results will be returned at the slave and after persistence.

The script is modified as follows:

redis.replicate_commands()

local now = redis.call('time')[1]

redis.call('set','now',now)

return redis.call('get','now')

Execute it again to achieve random write:

$redis-cli --eval escript
"1528191578"

$redis-cli --eval escript
"1528191804"

LFU-based Hotspot Key Discovery Mechanism

LFU is a memory eviction policy newly introduced in Redis 4.0. LFU provides a more precise memory eviction algorithm. Its core function is to record the key access frequency within a period of time. It brings an additional benefit - hotspot key discovery.

Simply put, LFU uses 0-255 to indicate the access frequency of a key. A larger value indicates a higher frequency. The frequency increase is recorded in a logarithm-based manner. When LFU is 255, it indicates the key has been accessed by 1 million times.

We can use the OBJECT FREQ command to obtain a specified key's access frequency. But before that, we must set the memory eviction policy to allkeys-lfu or volatile-lfu:

127.0.0.1:6379> config get maxmemory-policy
1) "maxmemory-policy"
2) "noeviction"
127.0.0.1:6379> object freq counter:000000006889
(error) ERR An LFU maxmemory policy is not selected, access frequency not tracked. Please note that when switching between policies at runtime LRU and LFU data will take some time to adjust.

127.0.0.1:6379> config set maxmemory-policy allkeys-lfu
OK
127.0.0.1:6379> object freq counter:000000006889
(integer) 3

Use the SCAN command to traverse all keys, use OBJECT FREQ to retrieve the access frequency, and then sort the results to obtain the hotspot key. To ease user usage, Redis' built-in client redis-cli also provides the hotspot discovery function. You can simply append --hotkeys to the end of redis-cli and then execute it:

$./redis-cli --hotkeys

# Scanning the entire keyspace to find hot keys as well as
# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).

[00.00%] Hot key 'counter:000000000002' found so far with counter 87
[00.00%] Hot key 'key:000000000001' found so far with counter 254
[00.00%] Hot key 'mylist' found so far with counter 107
[00.00%] Hot key 'key:000000000000' found so far with counter 254
[45.45%] Hot key 'counter:000000000001' found so far with counter 87
[45.45%] Hot key 'key:000000000002' found so far with counter 254
[45.45%] Hot key 'myset' found so far with counter 64
[45.45%] Hot key 'counter:000000000000' found so far with counter 93

-------- summary -------

Sampled 22 keys in the keyspace!
hot key found with counter: 254    keyname: key:000000000001
hot key found with counter: 254    keyname: key:000000000000
hot key found with counter: 254    keyname: key:000000000002
hot key found with counter: 107    keyname: mylist
hot key found with counter: 93    keyname: counter:000000000000
hot key found with counter: 87    keyname: counter:000000000002
hot key found with counter: 87    keyname: counter:000000000001
hot key found with counter: 64    keyname: myset

MEMORY Analysis Command

Memory analysis helps optimize Redis usage. The completely new MEMORY command can help users perform this operation. The MEMORY command has 5 sub-commands. These 5 sub-commands can be viewed from MEMORY HELP.

127.0.0.1:6379> memory help
1) "MEMORY DOCTOR                        - Outputs memory problems report"
2) "MEMORY USAGE <key> [SAMPLES <count>] - Estimate memory usage of key"
3) "MEMORY STATS                         - Show memory usage details"
4) "MEMORY PURGE                         - Ask the allocator to release memory"
5) "MEMORY MALLOC-STATS                  - Show allocator internal stats"

You can learn more about theses sub-commands on Redis's official documentation.

Summary

We looked at some of the key features of Redis 4.0, focusing on features like Lazytree, Lua, and Memory. To learn more about Redis®* on Alibaba Cloud, visit the Tair (Redis® OSS-Compatible) product page.

*Redis is a registered trademark of Redis Ltd. Any rights therein are reserved to Redis Ltd. Any use by Alibaba Cloud is for referential purposes only and does not indicate any sponsorship, endorsement or affiliation between Redis and Alibaba Cloud.

Community

What's New in Redis 4.0?

Lazyfree Feature

Lua Script Supports Random Operations

LFU-based Hotspot Key Discovery Mechanism

MEMORY Analysis Command

Summary

Read previous post:

Read next post:

ApsaraDB

You may also like

Comments

ApsaraDB

Related Products

Tair

Tair (Redis® OSS-Compatible)

ApsaraDB for HBase

Database Security Solutions