Containerized Redis clusters are easy to deploy and scale and store resources in isolated space. This improves your efficiency in deploying, managing, scaling up, and maintaining Redis clusters and reduces O&M costs of Redis clusters. This topic describes how to build a Redis cluster on an elastic container instance in a Container Service for Kunbernetes (ACK) Serverless cluster.
Background information
Redis instances can run in cluster mode. In this mode, Redis divides the storage space into 16,384 hash slots. Each master node in the Redis cluster is responsible for a database shard, which consists of a specific number of hash slots. When you write a data entry, Redis calculates hash slots for the data entry and then writes the data to the node that is responsible for the hash slots. You can add one or more slave nodes to each master node. When a master node is unavailable, one of the slave nodes automatically takes over the master node to provide services.
Redis provides high throughput and availability in cluster mode. However, Redis still cannot prevent data loss in this mode. For more information, see Official Redis documentation.
Prerequisites
An ACK Serverless cluster that meets the following conditions is created:
Internet access is enabled for the cluster, and the cluster can pull public images.
A storage plug-in is deployed in the cluster.
We recommend that you deploy the Container Storage Interface (CSI) plug-in. Make sure that the Alibaba Cloud CSI-Provisioner component is deployed in the CSI plug-in.
CoreDNS is deployed in the cluster.
Build a Redis cluster
In this topic, the official Redis image v6.0.8 is used as an example to create a Redis cluster that consists of six nodes, including three master nodes and three slave nodes. Each node has a status and an identifier. You can use a Statefulset to create pods. You must attach a disk to each node to persistently store node data.
We recommend that you use Redis v5.0 or later. If you use a Redis version earlier than 5.0, you may need to use a command that is different from the command provided in this topic to initialize the cluster.
Create a ConfigMap to store and manage the configurations of the Redis cluster.
kubectl create -f redis-config.yaml
Sample content of the redis-config.yaml file:
apiVersion: v1 kind: ConfigMap metadata: name: redis-cluster data: redis.conf: | bind 0.0.0.0 port 6379 cluster-announce-bus-port 16379 cluster-enabled yes appendonly yes cluster-node-timeout 5000 dir /data cluster-config-file /data/nodes.conf requirepass pass123 masterauth pass123
The
dir
field specifies the persistent storage directory of node data in the Redis cluster. The/data
directory of pods must be a persistent storage directory.The
cluster-config-file
field specifies the node information of the Redis cluster. The node information is automatically generated and modified by the Redis node. The value of the field is also a persistent storage directory. After a node fails, other nodes in the cluster can take over the failed node to provide services.
Create a headless Service.
kubectl create -f redis-service.yaml
Sample content of the redis-service.yaml file:
apiVersion: v1 kind: Service metadata: name: redis-cluster-svc spec: clusterIP: None selector: app: redis-cluster
Create a Statefulset to deploy Redis.
When you create a StatefulSet, reference the Service that you created, mount a ConfigMap to the
/config
directory of each pod, create a persistent volume claim (PVC) for each pod, and then mount the PVC to the/data
directory.kubectl create -f redis.yaml
Sample content of the redis.yaml file:
apiVersion: apps/v1 kind: StatefulSet metadata: name: redis-cluster spec: selector: matchLabels: app: redis-cluster serviceName: redis-cluster-svc replicas: 6 template: metadata: labels: app: redis-cluster alibabacloud.com/eci: "true" spec: terminationGracePeriodSeconds: 10 affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - redis-cluster topologyKey: kubernetes.io/hostname weight: 100 containers: - name: redis image: redis:6.0.8 command: ["redis-server", "/config/redis.conf"] ports: - name: redis containerPort: 6379 protocol: TCP - name: election containerPort: 16379 protocol: TCP volumeMounts: - name: redis-conf mountPath: /config - name: pvc-essd-redis-data mountPath: /data volumes: - name: redis-conf configMap: name: redis-cluster items: - key: redis.conf path: redis.conf volumeClaimTemplates: - metadata: name: pvc-essd-redis-data spec: accessModes: [ "ReadWriteOnce" ] storageClassName: alicloud-disk-essd resources: requests: storage: 20Gi
After you create the Statefulset, wait until all pods in the Statefulset are in the Ready state.
kubectl get statefulset redis-cluster -o wide
The following command output is returned:
NAME READY AGE CONTAINERS IMAGES redis-cluster 6/6 8m52s redis redis:6.0.8
Initialize the cluster.
Obtain the IP address of each node.
Redis does not allow you to initialize a cluster by using the hostname. You must obtain the IP address of each node when you initialize a Redis cluster.
kubectl get pods -l app=redis-cluster -o wide
The following command output is returned:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES redis-cluster-0 1/1 Running 0 9m37s 172.16.55.6 virtual-kubelet-cn-beijing-k <none> <none> redis-cluster-1 1/1 Running 0 9m5s 172.16.55.7 virtual-kubelet-cn-beijing-k <none> <none> redis-cluster-2 1/1 Running 0 8m30s 172.16.55.8 virtual-kubelet-cn-beijing-k <none> <none> redis-cluster-3 1/1 Running 0 7m46s 172.16.55.9 virtual-kubelet-cn-beijing-k <none> <none> redis-cluster-4 1/1 Running 0 7m7s 172.16.55.10 virtual-kubelet-cn-beijing-k <none> <none> redis-cluster-5 1/1 Running 0 6m30s 172.16.55.11 virtual-kubelet-cn-beijing-k <none> <none>
Log on to one of the Redis nodes.
kubectl exec -ti redis-cluster-0 bash
Run the initialization command for the six nodes. Set the
--cluster-replicas
option to 1 to specify that each master node is assigned a slave node. This way, the cluster contains three master nodes and three slave nodes.redis-cli -a pass123 --cluster create 172.16.55.6:6379 172.16.55.7:6379 172.16.55.8:6379 172.16.55.9:6379 172.16.55.10:6379 172.16.55.11:6379 --cluster-replicas 1
If the following command output is returned, the cluster is initialized.
[OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
Use the Redis cluster.
After you create a headless Service, the kubernetes cluster assigns a Domain Name System (DNS) A record to the Service in the
{service name}.{service namespace}.svc.{domain}
format. The DNS record is mapped to the IP address of a backend pod. Each time you access the Service name, the Service name is resolved to a random Redis node.This allows you to access the Redis cluster by accessing a pod in the cluster.
redis-cli -a pass123 -c -h redis-cluster-svc.default.svc.cluster.local -p 6379
Sample test:
172.16.55.8> set k1 v1 OK 172.16.55.8> get k1 "v1"
Scale up a Redis cluster
You cannot dynamically scale up a Redis cluster due to the following reasons: First, every time a new node is added, Redis must reallocate hash slots to all nodes in the cluster. You can add a script to the Redis image to automate the reallocation of hash slots every time a pod starts. However, when a large number of data is added to the cluster, continuous resharding slows down the scale-up of the cluster. Sharding may also exhaust the bandwidth of the Redis cluster, which may cause all clients that depend on Redis to time out. Second, no suitable policy can be used to determine whether to use a restarted pod as a master node or a slave node.
To address the preceding issues, the following example shows how to scale up a Redis cluster by performing manual sharding.
Change the number of replicas in the Statefulset from 6 to 8.
kubectl scale statefulsets redis-cluster --replicas=8
Obtain the IP addresses of the new nodes.
After all pods are in the Ready state, run the following command to obtain the IP addresses of the new nodes:
kubectl get pods -l app=redis-cluster -o wide
The following command output is returned. redis-cluster-6 and redis-cluster-7 are new nodes.
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES redis-cluster-0 1/1 Running 0 88m 172.16.55.6 virtual-kubelet-cn-beijing-k <none> <none> redis-cluster-1 1/1 Running 0 88m 172.16.55.7 virtual-kubelet-cn-beijing-k <none> <none> redis-cluster-2 1/1 Running 0 87m 172.16.55.8 virtual-kubelet-cn-beijing-k <none> <none> redis-cluster-3 1/1 Running 0 87m 172.16.55.9 virtual-kubelet-cn-beijing-k <none> <none> redis-cluster-4 1/1 Running 0 86m 172.16.55.10 virtual-kubelet-cn-beijing-k <none> <none> redis-cluster-5 1/1 Running 0 85m 172.16.55.11 virtual-kubelet-cn-beijing-k <none> <none> redis-cluster-6 1/1 Running 0 52s 172.16.55.16 virtual-kubelet-cn-beijing-k <none> <none> redis-cluster-7 1/1 Running 0 32s 172.16.55.17 virtual-kubelet-cn-beijing-k <none> <none>
Add a master node.
Log on to one of the nodes. In this topic, log on to redis-cluster-0. Run the following command to add redis-cluster-6 as a master node of the cluster:
kubectl exec -ti redis-cluster-0 bash
redis-cli -a pass123 --cluster add-node 172.16.55.16:6379 172.16.55.6:6379
NoteIn the preceding command,
172.16.55.6:6379
is the address of an existing node. After you connect to the node, redis-cli automatically obtains the addresses of other nodes.Query the ID of redis-cluster-6.
redis-cli -a pass123 -c cluster nodes | grep 172.16.55.16:6379 | awk '{print $1}'
The following command output is returned:
47879390ecc7635f5c57d3e324e2134b24******
Reallocate hash slots.
After you add redis-cluster-6 as a master node to the cluster, four master nodes exist in the Redis cluster. The four nodes share 16,384 hash slots. Each master node is allocated
16,384/4 = 4,096
hash slots on average. Therefore, 4,096 hash slots must be moved from the existing three master nodes to the new master node.Run the following command to reallocate the hash slots.
172.16.55.6:6379
is the address of an existing node. After you connect to the node, redis-cli automatically obtains the addresses of other nodes.redis-cli -a pass123 --cluster reshard 172.16.55.6:6379
Enter the required information in sequence based on the prompts of the preceding interactive command.
How many slots do you want to move (from 1 to 16384)? 4096 What is the receiving node ID? 47879390ecc7635f5c57d3e324e2134b24****** Source node #1: all Do you want to proceed with the proposed reshard plan (yes/no)? yes
The number of hash slots to be allocated to the new node (redis-cluster-6): 4,096
The ID of the receiving node (the new node): 47879390ecc7635f5c57d3e324e2134b24******
The ID of the source nodes (all existing nodes): all
Confirm reallocation: yes
Wait until the hash slots are reallocated. During the reallocation, the Redis cluster can continue to provide services.
Add the slave node.
Run the following command to add redis-cluster-7 as a slave node of redis-cluster-6:
redis-cli -a pass123 --cluster add-node 172.16.55.17:6379 172.16.55.6:6379 --cluster-slave --cluster-master-id 47879390ecc7635f5c57d3e324e2134b24******
NoteIn the preceding command,
172.16.55.6:6379
is the address of an existing node. After you connect to the node, redis-cli automatically obtains the addresses of other nodes.47879390ecc7635f5c57d3e324e2134b24******
is the ID of the master node (redis-cluster-6).
After you add the slave node, you can see four master nodes that correspond to four slave nodes in the Redis cluster.
redis-cli -a pass123 -c cluster nodes
Sample output:
47879390ecc7635f5c57d3e324e2134b24****** 172.16.55.16:6379@16379 master - 0 1667383406000 7 connected 0-1364 5461-6826 10923-12287 75a7398a45f9696066eaa6ac7968b13a47****** 172.16.55.11:6379@16379 slave b8f6b826241b47f29a4bdde14104b28fe8****** 0 1667383406514 2 connected b7849f8577e43d5a6da51bc78ae809bbb1****** 172.16.55.17:6379@16379 slave 47879390ecc7635f5c57d3e324e2134b24****** 0 1667383406314 7 connected b8f6b826241b47f29a4bdde14104b28fe8****** 172.16.55.7:6379@16379 master - 0 1667383406815 2 connected 6827-10922 ffce547186e0be179830cb0dca47c203f6****** 172.16.55.6:6379@16379 myself,master - 0 1667383406000 1 connected 1365-5460 833a939cde93991c8a16c41fb2568b8642****** 172.16.55.8:6379@16379 master - 0 1667383406514 3 connected 12288-16383 a4df1a26394bfcb833914278744219db01****** 172.16.55.9:6379@16379 slave 833a939cde93991c8a16c41fb2568b8642****** 0 1667383407515 3 connected c4234d71343ca4dbe07822e17f7572ac30****** 172.16.55.10:6379@16379 slave ffce547186e0be179830cb0dca47c203f6****** 0 1667383405813 1 connected
Scale down a Redis cluster
During scale-down of Redis clusters, StatefulSets can only delete pods one by one in the reverse order that pods were created. If you want to delete a master node and a slave node from the Redis cluster, you must use the Statefulset to delete redis-cluster-7 and redis-cluster-6. Redis must allocate all hash slots on the two nodes to other nodes before the StatefulSet deletes the nodes. Otherwise, data loss occurs, and the Redis cluster rejects services.
Reallocate hash slots.
To prevent load skew on the remaining three master nodes, the StatefulSet must evenly allocate the hash slots of redis-cluster-6 to the remaining master nodes. In this case, Redis must perform three resharding operations. The procedure is similar to the scale-up procedure. However, in each resharding operation, you must set the receiving node to one of the remaining master nodes, and the source node to redis-cluster-6. Sample command:
redis-cli -a pass123 --cluster reshard 172.16.55.6:6379
Enter the required information in sequence based on the prompts of the preceding interactive command.
How many slots do you want to move (from 1 to 16384)? 4096 What is the receiving node ID? ffce547186e0be179830cb0dca47c203f6****** Source node #1: 47879390ecc7635f5c57d3e324e2134b24****** Source node #2: done Do you want to proceed with the proposed reshard plan (yes/no)? yes
The number of hash slots to be allocated to the remaining master nodes: 4,096
The ID of the receiving node (one of the remaining nodes): ffce547186e0be179830cb0dca47c203f6******
The ID of the source node (redis-cluster-6): 47879390ecc7635f5c57d3e324e2134b24******
Confirm reallocation: yes
Change the number of replicas in the Statefulset from 8 to 6.
kubectl scale statefulsets redis-cluster --replicas=6
View the pod information of the Redis cluster. You can see that the number of nodes in the cluster is reduced to six.
kubectl get pods -l app=redis-cluster -o wide
Delete a Redis cluster
To delete a Redis cluster, you must delete the StatefulSet, Service, and ConfigMap.
kubectl delete statefulset redis-cluster
kubectl delete svc redis-cluster-svc
kubectl delete cm redis-cluster
After you delete the StatefulSet, the related PVC is not automatically deleted. You must manually delete the PVC. After you delete the PVC, the corresponding disks and data are also deleted.
kubectl delete pvc -l app=redis-cluster