Container Service for Kubernetes:Best practices for data security of disk volumes

Scenarios

Data loss occurs due to accidental ESSD deletions

If the reclaim policy of an ESSD is Delete, after you delete the persistent volume claim (PVC) that is used to mount the ESSD, the related persistent volume (PV) and ESSD are also deleted. When you accidentally deleted a disk, you can use a snapshot of the disk to restore the disk data to the point in time when the snapshot was created. You cannot restore disk data if no snapshot is created. You cannot restore the data that is written into the disk after the snapshot was created.

To ensure that you can restore disk data when you accidentally delete an ESSD, ACK can automatically create IA snapshots for ESSDs before disk deletion. The IA snapshots created by ACK are retained for a specific period of time. This allows you to restore the data in ESSDs by using the IA snapshots created by ACK.

Benefits

• If a disk is accidentally deleted, you can restore the disk data by using an IA snapshot of the disk.

• The IA snapshot feature does not affect the deletion operation because IA snapshots can be created within seconds.

  Note

  Only ESSDs support IA snapshots.

Configurations

csi-provisioner configurations

Run the following command to add the following environment variable to the containers parameter of csi-provisioner in the kube-system namespace: VOLUME_DEL_AUTO_SNAP: "true".

kubectl patch deploy csi-provisioner -n kube-system -p '{"spec":{"template":{"spec":{"containers":[{"name":"csi-provisioner","env":[{"name":"VOLUME_DEL_AUTO_SNAP","value":"true"}]}]}}}}'

StorageClass configurations

Create a StorageClass based on the following template:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-datasafe-essd
provisioner: diskplugin.csi.alibabacloud.com
parameters:
  type: cloud_essd
  volumeDeleteSnapshotRetentionDays: "3"   # This setting takes effect only when the reclaimPolicy parameter is set to "Delete". 
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true

Create an IA snapshot

In this section, a MySQL application is created and has an ESSD mounted.

1. Create a file named mysql.yaml based on the following requirements:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: essd-pvc
     namespace: autosnapshot
   spec:
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: 25Gi
     storageClassName: alicloud-datasafe-essd  # Use the StorageClass that you created in the previous section. 
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: mysql-pass
     namespace: autosnapshot
   type: Opaque
   data:
     username: dGVzdDEK
     password: dGVzdDEtdmFsdWUK
   ---
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: mysql-sts
     namespace: autosnapshot
   spec:
     selector:
       matchLabels:
         app: mysql-sts
     serviceName: mysql-sts
     template:
       metadata:
         labels:
           app: mysql-sts
       spec:
         containers:
         - name: mysql-sts
           image: mysql:5.7
           env:
           - name: MYSQL_ROOT_PASSWORD
             valueFrom:
               secretKeyRef:
                 name: mysql-pass
                 key: password
           ports:
           - containerPort: 80
             name: mysql-sts
           volumeMounts:
           - name: mysql
             mountPath: /var/lib/mysql
             subPath: mysql
         volumes:
           - name: mysql
             persistentVolumeClaim:
               claimName: essd-pvc

2. Run the following command to deploy the MySQL application:

   kubectl apply -f mysql.yaml

3. Run the following command to query the disk volume that is mounted to the MySQL application:

   kubectl get pvc -nautosnapshot | grep essd-pvc

   Expected output:

   essd-pvc   Bound    d-2zeit7uza22vjya1****   25Gi       RWO            alicloud-datasafe-essd   54s

   The ID of the mounted ESSD is d-2zeit7uza22vjya1****.

4. Run the following command to access the container of the MySQL application:

   kubectl -n autosnapshot exec -it mysql-sts-0 -- /bin/sh

5. Run the following command in a shell of the container to write data into the container:

   dd if=/dev/urandom of=/var/lib/mysql/mysql/record.txt bs=1M count=1000

6. Run the following command to query the size of the data that is written into the container:

   ls /var/lib/mysql/mysql -l | grep record

   Expected output:

   -rw-r--r-- 1 root  root  1048576000 Nov  8 02:36 record.txt

Delete the disk

1. Run the following command to reduce the number of replicated pods to 0 for the MySQL application:

   kubectl scale sts/mysql-sts -n autosnapshot --replicas=0

2. Run the following command to delete the PVC that is used by the MySQL application:

   kubectl delete pvc essd-pvc -n autosnapshot

3. Run the following command to query the VolumeSnapshot that is created for the IA snapshot:

   kubectl get volumesnapshot

   Expected output:

   d-2zeit7uza22vjya1****-delprotect                   true                                d-2zeit7uza22vjya1****-delprotect-content   30Gi                               d-2zeit7uza22vjya1****-delprotect-content          6s             6s

   The following list describes some parameters in the preceding output:

   • d-2zeit7uza22vjya1****: the ID of the ESSD that is deleted. The ID is the same as the ID of the ESSD that is mounted by using the essd-pvc PVC.

   • d-2zeit7uza22vjya1****-delprotect: the name of the VolumeSnapshot that is created for the IA snapshot.

   • d-2zeit7uza22vjya1****-delprotect-content: the name of the VolumeSnapshotContent that is created for the IA snapshot.

Use the IA snapshot to restore data after the disk is deleted

In this section, the IA snapshot created in the previous Delete the disk section is used to restore the disk data after the disk is deleted.

1. Run the following command to query the value of the snapshotHandle parameter of the VolumeSnapshotContent that is created for the IA snapshot:

   kubectl get volumesnapshotcontent d-2zeit7uza22vjya1****-delprotect-content -oyaml | grep snapshotHandle

   Expected output:

   snapshotHandle: s-2zegw6gmuc866xgc****

2. Create a VolumeSnapshotContent by using the following YAML template. The template specifies the VolumeSnapshot to be created.

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshotContent
   metadata:
     name: datasafe-volumesnapshotcontent
   spec:
     deletionPolicy: Retain
     driver: diskplugin.csi.alibabacloud.com
     source:
       snapshotHandle: s-2zegw6gmuc866xgc****   # Set to the value of the snapshotHandle parameter of the original VolumeSnapshotContent. 
     volumeSnapshotRef:
       name: datasafe-volumesnapshot       # The name of the VolumeSnapshot to be created. 
       namespace: autosnapshot             # The namespace in which the MySQL application is deployed. 

3. Create a VolumeSnapshot in the namespace in which the MySQL application is deployed by using the following YAML template:

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: datasafe-volumesnapshot
     namespace: autosnapshot
   spec:
     source:
       volumeSnapshotContentName: datasafe-volumesnapshotcontent

4. Create a PVC by using the following YAML template. The template specifies the VolumeSnapshot to be used.

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: essd-pvc       # Set to the name of the PVC used by the MySQL application. 
     namespace: autosnapshot
   spec:
     accessModes: [ "ReadWriteOnce" ]
     storageClassName: alicloud-datasafe-essd
     resources:
       requests:
         storage: 30Gi
     dataSource:
       name: datasafe-volumesnapshot
       kind: VolumeSnapshot
       apiGroup: snapshot.storage.k8s.io

5. Run the following command to increase the number of replicated pods to 1 for the MySQL application:

   kubectl scale sts/mysql-sts -n autosnapshot --replicas=1

6. Run the following command to check whether the PVC is mounted to the MySQL application:

   kubectl describe pvc essd-pvc -n autosnapshot | grep "Used By"

   Expected output:

   Used By:     mysql-sts-0

7. Run the following command to access the container of the MySQL application:

   kubectl -n autosnapshot exec -it mysql-sts-0 -- /bin/sh

8. Run the following command to check whether the data that you previously wrote into the container is restored:

   ls /var/lib/mysql/mysql -l | grep record

   Expected output:

   -rw-r--r-- 1 root  root  1048576000 Nov  8 02:36 record.txt

   In the output, 1048576000 is returned for the size of the data. The size is the same as the size of the data that you previously wrote into the container. This indicates that the data restoration is successful.