Use FlexVolume to persist data based on OSS - Container Service for Kubernetes

When a node fails, data stored in containers of stateful applications may be lost or become unreliable. Use persistent storage to prevent data loss. This topic describes how to use an Object Storage Service (OSS) volume to persist data.

Background information

OSS is a secure, cost-effective, high-capacity, and highly reliable cloud storage service provided by Alibaba Cloud. You can mount an OSS bucket to multiple pods in a Container Service for Kubernetes (ACK) cluster.

Typical use cases include the following:

Low disk I/O.
Shared workloads such as configuration files, images, and short video files.

How to use OSS

Create an OSS bucket.
Obtain your AccessKey ID and AccessKey secret.
Create a persistent volume (PV) and persistent volume claim (PVC) using a Secret.

Prerequisites

Create an ACK managed cluster.
Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.
You have created an OSS bucket in the OSS console. For more information, see Create buckets.

Usage notes

Upgrading your ACK cluster restarts kubelet and the ossfs driver. As a result, mounted OSS directories become unavailable. In this case, you must recreate the pods that use the OSS volume. Add health check settings to your pod’s YAML file so Kubernetes automatically restarts the pod and remounts the OSS volume when the OSS directory becomes unavailable.
This issue has been resolved by mounting OSS using the latest version.

Create a PV

Run the following command to create the Secret:
Replace <your AccessKey ID> and <your AccessKey Secret> in the following command with the actual AccessKey ID and AccessKey secret of your Alibaba Cloud account. To obtain the AccessKey pair of your Alibaba Cloud account, go to the ACK console, move your pointer over the icon and click AccessKey.
```
kubectl create secret generic osssecret --from-literal=akId='<your AccessKey ID>' --from-literal=akSecret='<your AccessKey Secret>' --type=alicloud/oss -n default
```
osssecret: the name of the Secret. You can specify a custom name.
akId: the AccessKey ID.
akSecret: the AccessKey secret.
--type: the type of Secret. In this example, the value is set to alicloud/oss. The Secret and the pod that uses the Secret must belong to the same namespace.

Use the pv-oss.yaml file to create a PV.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-oss
  labels:
    alicloud-pvname: pv-oss
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  storageClassName: oss
  flexVolume:
    driver: "alicloud/oss"
    secretRef:
      name: "osssecret"  # Replace with the name of the Secret you created in the previous step.
    options:
      bucket: "docker"                        // Replace with your bucket name.
      path: /path                             // Replace with your relative subdirectory path.
      url: "oss-cn-hangzhou.aliyuncs.com"     // Replace with your endpoint.
      otherOpts: "-o max_stat_cache_size=0 -o allow_other"   // Replace with your custom parameters.

Parameters:

alicloud-pvname: Name of the PV. Use this label in the selector field of a PVC to bind the PV to the PVC.
bucket: Name of the OSS bucket.
path: Relative path from the root of the bucket. Default is /. Supported in csi-plugin v1.14.8.32-c77e277b-aliyun and later.
url: Endpoint of the OSS bucket. To find it:
1. Log on to the OSS console.
2. In the left-side navigation pane, click Buckets. On the Buckets page, click the name of the bucket whose internal endpoint you want to obtain.
3. In the left-side navigation tree of the target bucket, click Overview.
4. In the Port section, view the bucket’s endpoint.
otherOpts: Custom mount parameters. Format: -o *** -o ***.

Run the following command to create the PV:
```
kubectl create -f pv-oss.yaml
```

Expected result:

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, find the cluster you want and click its name. In the left navigation pane, choose Volumes > Persistent Volumes.
On the Persistent Volumes page, you see the PV you just created.

Create a PVC

Create a persistent volume claim (PVC) for the OSS bucket. Use the selector field to match the PV. This ensures precise binding between the PVC and PV. Use the storageClassName field to restrict binding to only OSS-type PVs.

Create a file named pvc-oss.yaml.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-oss
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: oss
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      alicloud-pvname: pv-oss

Run the following command to create the PVC:
```
kubectl create -f pvc-oss.yaml
```

Expected result:

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click your cluster name. In the navigation pane on the left, choose Volumes > Persistent Volume Claims.
On the Persistent Volume Claims page, you see the PVC you just created.

Create an application

Create a file named oss-static.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: oss-static
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        ports:
        - containerPort: 80
        volumeMounts:
          - name: pvc-oss
            mountPath: "/data"
          - name: pvc-oss
            mountPath: "/data1"
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - cd /data
          initialDelaySeconds: 30
          periodSeconds: 30
      volumes:
        - name: pvc-oss
          persistentVolumeClaim:
            claimName: pvc-oss

Note

For details about the livenessProbe health check, see OSS volumes.

Run the following command to create the Deployment:
```
kubectl create -f oss-static.yaml
```

Expected result:

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click your cluster name. In the navigation pane on the left, choose Workloads > Deployments.
On the Deployments page, you can view the newly created Deployment.

OSS Persistent Storage

Run the following command to get the name of the pod running your Deployment:

kubectl get pod

Expected output:

NAME                             READY   STATUS    RESTARTS   AGE
oss-static-66fbb85b67-dqbl2      1/1     Running   0          1h

Run the following command to list files in the /data path:
```
kubectl exec oss-static-66fbb85b67-dqbl2 -- ls /data | grep tmpfile
```
Note
The /data path is empty.
Run the following command to create the file tmpfile in the /data directory.
```
kubectl exec oss-static-66fbb85b67-dqbl2 -- touch /data/tmpfile
```

Run the following command to list files in the /data path:

kubectl exec oss-static-66fbb85b67-dqbl2 -- ls /data | grep tmpfile

Expected output:

tmpfile

Run the following command to delete the pod named oss-static-66fbb85b67-dqbl2:
```
kubectl delete pod oss-static-66fbb85b67-dqbl2
```
Expected output:
```
pod "oss-static-66fbb85b67-dqbl2" deleted
```

In another terminal window, run the following command to watch the pod deletion and recreation process:

kubectl get pod -w -l app=nginx

Expected output:

NAME                             READY   STATUS    RESTARTS   AGE
oss-static-66fbb85b67-dqbl2      1/1     Running   0          78m
oss-static-66fbb85b67-dqbl2   1/1   Terminating   0     78m
oss-static-66fbb85b67-zlvmw   0/1   Pending   0     <invalid>
oss-static-66fbb85b67-zlvmw   0/1   Pending   0     <invalid>
oss-static-66fbb85b67-zlvmw   0/1   ContainerCreating   0     <invalid>
oss-static-66fbb85b67-dqbl2   0/1   Terminating   0     78m
oss-static-66fbb85b67-dqbl2   0/1   Terminating   0     78m
oss-static-66fbb85b67-dqbl2   0/1   Terminating   0     78m
oss-static-66fbb85b67-zlvmw   1/1   Running   0     <invalid>

Run the following command to get the name of the recreated pod:

kubectl get pod

Expected output:

NAME                             READY   STATUS    RESTARTS   AGE
oss-static-66fbb85b67-zlvmw      1/1     Running   0          40s

Run the following command to list files in the /data path. The tmpfile still exists. This confirms that data persists on the OSS volume.
```
kubectl exec oss-static-66fbb85b67-zlvmw -- ls /data | grep tmpfile
```
Expected output:
```
tmpfile
```