Container Service for Kubernetes:Use EFC to accelerate access to NAS or CPFS file systems

Overview of EFC

Use Fluid EFCRuntime to accelerate access to NAS file systems

Fluid uses custom Kubernetes resources related to Fluid EFCRuntime to connect to EFC. This helps implement dataset observability and scalability.

Limits

The following limits apply to Fluid EFCRuntime:

• Fluid EFCRuntime does not support DataLoad cache prefetching. Fluid EFCRuntime caches data only when the data is accessed for the first time.

• Fluid EFCRuntime does not expose the caching status of datasets.

• Fluid EFCRuntime is supported only in the following regions: China (Zhangjiakou), China (Beijing), China (Guangzhou), China (Shenzhen), and China (Shanghai).

How Fluid EFCRuntime works

The following figure shows how Fluid EFCRuntime caches data from NAS to the local storage to accelerate data access. The following section describes how Fluid EFCRuntime works:

1. You can create custom resource definitions (CRDs) of datasets and EFCRuntimes to specify information about the source NAS file systems.

2. Fluid controllers deploy the EFC Cache Worker and EFC FUSE components based on the information about the source file systems.

3. When you create a pod, you can use a persistent volume claim (PVC) to mount the mount target of a file system exposed by the EFC FUSE client to the pod.

4. When you access the data in a mounted NAS file system, the EFC FUSE client forwards the request to EFC Cache Worker. EFC Cache Worker checks whether the data is cached in the local storage. If the data is cached in the local storage, you can directly access the cache. If the data is not cached in the local storage, EFC Cache Worker reads the data from the NAS file system and caches the data to the local storage. Then, you can access the cache in the local storage.

• Dataset: a CRD defined by Fluid. A dataset is a collection of logically related data that is used by upper-layer compute engines.

• EFCRuntime: a runtime that accelerates access to datasets. EFCRuntimes use EFC as the caching engine. The EFC caching engines include the EFC Cache Worker and EFC FUSE components.

  • EFC Cache Worker: a server-side component that enables caching based on consistent hashing. You can disable this component based on your requirements. After you disable this component, distributed read-only caches are disabled. Other features are not affected.

  • EFC FUSE: a client-side component of EFC that exposes data access interfaces over the POSIX protocol.

Procedure

Step 1: Install ack-fluid

Install the cloud-native AI suite and ack-fluid 0.9.10 or later.

Step 2: Write data to the NAS file system

1. Mount the NAS file system to an ECS instance. For more information, see Mount an NFS file system in the NAS console.

2. Run the following command to query the mount target of the CPFS file system:

   findmnt /mnt

   Expected output:

   TARGET   SOURCE                                         FSTYPE OPTIONS
   /mnt/nfs xxxxxxxxxxx-xxxxx.cn-beijing.nas.aliyuncs.com:/ nfs    rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,no

3. Run the following command to create a file of 10 GB in size in the mount directory of the NAS file system:

   dd if=/dev/zero of=/mnt/nfs/allzero-demo count=1024 bs=10M

   Expected output:

   1024+0 records in
   1024+0 records out
   10737418240 bytes (11 GB) copied, 50.9437 s, 211 MB/s

Step 3: Create a dataset and an EFCRuntime

1. Create a file named dataset.yaml and copy the following content to the file:

   Sample template of a NAS file system

   apiVersion: data.fluid.io/v1alpha1
   kind: Dataset
   metadata:
     name: efc-demo
   spec:
     mounts:
       - mountPoint: "nfs://<nas_url>:<nas_dir>"
         name: efc
         path: "/"
   ---
   apiVersion: data.fluid.io/v1alpha1
   kind: EFCRuntime
   metadata:
     name: efc-demo
   spec:
     replicas: 3
     master:
       networkMode: ContainerNetwork
     worker:
       networkMode: ContainerNetwork
     fuse:
       networkMode: ContainerNetwork
     tieredstore:
       levels:
         - mediumtype: MEM
           path: /dev/shm
           quota: 15Gi

   The dataset.yaml file is used to create a dataset and an EFCRuntime.

   • The dataset specifies information about the NAS file system, such as the URL of the NAS file system and the directory that you want to mount.

   • The EFCRuntime starts an EFC caching system to provide caching services. You can specify the number of replicated pods for the worker component of the EFC caching system and the cache capacity of each worker component.

   Parameter	Description
   mountPoint	If you use a NAS file system, set the value in the nfs://<nas_url>:<nas_dir> format. nas_url: the URL of the NAS file system. To obtain the URL of the NAS file system, perform the following operations: Log on to the NAS console. In the left-side navigation pane, choose File System > File System List. On the File System List page, find the NAS file system that you want to mount and click Manage in the Actions column. On the page that appears, click Mount Targets. For more information, see Manage mount targets. nas_dir: the subdirectory that you want to mount. In most cases, you can set the value to the root directory. For example, a value of efc://xxxxxxxxxxx-xxxxx.cn-beijing.nas.aliyuncs.com:/ specifies the root directory of a NAS file system.
   replicas	The number of replicated pods that are created for the worker component of the EFC caching system. You can set the value based on the memory size of the compute node and the size of the dataset. We recommend that you ensure that the product of the value of quota and the value of replicas is greater than the size of the dataset.
   networkMode	Valid values: ContainerNetwork and HostNetwork. In the ACK environment, we recommend that you set the value to ContainerNetwork. This network mode does not compromise the network performance.
   mediumtype	The cache type. Valid values: HDD, SSD, and MEM. A value of MEM indicates memory. In AI training scenarios, we recommend that you set this parameter to MEM. If you set this parameter to MEM, you must set the path parameter to a memory file system, such as tmpfs.
   path	The cache directory in the pods of the worker components of the EFC caching system. We recommend that you set the value of this parameter to /dev/shm.
   quota	The cache capacity of each worker component. You can set the value based on the memory size of the compute node and the size of the dataset. We recommend that you ensure that the product of the value of quota and the value of replicas is greater than the size of the dataset.

2. Run the following command to create an EFCRuntime and a dataset:

   kubectl create -f dataset.yaml

3. Run the following command to check whether the Dataset is deployed:

   kubectl get dataset efc-demo

   Expected output:

   NAME       UFS TOTAL SIZE   CACHED   CACHE CAPACITY   CACHED PERCENTAGE   PHASE   AGE
   efc-demo                                                                  Bound   24m

   The dataset is in the Bound state. This indicates that the EFC caching system runs as expected in the cluster and application pods can access the data provided by the dataset.

4. Run the following command to check whether the EFCRuntime is deployed:

   kubectl get efcruntime

   Expected output:

   NAME       MASTER PHASE   WORKER PHASE   FUSE PHASE   AGE
   efc-demo   Ready          Ready          Ready        27m

   The results indicate that the master, worker, FUSE components are in the Ready state.

5. Run the following commands to check whether the persistent volume (PV) and PVC are created:

   After the dataset and the EFC caching system are ready, Fluid automatically creates a PVC and a PV.

   kubectl get pv,pvc

   Expected output:

   NAME                                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   REASON   AGE
   persistentvolume/default-efc-demo   100Gi      ROX            Retain           Bound    default/efc-demo         fluid                   94m
   
   NAME                                   STATUS   VOLUME             CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   persistentvolumeclaim/efc-demo         Bound    default-efc-demo   100Gi      ROX            fluid          94m

Step 4: Create an application to access data

Create an application to check whether access to data is accelerated. In this example, an application that provisions two pods is created and used to access the NAS file system multiple times from two nodes. You can evaluate the acceleration performance of Fluid EFCRuntime based on the amount of time that is required for accessing data.

1. Create a file named app.yaml and copy the following content to the file.

   The following content defines a StatefulSet named efc-app. The StatefulSet contains two pods. Each pod has the efc-demo PVC mounted to the /data directory.

   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: efc-app
     labels:
       app: nginx
   spec:
     serviceName: nginx
     replicas: 2
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - name: nginx
           image: registry.openanolis.cn/openanolis/nginx:1.14.1-8.6
           command: ["/bin/bash"]
           args: ["-c", "sleep inf"]
           volumeMounts:
           - mountPath: "/data"
             name: data-vol
         volumes:
           - name: data-vol
             persistentVolumeClaim:
               claimName: efc-demo

2. Run the following command to create a StatefulSet named efc-app:

   kubectl create -f app.yaml

3. Run the following command to query the size of the specified file:

   kubectl exec -it efc-app-0 -- du -h /data/allzero-demo

   Expected output:

   10G     /data/allzero-demo

4. Query the amount of time required for reading the specified file from the application.

   Note

   The amount of time and throughput may vary based on the runtime environment and measuring method. In this topic, the cluster has three ECS instances of the ecs.g7ne.8xlarge type. The efc-demo EFCRuntime has three worker pods that run on the same ECS instance. The efc-app StatefulSet has two pods that separately run on the other two ECS instances. The amount of time required for data access is not affected by the kernel cache of the node on which the EFC FUSE client runs.

   1. Run the following command to check the amount of time required for reading the specified file from the efc-app-0 pod of the StatefulSet:

      Note

      If you want to read another file, replace /data/allzero-demo with the path of the file.

      kubectl exec -it efc-app-0 -- bash -c "time cat /data/allzero-demo > /dev/null"

      Expected output:

      real    0m15.792s
      user    0m0.023s
      sys     0m2.404s

      The results indicate that 15.792 seconds is required for reading a file of 10 GB in size and the read speed is 648 MiB/s.

   2. Run the following command to check the amount of time required for reading the same file of 10 GB in size from the other pod of the StatefulSet:

      Note

      If you want to read another file, replace /data/allzero-demo with the path of the file.

      kubectl exec -it efc-app-1 -- bash -c "time cat /data/allzero-demo > /dev/null"

      Expected output:

      real    0m9.970s
      user    0m0.012s
      sys     0m2.283s

      The results indicate that 9.970 seconds is required for reading a file of 10 GB in size and the read speed is 1,034.3 MiB/s.

   After Fluid EFCRuntime is used, the read speed is increased from 648 MiB/s to 1,034.3 MiB/s. For the same file, Fluid EFCRuntime increases the read speed by about 100%.