Container Service for Kubernetes:Best practices for the cache optimization policies of Fluid

Policy 1: Select proper ECS instance types

Performance estimation

A distributed cache system can aggregate the storage and bandwidth resources on different nodes and provide larger cache capacity and higher bandwidth for upper-layer applications. You can calculate the cache capacity of the cache system, the cache bandwidth, and the maximum bandwidth of a pod based on the following formulas:

• Cache capacity of the cache system= Cache capacity of each cache system worker pod × Number of cache system worker pods

• Cache bandwidth = Number of cache system worker pods × min{Maximum ECS bandwidth of worker pods, I/O throughput of the cache media used by worker pods} The cache bandwidth is not equal to the actual bandwidth for cache access. The actual bandwidth depends on the bandwidth of the client ECS instance and the access mode, such as sequential read or random read.

• Maximum bandwidth for a pod = min{Bandwidth of the hosting ECS instance, Cache bandwidth} When multiple pods concurrently access the cache, the cache bandwidth is shared by the pods.

Example

Assume that two ECS instances of the ecs.g7nex.8xlarge type are added to your Container Service for Kubernetes (ACK) cluster to build a distributed cache system. The cache system consists of two worker pods each of which provides 100 GiB of memory cache. The pods run on different ECS instances. An application pod is deployed on an ECS instance of the ecs.gn7i-c8g1.2xlarge type. The instance provides 8 vCPUs, 30 GiB of memory, and 16 Gbit/s of bandwidth. The cache capacity of the cache system, cache bandwidth, and maximum pod bandwidth can be calculated based on the following formulas:

• Cache capacity of the cache system = 100 GiB × 2 = 200 GiB

• Cache bandwidth = 2 × min{40 Gbit/s, Memory I/O throughput} = 80 Gbit/s

• When a cache hit occurs, the maximum pod bandwidth = min{80 Gbit/s, 16 Gbit/s} = 16 Gbit/s

Recommended ECS instance types

The bandwidth of a distributed cache cluster depends on the maximum bandwidth of ECS instances in the cluster and the cache media used by the cluster. To improve the performance of the distributed cache system, we recommend that you select ECS instances that provide high bandwidth and use memory, local hard disk drives (HDDs), and local SSDs as the cache media. We recommend that you select the following ECS instance types.

For more information about ECS instance families, see Overview of instance families.

Policy 2: Select proper cache media

The bandwidth of a distributed cache cluster depends on the maximum bandwidth of ECS instances in the cluster and the cache media used by the cluster. To improve the performance of the distributed cache system, we recommend that you select ECS instances that provide high bandwidth and use memory or local SSDs as the cache media. Fluid allows you to use the spec.tieredstore parameter of the cache runtime to configure different cache media and corresponding cache capacities.

spec:
  tieredstore:
    levels:
      - mediumtype: MEM
        volumeType: emptyDir
        path: /dev/shm
        quota: 30Gi # The cache capacity provided by each worker pod of the distributed cache system. 
        high: "0.99"
        low: "0.95"

Policy 3: Schedule pods based on cache affinity

When a cache hit occurs, application pods read data from the cache system. In this case, if the pod and the cache system are deployed in different zones, the pod needs to access cache across zones. Network jitters may occur during cross-zone access. To avoid network jitters, you can configure affinity rules between the application pod and the cache system pods. Details:

• Configure affinity rules to deploy the worker pods of the cache system in the same zone.

• Configure affinity rules to deploy application pods in the zone where the worker pods of the cache system are deployed.

Fluid allows you to use the spec.nodeAffinity parameter of a dataset to specify an affinity rule for the worker pods of the cache system. Sample code:

apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
  name: demo-dataset
spec:
  ...
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
                - <ZONE_ID> # The ID of the zone where you want to deploy the worker pods of the cache system. Example: cn-beijing-i.

The preceding code block can be used to schedule the worker pods of the cache system on ECS instances in the specified zone (zone ID: <ZONE_ID>).

In addition, Fluid automatically injects the cache affinity rule into the application pod specification. Based on the affinity rule, the system attempts to schedule the application pods to the zone where the worker pods of the cache system are deployed. For more information, see Schedule pods based on cache affinity.

Policy 4: Optimize cache system parameters for large file full sequential reads

Full data sequential reads on large files are used in data-intensive scenarios. For example, full data sequential reads on large files are required when you use datasets of the TFRecord or TAR format to train models, load one or more model files during the startup of an inference service, or read data from Parquet files to implement distributed data analytics. In the preceding scenarios, you can enable prefetching to accelerate data access. For example, you can increase the prefetching concurrency and the size of prefetched data.

The parameters that you can use to configure prefetching vary based on the cache runtime that you use in Fluid.

kind: JindoRuntime
metadata:
  ...
spec:
  fuse:
    properties:
      fs.oss.download.thread.concurrency: "200"
      fs.oss.read.buffer.size: "8388608"
      fs.oss.read.readahead.max.buffer.count: "200"
      fs.oss.read.sequence.ambiguity.range: "2147483647" # Approximately 2G.

kind: JuiceFSRuntime
metadata:
  ...
spec:
  fuse:
    options:
      buffer-size: "2048"
      cache-size: "0"
      max-uploads: "150"
  worker:
    options:
      buffer-size: "2048"
      max-downloads: "200"
      max-uploads: "150"
  master:
    resources:
      requests:
        memory: 2Gi
      limits:
        memory: 8Gi
  ...

Policy 1: Do not mount large directories to the underlying data source

The cache system maintains the metadata of all files in the directories mounted to the cache system and records additional information about the files, such as the cache status. If the cache system is mounted with large directories, such as the root directories of large storage systems, the cache system consumes a large amount of memory resources to store the metadata of the files in the mounted directories and consumes additional CPU resources to process access to the metadata.

Fluid defines the dataset concept. A dataset is a collection of logically related data intended for upper-layer applications. A dataset corresponds to a distributed cache system. We recommend that you use a separate dataset to accelerate data access for a data-intensive application or relevant tasks. If you need to run multiple applications, we recommend that you create separate datasets for the applications and mount different subdirectories of the underlying data source to different datasets. A mounted subdirectory contains the data required by the application to which the subdirectory is mounted. This provides improved data isolation when different applications use the cache systems of the mounted datasets to access data. This also ensures the stability and performance of the cache systems. Sample dataset configurations:

apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
  name: demo-dataset
spec:
  ...
  mounts:
  - mountPoint: oss://<BUCKET>/<PATH1>/<SUBPATH>/
    name: sub-bucket

Multiple cache systems may increase the complexity of maintenance. You can select flexible cache system architectures based on your business requirements:

• If the datasets that you want to use are small in size, strongly rely on each other, stored in the same storage system, and contain a small number of files, we recommend that you create one Fluid dataset and one distributed cache system.

• If the datasets that you want to use are large in size and contain a large number of files, we recommend that you create multiple Fluid datasets and multiple distributed cache systems. You can mount one or multiple Fluid datasets to your application pods.

• If the datasets that you want to use belong to different storage systems of different users and data isolation between users is required, we recommend that you create separate short-lived Fluid datasets for each user or the relevant data processing jobs. You can develop Kubernetes operators to manage multiple Fluid datasets.

Policy 2: Persist metadata to improve cache system stability

Most distributed cache systems adopt a master-worker architecture. The master pod is responsible for maintaining the metadata of all files in the directories mounted to the cache system and recording the cache status of the files. When an application uses such a cache system to access a file, the application needs to first obtain the metadata of the file from the master pod and then use the metadata to retrieve the file from the underlying storage or the worker pod of the cache system. Therefore, the stability of the master pod is critical to the high availability of the cache system.

If you use JindoRuntime, you can use the following configurations to improve the availability of the master pod:

apiVersion: data.fluid.io/v1alpha1
kind: JindoRuntime
metadata:
  name: sd-dataset
spec:
  ...
  volumes:
    - name: meta-vol
      persistentVolumeClaim:
        claimName: demo-jindo-master-meta
  master:
    resources:
      requests:
        memory: 4Gi
      limits:
        memory: 8Gi
    volumeMounts:
      - name: meta-vol
        mountPath: /root/jindofs-meta
    properties:
      namespace.meta-dir: "/root/jindofs-meta"

The preceding YAML file uses an existing persistent volume claim (PVC) named demo-jindo-master-meta, which is used to mount an ESSD as a volume. This means that the metadata maintained by the master pod is persisted and can be migrated together with the pod. For more information, see Use JindoRuntimes to persist storage for the JindoFS master.

Policy 3: Properly allocate resources to the FUSE pod

The cache system client runs in the FUSE pod, which mounts the FUSE file system to the specified path in the application pod on the host and uses POSIX (Portable Operating System Interface) to expose data. This way, you can accelerate data access from application pods without the need to modify the application code. The access to remote storage is as fast as the access to local storage. The FUSE pod is responsible for maintaining the data tunnel between the application and the cache system. We recommend that you properly allocate resources to the FUSE pod.

Fluid allows you to use the spec.fuse.resources parameter of the cache runtime to allocate resources to the FUSE pod. If out of memory (OOM) errors occur in the FUSE pod, the FUSE file system may be unmounted from the application pod. To prevent this issue, we recommend that you do not specify a memory limit for the FUSE pod or specify a high memory limit for the FUSE pod. For example, you can set the memory limit to a value close to the amount of memory provided by the ECS instance.

spec:
  fuse:
    resources:
      requests:
        memory: 8Gi
    # limits:
    #   memory: <ECS_ALLOCATABLE_MEMORY>

Policy 4: Enable auto repair for the FUSE component to improve client availability

The FUSE pod is responsible for maintaining the data tunnel between the application and the cache system. By default, if the FUSE pod crashes due to unexpected reasons, the application pod can no longer use the FUSE file system to access data, even though the FUSE pod is restarted to run as normal. To resolve this issue, you must restart the application pod to restore data access. When the pod is restarted, the FUSE file system is remounted to the pod. However, this may affect the availability of the application.

Fluid provides the auto repair feature for FUSE. This feature can automatically restore data access to the FUSE file system for the application pod within a specific time period after the FUSE pod is restarted. In addition, you do not need to restart the application pod.

For more information about how to enable the auto repair feature of FUSE, see How to enable auto repair for FUSE. The auto repair feature of FUSE has certain limits. Therefore, we recommend that you enable this feature only in suitable scenarios, such as interactive program development scenarios where you develop applications by using online notebooks or other interactive programming environments.