Enable memory caching for a local directory

Background information

In most model inference scenarios, the EAS service mounts model files to a local directory by using OSS, NAS, or a Docker image. For more information, see Mount storage to services. When you perform operations, such as reading models, switching models, and scaling containers, the operations may be affected by the network bandwidth. In common scenarios such as scenarios in which Stable Diffusion is used, inference requests involve frequent switchover between basic models and Low-Rank Adaptation (LoRA) models. Each switchover requires reading models from OSS or NAS, which increases network latency.

To resolve this issue, EAS provides memory caching for local directories. The following figure shows the principles.

• You can cache model files in a local directory to memory.

• Caching supports the least recently used (LRU) policy and allows you to share files among instances. The cached files are displayed as the file system directory.

• The service directly reads the cached files in the memory. You can accelerate the operation without the need to modify business code.

• The instances of a service share a P2P network. When you scale out a service cluster, the added instances can read cached files from nearby instances over the P2P network to accelerate cluster scale-out.

Precautions

• To ensure data consistency, the cache directory is read-only.

• If you want to add a model file, you can add the model file to the source directory. The model file can be directly read from the cache directory.

• We recommend that you do not directly modify or delete model files in the source directory. Otherwise, dirty data may be cached.

Procedure

In this example, Stable Diffusion is used. The following parameters are configured for service deployment:

• Image startup command: ./webui.sh --listen --port 8000 --skip-version-check --no-hashing --no-download-sd-model --skip-prepare-environment --api --filebrowser

• OSS directory in which model files are stored: oss://path/to/models/

• Directory in which model files are stored in a container: /code/stable-diffusion-webui/data-slow

The /code/stable-diffusion-webui/data-slow directory is the source directory in which model files are stored. Then, the model files are mounted to the cache directory /code/stable-diffusion-webui/data-fast. This way, the service can directly read the model files that are stored in the source directory /code/stable-diffusion-webui/data-slow from the cache directory /code/stable-diffusion-webui/data-fast.

   {
    "cloud": {
        "computing": {
            "instances": [
                {
                    "type": "ml.gu7i.c16m60.1-gu30"
                }
            ]
        }
    },
    "containers": [
        {
            "image": "eas-registry-vpc.cn-hangzhou.cr.aliyuncs.com/pai-eas/stable-diffusion-webui:4.2",
            "port": 8000,
            "script": "./webui.sh --listen --port 8000 --skip-version-check --no-hashing --no-download-sd-model --skip-prepare-environment --api --filebrowser --ckpt-dir /code/stable-diffusion-webui/data-fast"
        }
    ],
    "metadata": {
        "cpu": 16,
        "enable_webservice": true,
        "gpu": 1,
        "instance": 1,
        "memory": 60000,
        "name": "sdwebui_test"
    },
    "options": {
        "enable_cache": true
    },
    "storage": [
        {
            "cache": {
                "capacity": "20G",
                "path": "/code/stable-diffusion-webui/data-slow"
            },
            "mount_path": "/code/stable-diffusion-webui/data-fast"
        },
        {
            "mount_path": "/code/stable-diffusion-webui/data-slow",
            "oss": {
                "path": "oss://path/to/models/",
                "readOnly": false
            },
            "properties": {
                "resource_type": "model"
            }
        }
    ]
}

Acceleration effect

The following table describes the acceleration effect in the sample model switching scenario in which Stable Diffusion is used. Unit: seconds. The acceleration effect varies based on the actual situation.

• If no model files exist in the memory cache, CacheFS automatically reads model files from the source directory. For example, if a file is mounted to an OSS bucket, CacheFS reads the file from the OSS bucket.

• If the service has multiple instances, the instance cluster shares the memory. When an instance loads the model, the instance can directly read the model files from the memory of other instances in the cluster. The amount of time required to read the file varies based on the file size.

• When you scale out a service cluster, the additional instances can automatically read the model files from the memory of other instances in the cluster during initialization. This way, the service is scaled out in a faster and more flexible manner.