Platform For AI:Scalable Job

Deploy a service

Deploy an inference service

The process is similar to the process of creating an asynchronous inference service. Prepare a service configuration file based on the following example:

{ 
  "containers": [
    {
      "image": "registry-vpc.cn-shanghai.aliyuncs.com/eas/eas-container-deploy-test:202010091755",
      "command": "/data/eas/ENV/bin/python /data/eas/app.py"
      "port": 8000,
    }
  ],
  "metadata": {
    "name": "scalablejob",
    "type": "ScalableJob",
    "rpc.worker_threads": 4,
    "instance": 1,
  }
}

Set type to ScalableJob to deploy the inference service as a scalable job. For information about other parameters, see Parameters of model services. For information about how to deploy an inference service, see Deploy online portrait service as a scalable job.

After you deploy the service, the system automatically creates a queue service and a scalable job and enables auto scaling.

Deploy a training service

The integrated deployment and independent deployment methods are supported. The following section describes how the deployment works and configuration details. For more information about the deployment method, see Deploy an auto-scaling Kohya training as a scalable job.

• How the deployment works

  

  • Integrated deployment: EAS creates a frontend service in addition to a queue service and a Scalable Job service. The frontend service is used to receive requests and forward requests to the queue service. The frontend service functions as a client of the Scalable Job service. The Scalable Job service is associated with a unique frontend service. In this case, the Scalable Job service can only execute training tasks forwarded by the associated frontend service.

  • Independent deployment: Independent deployment is suitable for multi-user scenarios. In this mode, the Scalable Job service is used as a common backend service and can be associated with multiple frontend services. Each user can send training tasks from their frontend service. The backend Scalable Job service creates corresponding Scalable Job instances to execute training tasks and each job instance can execute multiple training tasks based on the sequence that they are submitted. This allows resource sharing among multiple users. You do not need to create training tasks multiple times, which effectively reduces the costs.

• Configurations

  When you deploy a Scalable Job service, you need to provide a custom image. Note that if you want to deploy Kohya-based services, you can use the Kohya_ss image that is preset in EAS. The image must contain all dependencies of the training task. The image is used as the execution environment of the training task. You do not need to configure the startup command or port number. If you need to perform initialization tasks before the training task, you can configure initialization commands. EAS creates a separate process in the Scalable Job instance to perform initialization tasks. For information about how to prepare a custom image, see Deploy a model service by using a custom image. Example preset image in EAS:

   "containers": [
      {
        "image": "eas-registry-vpc.cn-shanghai.cr.aliyuncs.com/pai-eas/kohya_ss:2.2"
      }
    ]

  Integrated deployment

  Prepare a service configuration file. The following section provides a sample configuration file by using the preset Kohya_ss image in EAS.

  {
    "containers": [
      {
        "image": "eas-registry-vpc.cn-shanghai.cr.aliyuncs.com/pai-eas/kohya_ss:2.2"
      }
    ],
    "metadata": {
      "cpu": 4,
      "enable_webservice": true,
      "gpu": 1,
      "instance": 1,
      "memory": 15000,
      "name": "kohya_job",
      "type": "ScalableJobService"
    },
    "front_end": {
      "image": "eas-registry-vpc.cn-shanghai.cr.aliyuncs.com/pai-eas/kohya_ss:2.2",
      "port": 8001,
      "script": "python -u kohya_gui.py --listen 0.0.0.0 --server_port 8001 --data-dir /workspace --headless --just-ui --job-service"
    }
  }

  The following section describes the key parameters. For information about how to configure other parameters, see Parameters of model services.

  • Set type to ScalableJobService.

  • By default, the resource group used by the frontend service is the same as the resource group used by the Scalable Job service. The system allocates 2 vCPUs and 8 GB memory for the frontend service.

    • Customize the resource group or resources based on the following sample code:

      {
          "front_end": {
            "resource": "", # Modify the dedicated resource group that the frontend service uses. 
            "cpu": 4,
            "memory": 8000
          }
      }

    • Customize the instance type that is used for deployment based on the following sample code:

      {
          "front_end": {
            "instance_type": "ecs.c6.large"
          }
      }

  Independent deployment

  Prepare a service configuration file. The following section provides a sample configuration file by using the preset Kohya_ss image in EAS.

  {
    "containers": [
      {
        "image": "eas-registry-vpc.cn-shanghai.cr.aliyuncs.com/pai-eas/kohya_ss:2.2"
      }
    ],
    "metadata": {
      "cpu": 4,
      "enable_webservice": true,
      "gpu": 1,
      "instance": 1,
      "memory": 15000,
      "name": "kohya_job",
      "type": "ScalableJob"
    }
  }

  Set type to ScalableJobService. For information about how to configure other parameters, see Parameters of model services.

  In this mode, you need to deploy a frontend service and implement a proxy in the frontend service that forwards the received requests to the queue of the Scalable Job service. This way, the frontend service and the backend Scalable Job service are associated. For more information, see the "Send data to the queue service" section in the Access queue service topic.

Call the service

To distinguish between the training service and the inference service, you need to configure a taskType:command/query field when you call the Scalable Job service. Parameters:

• command: used to identify a training service.

• query: used to identify an inference service.

When you call a service by using HTTP or an SDK, you need to explicitly specify taskType. Example:

• Set tskType to query when you send an HTTP request:

  curl http://166233998075****.cn-shanghai.pai-eas.aliyuncs.com/api/predict/scalablejob?taskType={Wanted_TaskType} -H 'Authorization: xxx' -D 'xxx'

• Use the tags parameter to specify taskType when you call an SDK:

  # Create an input queue to send tasks or requests. 
  queue_client = QueueClient('166233998075****.cn-shanghai.pai-eas.aliyuncs.com', 'scalabejob')
  queue_client.set_token('xxx')
  queue_client.init()
  
  tags = {"taskType": "wanted_task_type"}
  # Send a task or request to the input queue. 
  index, request_id = inputQueue.put(cmd, tags)

Receive responses:

• Inference service: You can use the SDK for the EAS queue service to obtain the results in the output queue. For more information, see the "Subscribe to the queue service" section in the Access queue service topic.

• Training service: When you deploy the service, we recommend that you mount an Object Storage Service (OSS) bucket to store the training results in the OSS bucket for persistent storage. For more information, see Deploy a Kohya-based training as a scalable job.

Configure log collection

You can configure enable_write_log_to_queue to obtain real-time logs.

{
  "scalable_job": {
    "enable_write_log_to_queue": true
  }
}

• In training scenarios, this configuration is enabled by default. The system writes real-time logs back to the output queue. You can use the SDK of the EAS queue service to obtain training logs in real time. For more information, see Deploy an auto-scaling Kohya training as a scalable job.

• In inference scenarios, this configuration is disabled by default. The system outputs logs by using stdout.

References

For more information about the scenarios of the Scalable Job service, see the following topics:

• Deploy an auto-scaling Kohya training as a scalable job

• Deploy online portrait service as a scalable job