Platform For AI:Use Stable Diffusion web UI to deploy an AI painting service

Background information

Features and Benefits

The deployment in EAS provides the following features and benefits:

• Ease of use: You can quickly deploy a ready-to-use model service in EAS. EAS also allows you to dynamically switch underlying resources, such as GPUs based on your business requirements.

• Enterprise-level features: The individual mechanisms for frontend and backend services support multiple users for scheduling multi-GPUs, user isolation, and bill splitting.

• Plug-ins and optimization: PAI provides the open source edition of Stable Diffusion WebUI, the PAI-Blade tool, and the FileBrower and ModelZoo plugins. The PAI-Blade tool is used for performance optimization. The FileBrowser plug-in is used to upload models and images from or download models and images to on-premises clients.The self-developed ModelZoo plug-in is used to accelerate downloading of open source models.

Editions

The Stable Diffusion WebUI service provides the following editions:

• Standard Edition: suitable for a single user who wants to call the service by using the web UI or by calling the API of the service that is deployed on a single instance.

• Cluster Edition WebUI: suitable for group work scenarios in which multiple members use the web UI to generate images. The system creates an instance to allow the proxy to process requests from the web UI in addition to the inference service instances. Cluster Edition has the following benefits:

  • The system assigns a separate working directory for each user to facilitate storage and management of files, such as models and generated images or videos.

  • The system provides a separate backend environment for each user. If you configure multiple instances for the service, tasks of a single user run in sequence and tasks of multiple users are distributed among instances to implement efficient GPU sharing.

• API Edition: suitable for high-concurrency scenarios. The system automatically deploys the service as an asynchronous service. The system creates a queue service instance in addition to the asynchronous inference service instances.

• Serverless Edition: is available only in the China (Shanghai) and China (Hangzhou) regions. The deployment of a Serverless Edition service is provided free of charge. You are billed based on the duration of service calls. The system automatically scales the service based on the service requests. You can call the Serverless Edition service only by using the web UI.

Procedure

1. Deploy a model service

   Select a deployment method based on your business scenario.

2. Call the model service

   Select one of the following methods based on the edition you use:

   • Call the service by using the web UI

     Use the web UI to send a service request. This method is supported for Standard Edition, Cluster Edition and Serverless Edition.

   • Call the service by using API operations

     Send a service request by calling API operations. This method is supported only for Standard Edition and API Edition. Standard Edition supports synchronous service calls. API Edition supports asynchronous service calls.

3. Install plug-ins for enhanced features

   PAI provides various plug-ins to provide diverse features. You can view and install the plug-in on the Extensions tab of the web UI page.

Prerequisites

• An File Storage NAS (NAS) file system or Object Storage Service (OSS) bucket is created to store the model files and generated images.

  • Create a General-purpose NAS file system. For more information, see Create a NAS file system.

  • Create an OSS bucket. For more information, see Create a bucket.

• If you call a service by using the API operations, you must configure environment variables. For more information, see Configure access credentials.

Deploy a model service

The following deployment methods are supported:

Call the model service

Call the service by using the web UI

You can use the web UI to call a service of the Standard Edition, Cluster Edition WebUI, and Serverless Edition. Perform the following steps:

1. Find the service that you want to manage and click View Web App in the Service Type column.

2. Perform model inference.

   On the txt2img tab of the Stable Diffusion WebUI page, enter a prompt, such as cute dog, and then click Generate. The following figure provides an sample response.

   

Call the service by using API operations

You can use API operations to call services of Standard Edition or API Edition. API calls support synchronous and asynchronous service calls. The Standard Edition service supports only synchronous calls. The API Edition service supports synchronous and asynchronous service calls.

• Synchronous call: If you send a synchronous request, the client pauses execution and waits for the result to return.

• Asynchronous call: The client uses the queue service of EAS to send requests to the input queue and subscribes to the inference result in the output queue.

Synchronous call

After you deploy a Standard Edition or API Edition service, you can perform the following steps to send a synchronous request.

1. Obtain the call information.

   1. After you deploy a service, click the service name to go to the Service Details page.

   2. In the Basic Information section, click View Endpoint Information.

   3. In the Invocation Method dialog box, obtain the endpoint and token of the service.

      • If you deploy an API Edition service, obtain the endpoint and token of the service on the Public Endpoint tab of the Synchronous Call tab.

      • If you deploy a Standard Edition service, obtain the endpoint and token of the service on the Public Endpoint tab.

2. To send a synchronous request, use one of the following methods.

   Use a cURL command

   Sample file:

   curl --location --request POST '<service_url>/sdapi/v1/txt2img' \
   --header 'Authorization: <token>' \
   --header 'Content-Type: application/json' \
   --data-raw '{
     "prompt":"cut dog ",
     "steps":20
   }'

   Modify the following parameters in the preceding code:

   • <service_url>: Replace the value with the endpoint that you obtained in Step 1. Delete /at the end of the endpoint.

   • <token>: Set this parameter to the token that you obtained in Step 1.

   After you run the command, the system returns the Base64-encoded image.

   Use Python code

   For information about how to send requests to the SD API, see the GitHub project wiki.

   • Example 1 (recommended): We recommend that you mount an OSS bucket to the EAS service to save the generated images. In the following example, the OSS mount path is used in the request body to save the image to OSS, and the oss2 SDK is used to download the image from OSS to your on-premises device.

     import requests
     import oss2
     from oss2.credentials import EnvironmentVariableCredentialsProvider
     
     # Step 1: Send a request. The generated image is saved to OSS. 
     url = "<service_url>"
     
     # The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
     auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
     
     # In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
     bucket = oss2.Bucket(auth, '<endpoint>', '<examplebucket>')
     
     payload = {
         "alwayson_scripts": {
             "sd_model_checkpoint": "deliberate_v2.safetensors",
             "save_dir": "/code/stable-diffusion-webui/data/outputs"
         },
         "steps": 30,
         "prompt": "girls",
         "batch_size": 1,
         "n_iter": 2,
         "width": 576,
         "height": 576,
         "negative_prompt": "ugly, out of frame"
     }
     
     session = requests.session()
     session.headers.update({"Authorization": "<token>"})
     
     response = session.post(url=f'{url}/sdapi/v1/txt2img', json=payload)
     if response.status_code != 200:
         raise Exception(response.content)
     
     data = response.json()
     
     
     # Step 2: Obtain images from OSS and download the images to your on-premises device. 
     
     # The mount_path configuration for OSS that you specified when you deployed the service. 
     mount_path = "/code/stable-diffusion-webui/data"
     
     # The OSS path that you specified when you deployed the service. 
     oss_url = "oss://examplebucket/data-oss"
     
     for idx, img_path in enumerate(data['parameters']['image_url'].split(',')):
         # Obtain the actual URL of the generated image in OSS. 
         img_oss_path = img_path.replace(mount_path, oss_url)
         print(idx, img_oss_path, url)
     
         # Download the OSS object to an on-premises file system. Replace <examplebucket> with the name of the OSS bucket that you created. 
         bucket.get_object_to_file(img_oss_path[len("oss://examplebucket/"):], f'output-{idx}.png')

     The following table describes the key parameters.

     Parameter	Description
     url	Replace <service_url> with the endpoint that you obtained in Step 1. Delete / at the end of the endpoint.
     bucket	Take note of the following parameters: Replace <endpoint> with the endpoint that is used to access OSS. For example, the http://oss-cn-shanghai.aliyuncs.com endpoint is used for the China (Shanghai) region. If your service is deployed in another region, specify your actual endpoint. For more information, see OSS regions and endpoints. Replace <examplebucket> with the name of the OSS bucket that you created.
     <token>	Set this parameter to the token that you obtained in Step 1.
     mount_path	The OSS mount path that you configured when you deployed the service.
     oss_url	The OSS storage path that you configured when you deployed the service.

     If the code successfully runs, the following results are returned. You can go to the OSS console and view the generated image in the outputs directory which you mounted when you deployed the service.

     0 /code/stable-diffusion-webui/data/outputs/txt2img-grids/2024-06-26/grid-093546-9ad3f23e-a5c8-499e-8f0b-6effa75bd04f.png oss://examplebucket/data-oss/outputs/txt2img-grids/2024-06-26/grid-093546-9ad3f23e-a5c8-499e-8f0b-6effa75bd04f.png
     1 /code/stable-diffusion-webui/data/outputs/txt2img-images/2024-06-26/093536-ab4c6ab8-880d-4de6-91d5-343f8d97ea3c-3257304074.png oss://examplebucket/data-oss/outputs/txt2img-images/2024-06-26/093536-ab4c6ab8-880d-4de6-91d5-343f8d97ea3c-3257304074.png
     2 /code/stable-diffusion-webui/data/outputs/txt2img-images/2024-06-26/093545-6e6370d7-d41e-4105-960a-b4739af30c0d-3257304075.png oss://examplebucket/data-oss/outputs/txt2img-images/2024-06-26/093545-6e6370d7-d41e-4105-960a-b4739af30c0d-3257304075.png

   • Example 2: Save an image to an on-premises path. Run the following Python code to obtain the Base64-encoded image and save the image file to an on-premises directory.

     import requests
     import io
     import base64
     from PIL import Image, PngImagePlugin
     
     url = "<service_url>"
     
     payload = {
         "prompt": "puppy dog",
         "steps": 20,
         "n_iter": 2
     }
     
     session = requests.session()
     session.headers.update({"Authorization": "<token>"})
     
     
     response = session.post(url=f'{url}/sdapi/v1/txt2img', json=payload)
     if response.status_code != 200:
         raise Exception(response.content)
     
     data = response.json()
     
     # Obtain a Base64-encoded image file. We recommend that you obtain an image URL instead of the Base64-encoded image file, as shown in Example 2. 
     for idx, im in enumerate(data['images']):
         image = Image.open(io.BytesIO(base64.b64decode(im.split(",", 1)[0])))
     
         png_payload = {
             "image": "data:image/png;base64," + im
         }
         resp = session.post(url=f'{url}/sdapi/v1/png-info', json=png_payload)
     
         pnginfo = PngImagePlugin.PngInfo()
         pnginfo.add_text("parameters", resp.json().get("info"))
         image.save(f'output-{idx}.png', pnginfo=pnginfo)
     

     Take note of the following parameters:

     • <service_url>: Replace the value with the endpoint that you obtained in Step 1. Delete /at the end of the endpoint.

     • <token>: Set this parameter to the token that you obtained in Step 1.

   You can also use LoRA and ControlNet data in the request to enable a specific functionality.

   LoRA configurations

   Add <lora:yaeMikoRealistic_Genshin:1000> in the prompt parameter to use LoRA models. For more information, see LORA and alwayson_scripts example.

   Sample request body:

   {
     "prompt":"girls <lora:yaeMikoRealistic_Genshin:1>",
     "steps":20,
     "save_images":true
   }

   ControlNet configurations

   You can add the controlnet parameter in the request to perform common operations on the generated image, such as keeping the image horizontal or vertical. For more information, see the "Example: txt2img with ControlNet" section of this topic.

Call an asynchronous service

After you deploy an API Edition service, you can send an asynchronous request to the service. The client subscribes to the inference result in the output queue. Perform the following steps:

1. Obtain the invocation information.

   Click Invocation Method in the Service Type column of the service. On the Public Endpoint tab of the Asynchronous Call tab, view the service endpoint and token.

2. Send asynchronous requests. You can use the SDK for Python or SDK for Java.

   Important

   • The queue service requires that input or output queues cannot exceed 8 KB in size. Take note of the following items:

     • If the request data contains an image, we recommend that you use a URL to pass the image data. SD WebUI automatically downloads and parses the image data.

     • To ensure that the response does not contain original image data, we recommend that you use the save_dir parameter to specify the path where the generated image is saved. For more information, see the "What parameters can I configure for API operations?" section of this topic.

   • By default, EAS cannot access the Internet. If you set the image_link parameter to an Internet URL, you must configure network connectivity and Internet access so that EAS can access the image.

   Method 1: Use SDK for Python

   Sample code:

   import requests
   
   url = "<service_url>"
   session = requests.session()
   session.headers.update({"Authorization": "<token>"})
   
   prompts = ["cute dog", "cute cat", "cute girl"]
   
   for i in range(5):
       p = prompts[i % len(prompts)]
       payload = {
           "prompt": p,
           "steps": 20,
           "alwayson_scripts": {
               "save_dir": "/code/stable-diffusion-webui/data-oss/outputs/txt2img"
           },
       }
       response = session.post(url=f'{url}/sdapi/v1/txt2img?task_id=txt2img_{i}', json=payload)
       if response.status_code != 200:
           exit(f"send request error:{response.content}")
       else:
           print(f"send {p} success, index is {response.content}")
   
   
   for i in range(5):
       p = prompts[i % len(prompts)]
       payload = {
           "prompt": p,
           "steps": 20,
           "alwayson_scripts": {
               "save_dir": "/code/stable-diffusion-webui/data-oss/outputs/img2img",
               "image_link": "https://eas-cache-cn-hangzhou.oss-cn-hangzhou-internal.aliyuncs.com/stable-diffusion-cache/tests/boy.png",
           },
       }
       response = session.post(url=f'{url}/sdapi/v1/img2img?task_id=img2img_{i}', json=payload)
       if response.status_code != 200:
           exit(f"send request error:{response.content}")
       else:
           print(f"send {p} success, index is {response.content}")
   

   Take note of the following parameters:

   • <service_url>: Replace the value with the endpoint that you obtained in Step 1. Delete / at the end of the endpoint.

   • <token>: Replace the value with the token that you obtained in Step 1.

   Note

   • You can use SDK for Python to send POST requests to the API endpoints provided by SD WebUI. Select an endpoint based on your business requirements.

   • If you want to pass custom information to the service, specify a custom tag by using a URL parameter. For example, you can append ?task_id=task_abc to the request path to specify a tag named task_id. The tag information is included in the result specified by the tags parameter.

   If the code successfully runs, the following result is returned. Your actual result may vary.

   Method 2: Use SDK for Java

   Maven is used to manage Java projects. You must add EAS SDK for Java as a dependency in the pom.xml file. For more information, see SDK for Java.

   The following sample code provides an example on how to send asynchronous requests.

   import com.aliyun.openservices.eas.predict.http.HttpConfig;
   import com.aliyun.openservices.eas.predict.http.QueueClient;
   import com.aliyun.openservices.eas.predict.queue_client.QueueUser;
   import org.apache.commons.lang3.tuple.Pair;
   
   import java.util.HashMap;
   
   public class SDWebuiAsyncPutTest {
       public static void main(String[] args) throws Exception {
           // Create a client for the queue service. 
           String queueEndpoint = "http://166233998075****.cn-hangzhou.pai-eas.aliyuncs.com";
           String queueToken = "xxxxx==";
           // The name of the input queue consists of the service name and the request path that you want to use. 
           String inputQueueName = "<service_name>/sdapi/v1/txt2img";
   
           // Create the input queue. After you add quest data to the input queue, the inference service automatically reads the request data from the input queue. 
           QueueClient inputQueue =
               new QueueClient(queueEndpoint, inputQueueName, queueToken, new HttpConfig(), new QueueUser());
           // Clear queue data. Proceed with caution. 
           // input_queue.clear();
   
           // Add request data to the input queue. 
           int count = 5;
           for (int i = 0; i < count; ++i) {
               // Create request data. 
               String data = "{\n" +
                   "    \"prompt\": \"cute dog\", \n" +
                   "    \"steps\":20,\n" +
                   "    \"alwayson_scripts\":{\n" +
                   "        \"save_dir\":\"/code/stable-diffusion-webui/data-oss/outputs/txt2img\"\n" +
                   "    }\n" +
                   "  }";
               // Create a custom tag. 
               HashMap<String, String> map = new HashMap<String, String>(1);
               map.put("task_id", "txt2img_" + i);
               Pair<Long, String> entry = inputQueue.put(data.getBytes(), map);
   
               System.out.println(String.format("send success, index is %d, request_id is %s", entry.getKey(), entry.getValue()));
               // The queue service supports multi-priority queues. You can use the put function to set the priority level of the request. The default value is 0. A value of 1 specifies a high priority. 
               //  inputQueue.put(data.getBytes(), 0L, null);
           }
           // Close the client. 
           inputQueue.shutdown();
   
   
           inputQueueName = "<service_name>/sdapi/v1/img2img";
           inputQueue =
               new QueueClient(queueEndpoint, inputQueueName, queueToken, new HttpConfig(), new QueueUser());
           for (int i = 0; i < count; ++i) {
               // Create request data. 
               String data = "{\n" +
                   "    \"prompt\": \"cute dog\", \n" +
                   "    \"steps\":20,\n" +
                   "    \"alwayson_scripts\":{\n" +
                   "        \"save_dir\":\"/code/stable-diffusion-webui/data-oss/outputs/img2img\",\n" +
                   "        \"image_link\":\"https://eas-cache-cn-hangzhou.oss-cn-hangzhou-internal.aliyuncs.com/stable-diffusion-cache/tests/boy.png\"\n" +
                   "    }\n" +
                   "  }";
               HashMap<String, String> map = new HashMap<String, String>(1);
               map.put("task_id", "img2img_" + i);
               Pair<Long, String> entry = inputQueue.put(data.getBytes(), map);
   
               System.out.println(String.format("send success, index is %d, requestId is %s", entry.getKey(), entry.getValue()));
           }
   
           // Close the client. 
           inputQueue.shutdown();
       }
   }

   The following table describes the key parameters.

   • queueEndpoint: Set the value to the endpoint that you obtained in Step 1. Refer to the sample code to configure this parameter.

   • queueToken: Set the value to the token that you obtained in Step 1.

   • <service_name>: Set the value to the name of the asynchronous service that you deployed.

   Note

   If you want to pass custom information to the service, specify a custom tag in the put function. You can refer to the sample code to configure custom tags. The tag information is included in the result within the tags parameter.

   If the code successfully runs, the following result is returned. Your actual result may vary.

   send success, index is 21, request_id is 05ca7786-c24e-4645-8538-83d235e791fe
   send success, index is 22, request_id is 639b257a-7902-448d-afd5-f2641ab77025
   send success, index is 23, request_id is d6b2e127-eba3-4414-8e6c-c3690e0a487c
   send success, index is 24, request_id is 8becf191-962d-4177-8a11-7e4a450e36a7
   send success, index is 25, request_id is 862b2d8e-5499-4476-b3a5-943d18614fc5
   send success, index is 26, requestId is 9774a4ff-f4c8-40b7-ba43-0b1c1d3241b0
   send success, index is 27, requestId is fa536d7a-7799-43f1-947f-71973bf7b221
   send success, index is 28, requestId is e69bdd32-5c7b-4c8f-ba3e-e69d2054bf65
   send success, index is 29, requestId is c138bd8f-be45-4a47-a330-745fd1569534
   send success, index is 30, requestId is c583d4f8-8558-4c8d-95f7-9c3981494007
   
   Process finished with exit code 0

3. Subscribe to the results of the asynchronous requests.

   After the server completes processing related requests, the server automatically pushes the results to the client for efficient asynchronous communication. You can use SDK for Python or SDK for Java to subscribe to the results.

   Method 1: Use SDK for Python

   Sample code:

   import json
   import oss2
   
   from oss2.credentials import EnvironmentVariableCredentialsProvider
   from eas_prediction import QueueClient
   
   sink_queue = QueueClient('139699392458****.cn-hangzhou.pai-eas.aliyuncs.com', 'sd_async/sink')
   sink_queue.set_token('<token>')
   sink_queue.init()
   
   mount_path = "/code/stable-diffusion-webui/data-oss"
   oss_url = "oss://<examplebucket>/aohai-singapore"
   # The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
   auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
   # In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
   bucket = oss2.Bucket(auth, 'http://oss-cn-hangzhou.aliyuncs.com', '<examplebucket>')
   
   
   watcher = sink_queue.watch(0, 5, auto_commit=False)
   for x in watcher.run():
       if 'task_id' in x.tags:
           print('index {} task_id is {}'.format(x.index, x.tags['task_id']))
       print(f'index {x.index} data is {x.data}')
       sink_queue.commit(x.index)
       try:
           data = json.loads(x.data.decode('utf-8'))
           for idx, path in enumerate(data['parameters']['image_url'].split(',')):
               url = path.replace(mount_path, oss_url)
               # Download the OSS object to an on-premises file system. 
               bucket.get_object_to_file(url[len("oss://<examplebucket>/"):], f'{x.index}-output-{idx}.png')
               print(f'save {url} to {x.index}-output-{idx}.png')
       except Exception as e:
           print(f'index {x.index} process data error {e}')
   

   The following table describes the key parameters.

   Parameter	Description
   sink_queue	Replace 139699392458****.cn-hangzhou.pai-eas.aliyuncs.com with the endpoint that you obtained in the preceding step in the format shown in the sample code. Replace sd_async with the name of the asynchronous service that you deployed in the format shown in the sample code.
   <token>	Set the value to the service token that you obtained in Step 1.
   oss_url	Set the value to the OSS path that you specified when you deployed the service.
   bucket	Take note of the following parameters: Use the endpoint that is used to access OSS. In this example, http://oss-cn-hangzhou.aliyuncs.com is used for the China (Hangzhou) region. If your service is deployed in another region, configure the parameter based on actual situation. For more information, see OSS regions and endpoints. Replace <examplebucket> with the name of the OSS bucket that you created.

   Note

   • You can manually commit the data or set the auto_commit parameter to true to automatically commit the data.

   • If the client of the queue service stops consuming data, we recommend that you close the client to release resources.

   • You can also run a cURL command or call an API operation to subscribe to the results. For more information, see Asynchronous inference and queue service.

   The client continuously listens for results from the server by using the watcher.run() method. If the server returns no result, the client keeps waiting. If the server returns a result, the client prints the result. If the code successfully runs, the following result is returned. Your actual results may vary. You can go to the OSS console and view the generated images in the OSS path that you specified when you deployed the service.

   index 1 task_id is txt2img_0
   index 1 data is b'{"images":[],"parameters":{"id_task":null,"status":0,"image_url":"/code/stable-diffusion-webui/data/outputs/txt2img/txt2img-images/2024-07-01/075825-a2abd45f-3c33-43f2-96fb-****50329671-1214613912.png","seed":"1214613912","error_msg":"","image_mask_url":""},"info":"{\\"hostname\\": \\"***-8aff4771-5c86c8d656-hvdb8\\"}"}'
   save oss://examplebucket/xx/outputs/txt2img/txt2img-images/2024-07-01/075825-a2abd45f-3c33-43f2-96fb-****50329671-1214613912.png to 1-output-0.png
   index 2 task_id is txt2img_1
   index 2 data is b'{"images":[],"parameters":{"id_task":null,"status":0,"image_url":"/code/stable-diffusion-webui/data/outputs/txt2img/txt2img-images/2024-07-01/075827-c61af78c-25f2-47cc-9811-****aa51f5e4-1934284737.png","seed":"1934284737","error_msg":"","image_mask_url":""},"info":"{\\"hostname\\": \\"***-8aff4771-5c86c8d656-hvdb8\\"}"}'
   save oss://examplebucket/xx/outputs/txt2img/txt2img-images/2024-07-01/075827-c61af78c-25f2-47cc-9811-****aa51f5e4-1934284737.png to 2-output-0.png
   index 3 task_id is txt2img_2
   index 3 data is b'{"images":[],"parameters":{"id_task":null,"status":0,"image_url":"/code/stable-diffusion-webui/data/outputs/txt2img/txt2img-images/2024-07-01/075829-1add1f5c-5c61-4f43-9c2e-****9d987dfa-3332597009.png","seed":"3332597009","error_msg":"","image_mask_url":""},"info":"{\\"hostname\\": \\"***-8aff4771-5c86c8d656-hvdb8\\"}"}'
   save oss://examplebucket/xx/outputs/txt2img/txt2img-images/2024-07-01/075829-1add1f5c-5c61-4f43-9c2e-****9d987dfa-3332597009.png to 3-output-0.png
   index 4 task_id is txt2img_3
   index 4 data is b'{"images":[],"parameters":{"id_task":null,"status":0,"image_url":"/code/stable-diffusion-webui/data/outputs/txt2img/txt2img-images/2024-07-01/075832-2674c2d0-8a93-4cb5-9ff4-****46cec1aa-1250290207.png","seed":"1250290207","error_msg":"","image_mask_url":""},"info":"{\\"hostname\\": \\"***-8aff4771-5c86c8d656-hvdb8\\"}"}'
   save oss://examplebucket/xx/outputs/txt2img/txt2img-images/2024-07-01/075832-2674c2d0-8a93-4cb5-9ff4-****46cec1aa-1250290207.png to 4-output-0.png
   index 5 task_id is txt2img_4
   index 5 data is b'{"images":[],"parameters":{"id_task":null,"status":0,"image_url":"/code/stable-diffusion-webui/data/outputs/txt2img/txt2img-images/2024-07-01/075834-8bb15707-ff0d-4dd7-b2da-****27717028-1181680579.png","seed":"1181680579","error_msg":"","image_mask_url":""},"info":"{\\"hostname\\": \\"***-8aff4771-5c86c8d656-hvdb8\\"}"}'
   save oss://examplebucket/xx/outputs/txt2img/txt2img-images/2024-07-01/075834-8bb15707-ff0d-4dd7-b2da-****27717028-1181680579.png to 5-output-0.png

   Method 2: Use SDK for Java

   Sample code:

   import com.aliyun.openservices.eas.predict.http.HttpConfig;
   import com.aliyun.openservices.eas.predict.http.QueueClient;
   import com.aliyun.openservices.eas.predict.queue_client.DataFrame;
   import com.aliyun.openservices.eas.predict.queue_client.QueueUser;
   import com.aliyun.openservices.eas.predict.queue_client.WebSocketWatcher;
   
   public class SDWebuiAsyncWatchTest {
       public static void main(String[] args) throws Exception {
           // Create a client for the queue service. 
           String queueEndpoint = "http://166233998075****.cn-hangzhou.pai-eas.aliyuncs.com";
           String queueToken = "xxxxx==";
           // The name of the output queue consists of the service name and "/sink". 
           String sinkQueueName = "<service_name>/sink";
   
           // The output queue. The inference service processes the input data and writes the results to the output queue. 
           QueueClient sinkQueue =
               new QueueClient(queueEndpoint, sinkQueueName, queueToken, new HttpConfig(), new QueueUser());
   
           // Clear queue data. Proceed with caution. 
           //  sinkQueue.clear();
   
           // Subscribe to the queue and obtain the results. 
           WebSocketWatcher watcher = sinkQueue.watch(0L, 5L, false, false, null);
           try {
               while (true) {
                   DataFrame df = watcher.getDataFrame();
                   if (df.getTags().containsKey("task_id")) {
                       System.out.println(String.format("task_id = %s", df.getTags().get("task_id")));
                   }
                   System.out.println(String.format("index = %d, data = %s, requestId = %s", df.getIndex(), new String(df.getData()), df.getTags().get("requestId")));
                   sinkQueue.commit(df.getIndex());
               }
           } catch (Exception e) {
               System.out.println("watch error:" + e.getMessage());
               e.printStackTrace();
               watcher.close();
           }
   
           // Close the client. 
           sinkQueue.shutdown();
       }
   }

   Take note of the following parameters:

   • queueEndpoint: Set the value to the endpoint that you obtained in Step 1. You can refer to the sample code to configure this parameter.

   • queueToken: Set the value to the token that you obtained in Step 1.

   • <service_name>: Set the value to the name of the asynchronous service that you deployed.

   Note

   • You can manually commit the data or set the auto_commit parameter to true to automatically commit the data.

   • If the client of the queue service stops consuming data, we recommend that you close the client to release resources.

   • You can also run a cURL command or call an API operation to subscribe to the results. For more information, see Asynchronous inference and queue service.

   The client continuously listens for results from the server by using the watcher.getDataFrame() method. If the server returns no result, the client keeps waiting. If the server returns a result, the client prints the result. If the code successfully runs, the following result is returned. Your actual results may vary. You can go to the OSS console and view the generated images in the OSS path that you specified when you deployed the service.

   2023-08-04 16:17:31,497 INFO [com.aliyun.openservices.eas.predict.queue_client.WebSocketWatcher] - WebSocketClient Successfully Connects to Server: 1396993924585947.cn-hangzhou.pai-eas.aliyuncs.com/116.62.XX.XX:80
   task_id = txt2img_0
   index = 21, data = {"images":[],"parameters":{"id_task":null,"status":0,"image_url":"/code/stable-diffusion-webui/data-oss/outputs/txt2img/txt2img-images/2023-08-04/54363a9d-24a5-41b5-b038-2257d43b8e79-412510031.png","seed":"412510031","error_msg":"","total_time":2.5351321697235107},"info":""}, requestId = 05ca7786-c24e-4645-8538-83d235e791fe
   task_id = txt2img_1
   index = 22, data = {"images":[],"parameters":{"id_task":null,"status":0,"image_url":"/code/stable-diffusion-webui/data-oss/outputs/txt2img/txt2img-images/2023-08-04/0c646dda-4a53-43f4-97fd-1f507599f6ae-2287341785.png","seed":"2287341785","error_msg":"","total_time":2.6269655227661133},"info":""}, requestId = 639b257a-7902-448d-afd5-f2641ab77025
   task_id = txt2img_2
   index = 23, data = {"images":[],"parameters":{"id_task":null,"status":0,"image_url":"/code/stable-diffusion-webui/data-oss/outputs/txt2img/txt2img-images/2023-08-04/4d542f25-b9cc-4548-9db2-5addd0366d32-1158414078.png","seed":"1158414078","error_msg":"","total_time":2.6604185104370117},"info":""}, requestId = d6b2e127-eba3-4414-8e6c-c3690e0a487c
   task_id = txt2img_3

   After you send an asynchronous request and subscribe to the results, you can use search() methods to query the request status. For more information, see SDK for Python and SDK for Java.

EAS provides additional parameters on top of the SD WebUI API. You can configure these parameters to use advanced features and meet custom business requirements. For more information, see the "What parameters can I configure for API operations?" section of this topic.

Install extensions for enhanced features

You can configure extensions for Stable Diffusion WebUI to enable various features. PAI provides various preset extensions, such as the BeautifulPrompt extension, which is used to expand and polish a prompt. The following section uses the BeautifulPrompt extension as an example to describe how to install and use the extension.

Install the extension

You can view and install the extension on the Extensions tab of the web UI page. Perform the following steps:

1. Find the service that you want to manage and click View Web App in the Service Type column.

2. On the Extensions tab of the web UI page, check whether BeautifulPrompt is selected. If the extension is not selected, select the check box in front of BeautifulPrompt and click Apply and restart UI to load the BeautifulPrompt extension.

   When you install the extension, the web UI automatically restarts. After the extension is reloaded, you can perform model inference to check the effect.

Use the extension for inference

1. On the BeautifulPrompt tab, enter a simple prompt in the field and click Generate to generate a detailed prompt.

   PAI provides multiple prompt generation models. The prompt generated by each model varies. Where:

   • pai-bloom-1b1-text2prompt-sd-v2: suitable for generating prompts in complex scenarios.

   • pai-bloom-1b1-text2prompt-sd: suitable for generating prompts that describe a single object.

   You can select a model based on your business requirements.

2. Select the prompt that you want to use and click to txt2img next to the generated prompt.

   The txt2img tab appears and the prompt that you selected is automatically specified.

3. Click Generate to generate an image on the right side of the web UI page.

   BeautifulPrompt helps you improve the image quality and add more details to the image. The following table compares the effects before and after the BeautifulPrompt extension is used:

   Prompt that you enter	Without BeautifulPrompt	With BeautifulPrompt
   a cat
   a giant tiger

FAQ

How do I use my model and output directory?

After you deploy the service, the system automatically creates the following directory structure in the mounted OSS bucket or NAS file system:

Take note of the following parameters:

• models: This directory is used to store model files.

• After you send an inference request, the system automatically generates the result file to this directory based on the preset configurations in the API code.

To load and use a new model, you can store a LoRA or Stable Diffusion model that you downloaded from the open source community or trained in the preceding specified directory. Perform the following steps:

1. Upload the model file to the subdirectory of the models directory. For more information, see the "Upload an object" section in the Get started by using the OSS console topic.

2. On the Elastic Algorithm Service (EAS) page, find the service that you want to manage and click /> Restart Service in the Actions column. After the service restarts, the configuration takes effect.

3. On the Stable Diffusion web UI page, switch the model and perform model inference.

   

What do I do if the service freezes for a long period of time?

• You can reopen the Stable Diffusion web application page or restart the EAS service. Perform the following steps:

  • Find the service that you want to manage and click View Web App in the Service Type column to reopen the web application page.

  • Click />Restart Service in the Actions column of the service to restart the EAS service.

• If the issue persists, the service may need to download models or plug-ins from the Internet. By default, EAS is not connected to the Internet. You can load an image or mount a model offline. However, you may need Internet connection to download specific plug-ins. In this case, we recommend that you find the download path of the model or plug-in by viewing the service logs, download the model, and then upload and mount the model to OSS. For more information, see How do I use my model and output directory?. If you require Internet connection, you can configure network settings. For more information, see Configure network connectivity.

How do I change the default language of the web application interface to English?

1. On the web UI page, click 设置.

2. In the left-side navigation pane, click 用户界面. In the lower part of the 本地化 page, select 无.

3. In the upper part of the Stable Diffusion WebUI page, click 保存设置, and then click 重载前端.

   Refresh the web application interface. The page is displayed in English.

How do I manage my file system?

When you deploy a service of Standard Edition, API Edition, or Cluster Edition WebUI, the system automatically adds the --filebrowser parameter to the command. You can directly manage your file system by using the web UI. Perform the following steps:

1. After the service is deployed, click View Web App in the Service Type column.

2. On the web UI page, click the FileBrowser tab. You can view the file system, upload on-premises files to the file system, or download files to your on-premises computer.

   

What parameters can I configure when I create a service?

• Common parameters

  Parameter	Description	Usage notes
  --blade	Enables PAI-Blade to accelerate the image generation.	We recommend that you enable the feature.
  --filebrowser	Allows you to upload and download models or images.	By default, this feature is enabled.
  --data-dir /code/stable-diffusion-webui/data-oss	The path used to mount the persistent storage.	The default path is /code/stable-diffusion-webui/. You can also use a relative path.
  --api	The API calling mode of the web UI.	By default, this feature is enabled.
  --enable-nsfw-censor	By default, this feature is disabled. If you require security compliance, you can enable the content moderation feature.	Enable the feature based on your business requirements.
  --always-hide-tabs	Hide specific tabs.	Enable the feature based on your business requirements.
  --min-ram-reserved 40 --sd-dynamic-cache	Cache the Stable Diffusion model to memory.	N/A

• Cluster Edition parameters

  Note

  The ckpt and ControlNet models automatically load files in the public directory and the custom files.

  Parameter	Description	Usage notes
  --lora-dir	Specifies the public LoRA model directory. Example: --lora-dir /code/stable-diffusion-webui/data-oss/models/Lora.	By default, this parameter is not configured. All LoRA directories of the user are isolated and only LoRA models in the user folder are loaded. If you specify a directory, all users load the LoRA models in the public directory and the LoRA models in the user folder.
  --vae-dir	Specifies the public Variational Autoencoder (VAE) model directory. Example: --vae-dir /code/stable-diffusion-webui/data-oss/models/VAE.	By default, this parameter is not configured. All VAE directories of the user are isolated and only VAE models in the user folder are loaded. If you specify a directory, all users load the VAE models in the public directory.
  --gfpgan-dir	Specifies the public GFPGAN model directory. Example: --gfpgan-dir /code/stable-diffusion-webui/data-oss/models/GFPGAN.	By default, this parameter is not configured. All GFPGAN directories of the user are isolated and only GFPGAN models in the user folder are loaded. If you specify a directory, all users load the GFPGAN models in the public directory.
  --embeddings-dir	Specifies the public embeddings model directory. Example: --embeddings-dir /code/stable-diffusion-webui/data-oss/embeddings.	By default, this parameter is not configured. All embeddings directories of the user are isolated and only embedding models in the user folder are loaded. If you specify a directory, all users load the embedding models in the public directory.
  --hypernetwork-dir	Specify the public hypernetwork model directory. Example: --hypernetwork-dir /code/stable-diffusion-webui/data-oss/models/hypernetworks.	By default, this parameter is not configured. All hypernetwork directories of the user are isolated and only embedding models in the user folder are loaded. If you specify a directory, all users load the hypernetwork models in the public directory.
  --root-extensions	Uses the extension directory as a public directory. If you configure this parameter, all users can see the same extensions.	If you want to install or manage extensions in a centralized manner, use this parameter.

What parameters can I configure for an API operation?

EAS provides additional parameters on top of the SD WebUI API. You can configure these parameters to use advanced features and meet custom business requirements.

• Configure the SD model, the VAE model, and the path to save the generated images.

• Use URL parameters to send requests, for which status codes are returned.

• Access the generated images by using URLs, including the images that are processed by ControlNet.

Sample code:

{
      "alwayson_scripts": {
          "sd_model_checkpoint": "deliberate_v2.safetensors",  
          "save_dir": "/code/stable-diffusion-webui/data-oss/outputs",
          "sd_vae": "Automatic"
      },
      "steps": 20,
      "prompt": "girls",          
      "batch_size": 1,                                            
      "n_iter": 2,                                                 
      "width": 576, 
      "height": 576,
      "negative_prompt": "ugly, out of frame"
  }

# Sample asynchronous request: 

curl --location --request POST '<service_url>/sdapi/v1/txt2img' \
--header 'Authorization: <token>' \
--header 'Content-Type: application/json' \
--data-raw '{
      "alwayson_scripts": {
          "sd_model_checkpoint": "deliberate_v2.safetensors",
          "save_dir": "/code/stable-diffusion-webui/data-oss/outputs",
          "sd_vae": "Automatic"
      },
      "prompt": "girls",          
      "batch_size": 1,                                            
      "n_iter": 2,                                                 
      "width": 576, 
      "height": 576,
      "negative_prompt": "ugly, out of frame"
  }'

{
  "images": [],
  "parameters": {
    "id_task": "14837",
    "status": 0,
    "image_url": "/code/stable-diffusion-webui/data-oss/outputs/txt2img-grids/2023-07-24/grid-29a67c1c-099a-4d00-8ff3-1ebe6e64931a.png,/code/stable-diffusion-webui/data-oss/outputs/txt2img-images/2023-07-24/74626268-6c81-45ff-90b7-faba579dc309-1146644551.png,/code/stable-diffusion-webui/data-oss/outputs/txt2img-images/2023-07-24/6a233060-e197-4169-86ab-1c18adf04e3f-1146644552.png",
    "seed": "1146644551,1146644552",
    "error_msg": "",
    "total_time": 32.22393465042114
  },
  "info": ""
}

# Send the request data to the input queue. 
curl --location --request POST '<service_url>/sdapi/v1/txt2img' \
--header 'Authorization: <token>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "alwayson_scripts": {
        "sd_model_checkpoint": "deliberate_v2.safetensors",
        "id_task": "14837",
        "uid": "123",
        "save_dir": "tmp/outputs"
    },
    "prompt": "girls",
    "batch_size": 1,
    "n_iter": 2,
    "width": 576,
    "height": 576,
    "negative_prompt": "ugly, out of frame"
}'

{
    "alwayson_scripts": {
        "image_link":"https://eas-cache-cn-hangzhou.oss-cn-hangzhou-internal.aliyuncs.com/stable-diffusion-cache/tests/boy.png",
        "sd_model_checkpoint": "deliberate_v2.safetensors",
        "sd_vae": "Automatic",
        "save_dir": "/code/stable-diffusion-webui/data-oss/outputs"
    },
    "prompt": "girl",
    "batch_size": 1,                                            
    "n_iter": 2,                                                 
    "width": 576, 
    "height": 576,
    "negative_prompt": "ugly, out of frame",
    "steps": 20, # Sampling steps
    "seed": 111,   
    "subseed": 111, # Variation seed
    "subseed_strength": 0, # Variation strength
    "seed_resize_from_h": 0, # Resize seed from height
    "seed_resize_from_w": 0, # Resize seed from width
    "seed_enable_extras": false, # Extra
    "sampler_name": "DDIM", # Sampling method
    "cfg_scale": 7.5, # CFG Scale
    "restore_faces": true, # Restore faces
    "tiling": false, # Tiling
    "init_images": [], # image base64 str, default None
    "mask_blur": 4, # Mask blur
    "resize_mode": 1, # 0 just resize, 1 crop and resize, 2 resize and fill, 3 just resize
    "denoising_strength": 0.75, # Denoising strength
    "inpainting_mask_invert": 0, #int, index of ['Inpaint masked', 'Inpaint not masked'], Mask mode
    "inpainting_fill": 0, #index of ['fill', 'original', 'latent noise', 'latent nothing'], Masked content
    "inpaint_full_res": 0, # index of ["Whole picture", "Only masked"], Inpaint area
    "inpaint_full_res_padding": 32, #minimum=0, maximum=256, step=4, value=32, Only masked padding, pixels
    #"image_cfg_scale": 1, # resized by scale
    #"script_name": "Outpainting mk2", # The name of the script. Skip this field if you do not use a script.
    #"script_args": ["Outpainting", 128, 8, ["left", "right", "up", "down"], 1, 0.05] # The parameters of the script in the following order: fixed fields, pixels, mask_blur, direction, noise_q, and color_variation.
}

{
    "images":[],
    "parameters":{
        "id_task":"14837",
        "status":0,
        "image_url":"/data/api_test/img2img-grids/2023-06-05/grid-0000.png,/data/api_test/img2img-images/2023-06-05/00000-1003.png,/data/api_test/img2img-images/2023-06-05/00001-1004.png",
        "seed":"1003,1004",
        "error_msg":""
    },
    "info":""
}

{
    "alwayson_scripts": {
        "sd_model_checkpoint": "deliberate_v2.safetensors", # The name of the model. 
        "save_dir": "/code/stable-diffusion-webui/data-oss/outputs",
        "controlnet":{
            "args":[
                {
                    "image_link": "https://pai-aigc-dataset.oss-cn-hangzhou.aliyuncs.com/pixabay_images/00008b87bf3ff6742b8cf81c358b9dbc.jpg",
                    "enabled": true, 
                    "module": "canny", 
                    "model": "control_v11p_sd15_canny", 
                    "weight": 1, 
                    "resize_mode": "Crop and Resize", 
                    "low_vram": false, 
                    "processor_res": 512, 
                    "threshold_a": 100, 
                    "threshold_b": 200, 
                    "guidance_start": 0, 
                    "guidance_end": 1, 
                    "pixel_perfect": true, 
                    "control_mode": "Balanced", 
                    "input_mode": "simple", 
                    "batch_images": "", 
                    "output_dir": "", 
                    "loopback": false
                }
            ]
        }
    },
    # Key parameters.
    "prompt": "girls",          
    "batch_size": 1,                                            
    "n_iter": 2,                                                 
    "width": 576, 
    "height": 576,
    "negative_prompt": "ugly, out of frame"
}

{
    "images":[],
    "parameters":{
        "id_task":"14837",
        "status":0,
        "image_url":"/data/api_test/txt2img-grids/2023-06-05/grid-0007.png,/data/api_test/txt2img-images/2023-06-05/00014-1003.png,/data/api_test/txt2img-images/2023-06-05/00015-1004.png",
        "seed":"1003,1004",
        "error_msg":"",
        "image_mask_url":"/data/api_test/controlnet_mask/2023-06-05/00000.png,/data/api_test/controlnet_mask/2023-06-05/00001.png"
    },
    "info":""
}

References

You can also use EAS to deploy the following items:

• You can deploy an LLM application that can be called by using the web UI or API operations. After the LLM application is deployed, use the LangChain framework to integrate enterprise knowledge bases into the LLM application and implement intelligent Q&A and automation features. For more information, see Quickly deploy LLMs in EAS.

• You can deploy an AI video generation model service by using ComfyUI and Stable Video Diffusion models. This helps you complete tasks such as short video generation and animation on social media platforms. For more information, see Use ComfyUI to deploy an AI video generation model service.

• You can deploy a Retrieval-Augmented Generation (RAG)-based LLM chatbot that is suitable for Q&A, summarization, and other natural language processing (NLP) tasks that rely on specific knowledge bases. For more information, see RAG-based LLM chatbot.

• In AI painting scenarios, you can use a trained LoRA model in an SD service as an auxiliary model to improve the painting performance. For more information, see Deploy a LoRA SD model by using Kohya_ss in EAS.

• For more information about the SDKs that you can use to call services, see SDKs.

• For more information about billing rules of EAS, see Billing of EAS.