AnalyticDB:Spot instances

Overview

Idle Spark executor resources in AnalyticDB for MySQL may be sold at discounted prices. The resources are called spot instances.

The following section describes the differences between spot instance resources and AnalyticDB compute unit (ACU) elastic resources:

• Compared with the cost of ACU elastic resources, the cost of spot instance resources is approximately 30% lower.

• When you submit a Spark job in a job resource group for which the spot instance feature is enabled, the AnalyticDB for MySQL cluster preferentially preempts idle and low-cost spot instance resources. If the preemption succeeds, the Spark job is run by using the spot instance resources and you are charged for the spot instance resources. If the preemption fails, the Spark job is run by using ACU elastic resources and you are charged for ACU elastic resources.

• By default, Alibaba Cloud provides a 1-hour protection period for AnalyticDB for MySQL spot instance resources to ensure that the resources are not preempted or released within 1 hour after the resources are used. After the protection period of specific spot instance resources ends, the resources may be preempted. The probability of preemption increases together with the time for which the spot instance resources are used.

• Spark jobs that use spot instance resources have lower priority. For example, Spark job A is running by using spot instance resources and another AnalyticDB for MySQL cluster submits Spark job B that does not use spot instance resources. If Spark job B does not have sufficient resources, Spark job B may preempt the spot instance resources that are used by Spark job A and the spot instance resources are released. As a result, Spark job A fails. The probability of Spark job failures due to the release of spot instance resources is 1% to 5%.

Scenarios and suggestions

We recommend that you enable the spot instance feature in the following scenarios:

• Spark job development scenarios.

• Cost-sensitive scenarios.

• Scenarios that have high tolerance for Spark job failures.

• Scenarios in which most Spark jobs run within 1 hour.

For specific Spark jobs that require more than 1 hour to run and specific Spark jobs that run within 1 hour, we recommend that you perform the following operations:

• Create two job resource groups. Enable the spot instance feature and run only Spark jobs that require a short execution duration for the first job resource group. Do not enable the spot instance and run only Spark jobs that require a long execution duration for the second job resource group.

• Create a job resource group and enable the spot instance feature for the resource group. When you run a Spark job, configure the spark.adb.spotInstance.disabled parameter to disable the usage of spot instance resources for the Spark job. For more information, see the "(Optional) Disable the usage of spot instance resources for a Spark job" section of this topic.

Limits

• The spot instance feature is supported only in the following regions: China (Qingdao), China (Beijing), China (Zhangjiakou), China (Hangzhou), China (Shanghai), China (Shenzhen), China (Guangzhou), China (Hong Kong), Japan (Tokyo), Singapore, US (Silicon Valley), US (Virginia), Germany (Frankfurt), and UK (London).

• You can enable the spot instance feature only for job resource groups. After you enable the spot instance feature, Spark jobs in the job resource group attempt to use the spot instance resources, but this does not mean that the resources can be used.

• Only Spark jobs can use spot instance resources.

• Only the following specifications of Spark executor resources support spot instance resources: large and 2xlarge. For information about the specifications of Spark executor resources, see Spark application configuration parameters.

Billing rules

Compared with the cost of ACU elastic resources, the cost of spot instance resources is approximately 30% lower. For more information, see Pricing for Data Lakehouse Edition (V3.0).

Enable the spot instance feature

1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. On the Data Lakehouse Edition (V3.0) tab, find the cluster that you want to manage and click the cluster ID.

2. In the left-side navigation pane, choose Cluster Management > Resource Management. On the page that appears, click the Resource Groups tab.

   Enable the spot instance feature for an existing job resource group

   1. Find the job resource group that you want to manage and click Modify in the Actions column.

   2. In the Modify Resource Group panel, turn on Spot Instance.

   3. Click OK.

   4. Check whether the spot instance feature is enabled in the Spot Instance column of the job resource group.

      After you enable the spot instance feature, Spark jobs that run in the job resource group attempt to use the spot instance resources.

   Enable the spot instance feature when you create a job resource group

   1. In the upper-right corner of the tab, click Create Resource Group. In the panel that appears, configure the parameters that are described in the following table.

      Parameter	Description
      Resource Group Name	The name of the resource group. The name must be 2 to 30 characters in length and can contain letters, digits, and underscores (_). The name must start with a letter.
      Job Type	The job type of the resource group. Select Job from the drop-down list. Job resource groups are suitable for batch processing scenarios that require high throughput. In these scenarios, you can use on-demand scaling to scale computing resources.
      Minimum Computing Resources	Valid value: 0ACU.
      Maximum Computing Resources	The maximum amount of computing resources. Configure this parameter in the range of 8 ACUs to 1,024 ACUs in increments of 8 ACUs. The maximum amount of computing resources that can be configured in the console is 1,024 ACUs. If you want to modify the amount of resources, submit a ticket.
      Spot Instance	Specifies whether to enable the spot instance feature for the resource group. After you enable the spot instance feature, Spark jobs that run in the job resource group attempt to use the spot instance resources.

   2. Click OK.

   3. After you create the job resource group, check whether the spot instance feature is enabled in the Spot Instance column of the resource group.

      After you enable the spot instance feature, Spark jobs that run in the job resource group attempt to use the spot instance resources.

(Optional) Disable the usage of spot instance resources for a Spark job

If you do not want a Spark job to use spot instance resources, you can configure the spark.adb.spotInstance.disabled parameter for the Spark job.

After you run the Spark job, you can view the value of the Spot Instance Resource Usage parameter to check whether the usage of spot instance resources is disabled. If the value of the Spot Instance Resource Usage parameter is 0, the Spark job does not use spot instance resources.

View the usage of spot instance resources

After you run a Spark job, you can view the Spot Instance Resource Usage parameter in the Resource Group Monitoring panel of the job resource group. For more information, see the "View the information about computing resources of a resource group" section of the View monitoring information about resource groups topic.

Disable the spot instance feature

1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. On the Data Lakehouse Edition (V3.0) tab, find the cluster that you want to manage and click the cluster ID.

2. In the left-side navigation pane, choose Cluster Management > Resource Management. On the page that appears, click the Resource Groups tab.

3. Find the job resource group that you want to manage and click Modify in the Actions column.

4. In the Modify Resource Group panel, turn off Spot Instance.

5. Click OK.

6. Check whether the spot instance feature is disabled in the Spot Instance column of the job resource group.

Related operations