MaxCompute:Use the computing resource optimization feature to achieve cost-effectiveness

Precautions

• The prices in this topic are for reference only. You can view the actual prices on the product page.

• The evaluation method that is used in the following typical scenarios is simple. In the actual business implementation process, we recommend that you gradually adjust the resource configuration to apply the recommended plan based on your business requirements and take note of the effect after the resource configuration change.

Scenario 1: Some subscription computing resources are idle and the computing cost is high

A company has sufficient budget in the initial phase of data warehousing. To ensure that the output of important jobs can be generated before 08:00 every day, the company purchases 200 compute units (CUs) of subscription reserved computing resources for these jobs. A total of 520 jobs are scheduled to run on these resources every day. These jobs can be complete on schedule or even ahead of schedule every day, but the monthly computing cost reaches USD 4,400.

The company issues an Objectives and Key Results (OKR) for cost reduction to the big data department. However, the leader of the big data department cannot determine whether the output of important jobs can meet the business requirements after cost reduction and cannot determine the most appropriate cost reduction plan. A data O&M engineer introduces the computing resource optimization feature of MaxCompute to the leader and performs the following operations in the MaxCompute console to demonstrate this feature:

1. Go to the Cost Optimization page.

   1. Log on to the MaxCompute console. In the top navigation bar, select a region. In the left-side navigation pane, choose Cost Management > Cost Optimization.

   2. On the Cost Optimization page, select a level-1 subscription quota from the Select Quota drop-down list to view the data in the Estimated Daily CU Requests chart.

   The data in the Estimated Daily CU Requests chart reflects the current business situation. Jobs are initiated every hour. The number of requests between 05:00 and 08:00 every day is large, and the number of requests in other hours is relatively small.

2. Specify the estimation time point.

   In this example, two estimation time points 05:00 and 08:00 are specified in the Set Estimation Time Point step. This ensures that all jobs that are initiated before 05:00 can be complete before 05:00 and the important jobs that are initiated from 05:00 to 08:00 can be complete before 08:00.

3. View the evaluation result.

   Click Current Plan Estimation to view the job output based on the current resource configuration.

   

   In the CU Consumption Simulation (Current Plan Estimation) chart, the job output is not delayed based on the current CU configuration (200 reserved CUs), but some computing resources are idle. This indicates that cost reduction is feasible.

4. Specify the optimization objective.

   The Set Optimization Goal section below the CU Consumption Simulation (Current Plan Estimation) chart displays the current delay at the specified estimation time point in a table. The data in this table is consistent with the data in the CU Consumption Simulation (Current Plan Estimation) chart.

   The estimation time points are automatically generated in the Optimization Goal column, which are the same as the expected job completion time points. Then, click Generate Recommended Plan.

5. View the recommended plan.

   The optimization effect is presented in the CU Consumption Simulation (Recommended Plan) chart. The recommended plan shows that the company needs to purchase 50 reserved CUs and separately purchase 50 elastically reserved CUs for the periods of 04:00 to 05:00 and 06:00 to 08:00. After the optimization, the important jobs can be complete before 08:00, and the monthly computing cost is only USD 1,319.6. Compared with the current computing cost, the monthly cost can be reduced by about 70%.

   

   The leader is satisfied with the plan and expects to reduce more cost. If the output of jobs is delayed with 30 minutes, the business is not affected. Therefore, the optimization objective can be adjusted.

6. Adjust the optimization objective.

   Return to the Set Optimization Goal section and change the value in the Optimization Goal column that corresponds to 08:00 in the Estimation Time Point column to 08:30.

7. View the new recommended plan.

   Click Generate Recommended Plan again. The recommended plan shows that the company needs to purchase 50 reserved CUs and separately purchase 50 elastically reserved CUs for the periods of 04:00 to 05:00 and 06:00 to 07:00. After the optimization, the important jobs can be complete before 08:30, and the monthly computing cost is reduced to USD 1,246.4. Compared with the current computing cost, the monthly cost is reduced by about 71.7%.

8. Gradually adjust the resource configuration to apply the recommended plan.

   To avoid instability after the recommended plan is applied, the big data department first reduces the number of reserved CUs to 100. After a specific period of time, the department makes an evaluation again. The number of jobs does not significantly increase. The system still recommends reducing the resource configuration and using elastically reserved CUs. Therefore, the department adjusts the resource configuration based on the recommended plan in which the generation of job output is not delayed. After the trial operation period, the important jobs can still be complete on time almost every day. The computing cost is also reduced.

Scenario 2: The subscription computing resources are insufficient, and the job completion time is later than the expected time

A company purchases 60 subscription CUs for jobs in the initial phase of data warehousing. A total of 520 jobs are scheduled to run on these resources every day. A batch of important jobs are initiated from 05:00 to 08:00 every day. The business department expects that the important jobs can be complete before 08:00 every day. As the business expands, the amount of data scanned by the jobs continues to increase. The completion of the important jobs is often delayed. This is because the jobs are accumulated due to insufficient reserved resources. The company wants to adjust the resources to meet the completion requirements of the jobs without increasing the cost too much. The computing resource optimization feature can be used in this scenario. To use this feature, perform the following steps:

1. Go to the Cost Optimization page.

   1. Log on to the MaxCompute console. In the top navigation bar, select a region. In the left-side navigation pane, choose Cost Management > Cost Optimization.

   2. On the Cost Optimization page, select a level-1 subscription quota from the Select Quota drop-down list to view the data in the Estimated Daily CU Requests chart.

      The data in the Estimated Daily CU Requests chart reflects the current business situation. Jobs are initiated every hour. The number of requests between 05:00 and 08:00 every day is large, and the number of requests in other hours is relatively small.

2. Specify the estimation time point.

   In this example, two estimation time points 05:00 and 08:00 are specified in the Set Estimation Time Point step. This ensures that all jobs that are initiated before 05:00 can be complete before 05:00 and the important jobs that are initiated from 05:00 to 08:00 can be complete before 08:00.

3. View the evaluation result.

   Click Current Plan Estimation to view the job output based on the current resource configuration.

   

   In the CU Consumption Simulation (Current Plan Estimation) chart, the jobs that are initiated before 05:00 have a 3-minute delay and the important jobs that are initiated between 05:00 and 08:00 have a 48-minute delay based on the current CU configuration (60 reserved CUs). This estimation result is similar to the actual delay.

4. Specify the optimization objective.

   The Set Optimization Goal section below the CU Consumption Simulation (Current Plan Estimation) chart displays the current delay at the specified estimation time point in a table. The data in this table is consistent with the data in the CU Consumption Simulation (Current Plan Estimation) chart.

   The estimation time points are automatically generated in the Optimization Goal column, which are the same as the expected job completion time points. Then, click Generate Recommended Plan.

5. View the recommended plan.

   The optimization effect is presented in the CU Consumption Simulation (Recommended Plan) chart. The information displayed in the CU Consumption Simulation (Recommended Plan) chart is consistent with the information displayed in the CU Consumption Simulation (Current Plan Estimation) chart. The recommended plan shows that the company needs to purchase 50 reserved CUs and separately purchase 50 elastically reserved CUs for the periods of 04:00 to 05:00 and 06:00 to 08:00. This way, the jobs that are initiated before 05:00 and 08:00 can be complete without a delay. After the optimization, the computing cost is USD 0.4 per month less than the current computing cost.

   

6. Configure the recommended plan.

   The company thinks that the cost increase of this recommended plan is acceptable. To avoid instability after the recommended plan is applied, the company configures elastically reserved CUs based on the recommended plan but does not reduce the number of reserved CUs.

   1. In the left-side navigation pane of the MaxCompute console, choose Workspace > Quotas.

   2. On the Quotas page, find the level-1 quota that is evaluated and click Quota Configuration in the Actions column.

   3. On the Quota Plans tab of the Quota Configuration page, click Add Plan.

   4. In the Create Quota Plan dialog box, set Elastically Reserved CUs to 50 for the quota plan that is created and click OK.

   5. Configure the following scheduling plans for the quota plan based on the time points in the recommended plan. For more information, see Configure quotas.

      Start time	Quota plan
      00:00	Default
      04:00	Quota plan added in the previous step
      05:00	Default
      06:00	Quota plan added in the previous step
      08:00	Default

      Note

      The number of elastically reserved CUs for the Default plan is 0.

   Within the trial operation period, the important jobs can be complete on time almost every day. The R&D efficiency is improved and the computing cost just increases a little.