MaxCompute:Use resource observation

Procedure

1. Log on to the MaxCompute console, and select a region in the top navigation bar.

2. In the left-side navigation pane, click Resource Observation.

3. On the Resource Observation page, click the Job Performance Observation tab.

4. You can filter and group the jobs that you want to view based on the following parameters and group metric data in charts based on different dimensions.

   Parameter	Description
   Time Period	Required. The time range (start time and end time) that is used to filter completed jobs. You can select a preset time range or configure a custom time range. 1d: previous day. 3d: previous 3 days. 7d: previous 7 days. Custom time range: Click the drop-down list, select a date, and then click Select Time to select a time range. Note The default time range is the previous day. The maximum time range is 7 days and the minimum time range is 1 hour. You can search for only jobs in the previous 45 days.
   Project Name	The names of the MaxCompute projects that are used to filter completed jobs. Note By default, all projects are selected. You can select up to eight projects.
   Quota	The computing quotas that are used to filter completed jobs. Note By default, all computing quotas are selected. You can select up to eight level-2 quotas. For more information about computing quotas, see Manage quotas for computing resources in the new MaxCompute console.
   Group By	Required. The group of data in charts. You can define groups based on dimensions and chart types. Valid values: No Group: displays the trends of metrics over time for all jobs within the selected filter range. This is the default value. Project: displays the metrics of all jobs within the selected filter range by project. Note If you select Project, you must specify Project Name in the filter parameters and select up to eight projects. Quota: displays the metrics of all jobs within the selected filter range by level-2 quota. Note If you select Quota, you must specify Quota in the filter parameters and select up to eight level-2 quotas. Job Type: displays the metrics of all jobs within the selected filter range by job type. SQL: SQL job. SQLRT: MaxCompute Query Acceleration (MCQA) SQL job. LOT: MapReduce job. CUPID: Spark or Mars job. Algo_Task: machine learning job. GRAPH: graph computing job. Job End Status: displays the metrics of all jobs within the selected filter range based on the status when the job ends. Success: The job succeeds. Failed: The job fails. Canceled: The job is canceled.

5. Click Search to view the statistics of each metric.

6. Optional. Select Data Summary to view the statistics of each metric based on the selected time range.

   Parameter	Description
   By Hour	One hour is a time granularity. If you select By Hour, data statistics of jobs that are completed in the current hour are displayed. By default, data at hourly granularity is displayed. For example, if the current point in time is 14:00 on May 6, 2024, the statistics of each metric of jobs completed in the time range from 14:00 to 15:00 on May 6, 2024 are displayed.
   By Day	One day is a time granularity. If you select By Day, data statistics of jobs that are completed in the current day are displayed. For example, if the current date is May 6, 2024, the statistics of each metric of the jobs completed in the time range from 00:00 on May 6, 2024 to 00:00 on May 7, 2024, are displayed.

7. Select an option for Comparison Period to view the historical data statistics of the day or hour that are obtained by subtracting the number of days or hours specified in Comparison Period from the current date or hour.

   Default value: No Comparison. You can also select Previous 30 Days, Previous 7 Days, or Previous 1 Day. For example, if the current point in time is 14:00 on May 6, 2024 and Previous 30 Days is selected, the data at 14:00 on April 6, 2024 is used to compare with the current data.

Metrics

• CU Usage Trend (Unit: Core × Hour)

  Metric name	Description
  CPU-hour (Unit: Core × Hour)	The number of CPU-hours consumed for completed jobs within the selected filter range. 1 CPU-hour refers to that 1 CPU core is consumed for 1 hour. Number of CPU-hours = Number of CPU cores × Duration
  Memory-hour (Unit: GB × Hour)	The number of memory-hours consumed for completed jobs within the selected filter range. 1 memory-hour refers to that 1-GB memory space is consumed for 1 hour. Number of memory-hours = Memory space × Duration.
  Top 10 CPU-Hour Consumption Analysis or Top 10 Memory-Hour Consumption Analysis	Displays the top 10 jobs that consume the most CPU-hours or memory-hours and the top 10 signatures and ExtNodeIds of jobs that are ranked by the highest total CPU-hour, the highest average CPU-hour, the highest total memory-hour, and the highest average memory-hour within the selected filter range.

• Job Runtime Period (Unit: seconds)

  Metric name	Description
  Average Runtime Duration	The average job duration of completed jobs within the selected filter range.
  Maximum Runtime Duration	The maximum job duration of completed jobs within the selected filter range.
  Minimum Runtime Duration	The minimum job duration of completed jobs within the selected filter range.
  Select a quantile runtime duration	The duration that is taken to complete a specified quantile of jobs within the selected filter range. The quantile can be the 1st quantile, 5th quantile, 10th quantile, 50th quantile, 90th quantile, 95th quantile, or 99th quantile. For example, for the 99th quantile, this metric indicates the duration that is taken to complete 99% jobs.
  Top 10 Job Runtime Analysis	Displays the top 10 jobs that have the longest running durations and the top 10 signatures and ExtNodeIds of jobs that are ranked by the longest total running durations and longest average running durations.

• Job Count Trend (Unit: counts): displays the number of jobs that are completed within the selected filter range.

• Job Scan Amount Trend (Unit: GB): displays the amount of data that is scanned by completed jobs within the selected filter range. The unit is adaptively changed and the actually used unit is displayed in charts.

• Trend of Job Scan Amount per CU-Hour (Unit: GB): displays the average amount of data that is scanned by jobs per CU-hour within the selected filter range. The unit is adaptively changed and the actually used unit is displayed in charts. 1 CU contains 1 CPU core and 4 GB of memory. The value is calculated by using MAX(CPU-hours, Roundup(Memory-hours/4)).

You can also use the tenant-level Information Schema to collect statistics on the preceding metrics. You need to take note that the Information Schema task_history table contains task instances that are generated by all operations. However, the data of metrics displayed on the Job Performance Observation tab in the console is obtained only from the jobs that consume computing resources. Therefore, the statistical results obtained by using the tenant-level Information Schema may be different from the statistical results displayed on the Job Performance Observation tab.

The following SQL statements show an example.

SET odps.namespace.schema=TRUE;
SELECT to_char (end_time, 'yyyy-mm-dd hh'), -- The hour in which the job completes.
       -- to_char (end_time, 'yyyy-mm-dd'), -- The date on which the job completes. If daily granularity is used, use this line to replace the preceding line.
       sum(cast(cost_cpu/100/3600 as DECIMAL(18,5) )) cost_cpuh, -- The CPU-hours.
       sum(cast(cost_mem/1024/3600 as DECIMAL(18,5) )) cost_memh, -- The memory-hours.
       avg(datediff(end_time, start_time, 'ss')), -- The average runtime duration of jobs.
       min(datediff(end_time, start_time, 'ss')), -- The minimum runtime duration of jobs.
       max(datediff(end_time, start_time, 'ss'))  -- The maximum runtime duration of jobs.
       -- status, -- Group basis: status: job status; project: task_catalog; job type: task_type. 
FROM SYSTEM_CATALOG.INFORMATION_SCHEMA.tasks_history
WHERE ds>=to_char(date_add(getdate(),-7),'yyyymmdd')  -- You can add other filter conditions based on your business requirements.
and task_type in ('SQL','SQLRT','LOT','CUPID','ALgoTask')
GROUP BY to_char (end_time, 'yyyy-mm-dd hh')
         -- to_char (end_time, 'yyyy-mm-dd'), -- The date on which the job completes. If daily granularity is used, use this line to replace the preceding line.
         -- status, -- Group basis: status: job status; project: task_catalog; job type: task_type.
order BY to_char (end_time, 'yyyy-mm-dd hh') ASC;
         -- to_char (end_time, 'yyyy-mm-dd'); -- The date on which the job completes. If daily granularity is used, use this line to replace the preceding line.

FAQ

• Question 1:

  • Problem description: After jobs are grouped by project or quota, some projects or quotas are not displayed in charts.

  • Possible causes: No jobs are available in the projects or quotas.

• Question 2:

  • Problem description: After a comparison period is selected, the data that corresponds to the comparison period is not available.

  • Possible causes: The project or quota may not be created or may be deleted within the comparison period. No jobs are available in the project or quota within the comparison period.