How to use CloudLens for SLS to analyze resource usage - Simple Log Service

This topic describes the common scenarios in which CloudLens for SLS is used to perform custom analysis on the usage of Simple Log Service resources.

Background information

CloudLens for SLS uses billing data to monitor the usage of billable items. You are not charged for the use of billing data. Billing data is sourced from the Alibaba Cloud Billing Management console. Simple Log Service runs a data transformation job to pull billing data to a Tablestore instance for storage. You are not charged for the storage of billing data in the Tablestore instance. You can associate Simple Log Service with the Tablestore instance to query and analyze billing data. In this case, the Tablestore instance is considered an external data source.

CloudLens for SLS provides resource usage reports based on billing data. You can monitor the usage of Simple Log Service billable items in real time. You can also execute custom SQL statements to analyze the usage of Simple Log Service resources based on billing data and configure alerts for resource usage.

Entry point

Log on to the Log Service console.
In the Log Application section, click the Cloud Service Lens tab. Then, click CloudLens for SLS.
For more information, see CloudLens for SLS.

Billing data

Simple Log Service billing data contains the usage of billable items for each Simple Log Service instance. Before you perform custom analysis on the usage of Simple Log Service resources, you must understand the basic terms that are involved in Simple Log Service billing data.

Term	Field	Description
instance	InstanceID	The minimum resource granularity of Simple Log Service. An instance corresponds to a Logstore in Simple Log Service.
billable item	BillingItem	The item based on which you are charged for Simple Log Service. Examples: storage usage of the hot storage tier and index traffic of log data.

Instances

A Simple Log Service instance is the minimum resource granularity of Simple Log Service. An instance ID contains the following variables: ${aliUid};${Project};${Logstore};${Region}. The following table describes the variables.

Variable	Description
`${aliUid}`	The ID of your Alibaba Cloud account.
`${Project}`	The name of your project. A project in Simple Log Service is used to isolate the resources of different users and control access to specific resources. For more information, see Project.
`${Logstore}`	The name of your Logstore. A Logstore in Simple Log Service is used to collect, store, and query logs. For more information, see Logstore.
`${Region}`	The physical location where the data center of Simple Log Service is deployed. You can specify a region when you create a project. After the project is created, you cannot change the region. For more information, see Supported regions.

Billable items

You are charged for using the capabilities provided by Simple Log Service based on the billable items of Simple Log Service. You are separately charged based on each billable item of Simple Log Service. For example, if you collect and store logs, you are charged a write traffic fee and a log storage fee. For more information, see Billable items of pay-by-feature.

In billing data, the BillingItem field indicates a billable item. For more information about the billable items of Simple Log Service, see Billable items.

Common scenarios of SLS resource usage analysis

You can perform custom analysis on billing data to monitor your resource usage in real time.

Monitor resource usage trends and the resource usage details of instances

You can monitor resource usage trends and the resource usage details of instances to handle the exceptions that cause significant changes in billing data at the earliest opportunity.

Monitor resource usage trends
You can call the date_trunc function to analyze resource usage trends at different time granularities.
- Query statement
  The BillingItem field specifies a billable item, and the Usage field specifies resource usage.
```
* |
select
 date_trunc('day', __time__) as "Date",
 BillingItem,
 round(sum(Usage), 3) as "Usage"
FROM instance_bill
where
 productcode = 'sls'
group by
 "Date",
 BillingItem
order by
 "Date"
```
- Query and analysis results: You can use a flow chart (Pro) to display the query and analysis results in a visualized manner. For more information, see Flow chart (Pro). To display the query and analysis results in a flow chart (Pro), set Axis X Field to Date, Axis Y Field to Usage, and Aggregate Column to BillingItem.
Monitor the resource usage details of instances
The resource usage details include the fees and resource usage of each instance. To analyze the resource usage details of SLS instances, you can combine SQL statements that analyze fees, resource usage, and storage.
- The t1 analytic statement is used to analyze the statistics of billable item usage, including the billable item usage of each instance.
- The t2 analytic statement is used to analyze the fees of instances, including the payable amount of each instance and the growth rate of the payable amount in the current month compared with the payable amount in the previous month. The PretaxAmount field specifies the payable amount of each instance.
- The t3 analytic statement is used to analyze the total storage usage of instances. The storage usage of instances is calculated on a daily basis. The total storage usage of instances is all storage that is used until the last day within the specified time range.
  - Query statement
```
* | with t1 as (
	select
	InstanceID,
	BillingItem,
	sum(Usage) as Usage
	FROM instance_bill
	where
	ProductCode = 'sls'
	and BillingItem not like '%Storage%'
	and BillingItem not like '%Storage%'
	group by
	InstanceID,
	BillingItem
),
t2 as (
	select
	InstanceID,
	round(di [1], 3) as "Fee",
	concat(
		cast(round(di [3] * 100 -100, 3) as varchar),
		'%'
	) "Growth rate compared with the previous month"
	from(
		select
		InstanceID,
		compare(cost, 2592000) as di
		FROM (
			select
			InstanceID,
			round(sum(PretaxAmount), 2) as cost
			FROM instance_bill
			where
			ProductCode = 'sls'
			group by
			InstanceID
		)
		group by
		InstanceID
	)
),
t3 as (
	select
	InstanceID,
	max_by("Storage", t) as "Storage"
	FROM (
		select
		t,
		InstanceID,
		round(sum("Storage"), 2) as "Storage"
		from(
			select
			date_trunc('day', __time__) as t,
			InstanceID,
			case
			when BillingItem like '%Storage%'
			or BillingItem like '%Storage%' then Usage
			else 0
			end as "Storage"
			FROM instance_bill
			WHERE
			ProductCode = 'sls'
			group by
			t,
			InstanceID,
			"Storage"
		)
		group by
		t,
		InstanceID
		order by
		t asc
	)
	group by
	InstanceID
)
select
 t1.InstanceID as InstanceID,
 t1.BillingItem as BillingItem,
 t1.Usage as Usage,
 t2."Fee" as "Fee",
 t2."Growth rate compared with the previous month" as "Growth rate compared with the previous month (%)",
 t3."Storage" as "Storage"
FROM t1
 left join t2 on t1.InstanceID = t2.InstanceID
 left join t3 on t1.InstanceID = t3.InstanceID
order by
 "Fee" desc
limit
 150000
```
  - Query and analysis results
    You can use a cross table to display the details of billable item usage for each instance in a visualized manner. For more information, see Cross table. To display the query and analysis results in a cross table, set Category Column to BillIngItem, Metric Column to Usage, and Aggregate Column to InstanceID, Fee, Growth rate compared with the previous month (%), and Storage. After you configure the settings, you can monitor the fees and resource usage of each instance. You can sort instances by billable item and obtain the top instance for each billable item.

You can efficiently analyze resource usage based on the monitoring results of resource usage trends and resource usage details of instances. You can also analyze changes in the usage of billable items such as read and write traffic, storage, and index traffic based on the monitoring results.

Increase in the usage of read and write traffic

You can execute the following query statement to analyze the read and write traffic of each instance and display the query and analysis results in a chart:

Query statement

* |
select
 "Date",
 InstanceID,
"Usage"
from(
	select
	date_trunc('day', __time__) as "Date",
	InstanceID,
	round(sum(Usage), 3) as "Usage"
	FROM instance_bill
	where
	productcode = 'sls'
	and BillingItem = 'Read and write traffic' 
	group by
	"Date",
	InstanceID
)
group by
 "Date",
 "Usage",
 InstanceID
order by
 "Date"
limit
 10000

Increase in storage usage

Query statement

* |
select
 InstanceID,
 round(diffreal [1], 3) as "Usage",
 concat(
		cast(
			round(diffreal [1] / sum(diffreal [1]) over() * 100, 2) as varchar
		),
		'%'
	) as "Proportion of storage usage",
 concat(
		cast(
			coalesce(round(diffreal [5] * 100 -100), 2) as varchar
		),
		'%'
	) as "One day before",
 concat(
		cast(
			round(coalesce(diffreal [6] * 100 -100, 2)) as varchar
		),
		'%'
	) as "Two days before",
 concat(
		cast(
			round(coalesce(diffreal [7] * 100 -100), 2) as varchar
		),
		'%'
	) as "Three days before"
from(
	select
	InstanceID,
	compare(Usage, 86400, 172800, 259200) as diffreal
	from(
		select
		InstanceID,
		sum(Usage) as Usage
		FROM instance_bill
		where ProductCode='sls' and BillingItem = 'Storage'
		group by
		InstanceID
	)
	group by
	InstanceID
)
order by
 diffreal [1] desc
limit
 1000

You can execute the following query statement to analyze the increase in storage usage for instance5:

Query statement

Replace ${instance5} with the actual instance.

* |
select
 date_trunc('day', __time__) as "Date",
 BillingItem,
 round(sum(Usage), 3) as "Usage"
FROM instance_bill
where
 productcode = 'sls'
and InstanceID = ${instance5}
group by
 "Date",
 BillingItem
order by
 "Date"

No resource usage counted

Check whether resource plans are used to offset the fees of billable items based on billing data.
- Query statement
  Query the resource usage data and the resource usage for which fees are offset by using resource plans.
```
(*) |
select
	InstanceID,
 BillingItem,
 	Usage as "Usage",
	DeductedByResourcePackage as "Resource usage for which fees are offset by using resource plans"
FROM instance_bill
where
 productcode = 'sls'
```
Check whether the resource usage for which fees are offset by using resource plans is equal to the total usage of billable items.
- Query statement
  Query the resource usage before the fees are offset by using resource plans. The DeductedByResourcePackage field specifies the resource usage for which fees are offset by using resource plans. The resource usage before the fees are offset by using resource plans is calculated by using the following formula: Usage + DeductedByResourcePackage.
```
* |
select
 date_trunc('day', __time__) as "Date",
 BillingItem,
 round(sum(Usage + DeductedByResourcePackage), 3) as "Resource usage before the fees are offset by using resource plans"
FROM instance_bill
where
 productcode = 'sls'
group by
 "Date",
 BillingItem
order by
 "Date"
```

Configure alerts for resource usage

The alerting feature of Simple Log Service is an end-to-end intelligent O&M system that allows you to monitor data based on alert monitoring rules, denoise alerts, manage incidents, and distribute alert notifications. You can use the alerting feature to monitor the resource usage of Simple Log Service in real time. CloudLens for SLS is integrated with anomaly detection. You can access Alert Center in an efficient manner. For more information, see Configure alerts.

Configure alerts for the total resource usage of the previous day

To control the overall costs, you need to take note of the total resource usage. In this example, you can use the following query statement to query the storage usage:

Query statement

 * | 
select
	round(sum(Usage), 2) as "Storage usage"
from instance_bill
where ProductCode='sls' 
and BillingItem='Storage'

Configure alerts for the resource usage of an instance

To reduce costs, you need to take note of the resource usage of a specific instance. In this example, you can use the following query statement to query the storage usage. If the storage usage of an instance is greater than 10, an alert is triggered.

Query statement

* |
select
 InstanceID,
 "Storage usage"
from(
 select
 InstanceID,
 round(sum(Usage), 2) as "Storage usage"
 FROM instance_bill
 where
 ProductCode = 'sls'
 and BillingItem = 'Storage'
 group by
 InstanceID
 )
where
 "Storage usage" > 10

The preceding alerting scenarios are provided only for reference. You can configure alerts based on your business requirements. For more information, see Configure an alert monitoring rule in Simple Log Service.