Simple Log Service：Use the log analysis feature

Usage notes

Before you use the log analysis feature, you need to know basic information about the feature, how to calculate the required log storage capacity, and log sampling descriptions. For more information, see Overview.

Step 1: Enable the log analysis feature

1. Log on to the Anti-DDoS Proxy console.

2. In the top navigation bar, select the region of your instance.

   • Anti-DDoS Proxy (Chinese Mainland): If your instance is an Anti-DDoS Proxy (Chinese Mainland) instance, select Chinese Mainland.

   • Anti-DDoS Proxy (Outside Chinese Mainland): If your instance is an Anti-DDoS Proxy (Outside Chinese Mainland) instance, select Outside Chinese Mainland.

3. In the left-side navigation pane, choose Investigation > Log Analysis.

4. On the Log Analysis page, click Buy Now.

5. On the DDoS Log Service page, configure the parameters, click Buy Now, and then complete the payment.

   Parameter	Description
   Applicable Product	Select the instance type of Anti-DDoS Proxy.
   Log Storage Capacity	Select the capacity to store logs. Unit: TB. If log storage is large enough and within the validity period, logs are stored from the first day the feature is used. The logs that are generated within the following 180 consecutive days are stored. Logs from day 181 overwrite logs from day 1, which indicates that the logs generated only within the last 180 days are stored. Important After the log storage is exhausted, new logs cannot be stored.
   Duration	Select a validity period for the feature. Important If the log analysis feature expires, new logs cannot be stored.

   After you enable the feature, you must authorize Anti-DDoS Proxy to store logs in a dedicated Simple Log Service Logstore based on the instructions on the Log Analysis page.

Step 2: Enable the log collection feature

On the Log Analysis page, enable the log collection feature for the domain name of your website.

• Enable the log collection feature for a domain name: Select a domain name from the Select Domain Names drop-down list and turn on Status.

• Enable the log collection feature for multiple domain names at a time: Click Batch Configure in the upper-right corner of the page. In the Batch Configure panel, select multiple domain names and click Batch Enable.

After you enable the log collection feature, Simple Log Service automatically creates a dedicated project for Anti-DDoS Proxy. This dedicated project is used to manage the logs of Anti-DDoS Proxy. You can view the dedicated project on the homepage of the Simple Log Service console.

The name of the dedicated project for Anti-DDoS Proxy (Chinese Mainland) starts with ddoscoo-project. The name of the dedicated project for Anti-DDoS Proxy (Outside Chinese Mainland) starts with ddosdip-project. A dedicated project for Anti-DDoS Proxy contains the following resources:

• A dedicated Logstore that is used to store the logs of Anti-DDoS Proxy. The name of the dedicated Logstore for Anti-DDoS Proxy (Chinese Mainland) starts with ddoscoo-logstore. The name of the dedicated Logstore for Anti-DDoS Proxy (Outside Chinese Mainland) starts with ddosdip-logstore.

• Two preset log dashboards that are used to display the Log Analysis results in charts. The dashboards are DDoS Access Center and DDoS Operation Center. The information in the dashboards is the same for both Anti-DDoS Proxy (Chinese Mainland) and Anti-DDoS Proxy (Outside Chinese Mainland).

(Optional) Step 3: Query and analyze logs

1. On the Log Analysis page, select a domain name from the Select Domain Names drop-down list.

2. On the Full Logs tab, click 15 Minutes to specify a time range for the query.

   Note

   • Anti-DDoS Proxy logs are retained for 180 days. By default, you can query logs only of the previous 180 days.

   • The query results may contain logs that are generated 1 minute earlier or later than the specified time range.

3. Enter a query statement in the search box, and then click Search & Analyze.

   Each query statement consists of a search statement and an analytic statement. The search statement and the analytic statement are separated with a vertical bar (|). Format: Search statement|Analytics statement.

   Statement	Optional	Description
   Search statement	Yes	A search statement specifies search conditions, such as a keyword, a numeric value, a numeric value range, an asterisk (*), or a combination of search conditions. If you specify a space or an asterisk (*) as the search statement, no conditions are used for searching, and all logs are returned. For more information, see Search syntax. Note For more information about log fields, see Fields included in full logs.
   Analytics statement	Yes	An analytic statement is used to aggregate and compute the data in search results or all logs. If you leave the analytics statement empty, the search results are returned but analysis is not performed. For more information, see Log analysis overview. Note In an analytics statement, the from log part is similar to the from <table name> part in a standard SQL statement and can be omitted. By default, the first 100 log entries are returned. If you want to adjust this number, you can execute a LIMIT statement. For more information, see LIMIT clause.

   After a query statement is executed, analysis results are automatically displayed in tables. The analysis results can also be displayed in a variety of charts, such as a line chart, column chart, or pie chart. You can choose a display method based on your business requirements. For more information, see Chart overview

   You can also configure alert rules based on the charts in a dashboard to monitor service status in real time. For more information, see Overview.

   Common query statements

   • Queries the number of visits to a domain name.

     * | SELECT COUNT(*) as times, host GROUP by host ORDER by times desc limit 100

   • Queries the type of attacks that are blocked.

     * | select cc_action,cc_phase,count(*) as t group by cc_action,cc_phase order by t desc limit 10

   • Queries the queries per second (QPS).

     * | select time_series(__time__,'15m','%H:%i','0') as time,count(*)/900 as QPS group by time order by time

   • Queries the domain names that are attacked.

     * and cc_blocks:1 | select cc_action,cc_phase,count(*) as t group by cc_action,cc_phase order by t desc limit 10

   • Queries the URLs that are attacked.

     * and cc_blocks:1 | select count(*) as times,host,request_path group by host,request_path order by times

   • Queries the details of a request.

     * | select date_format(date_trunc('second',__time__),'%H:%i:%s') as time,host,request_uri,request_method,status,upstream_status,querystring limit 10

   • Queries the details of the 5XX status codes.

     * and status>499 | select host,status,upstream_status,count(*)as t group by host,status,upstream_status order by t desc

   • Queries the distribution of request latencies.

     * | SELECT count_if(upstream_response_time<20) as "<20",
     count_if(upstream_response_time<50 and upstream_response_time>20) as "<50",
     count_if(upstream_response_time<100 and upstream_response_time>50) as "<100",
     count_if(upstream_response_time<500 and upstream_response_time>100) as "<500",
     count_if(upstream_response_time<1000 and upstream_response_time>500) as "<1000",
     count_if(upstream_response_time>1000) as ">1000"

(Optional) Step 4: Query log reports

Simple Log Service provides dashboards for you to analyze data in real time. After you query and analyze logs by using query and analysis statements, you can save the charts of analysis results to a dashboard. Simple Log Analysis provides two preset dashboards: DDoS Access Center and DDoS Operation Center.

1. On the Log Analysis page, select a domain name from the Select Domain Names drop-down list.

2. On the Log Reports tab, click Select Time Range to specify a time range.

   Note

   Each chart on the dashboard is generated based on the statistics within a specific time range. For example, the default time range is 1 hour for a website access chart and 1 week for an access trend chart. After you specify a time range, all charts on the current page are displayed based on the specified time range.

3. View the preset dashboards.

   The log reports are displayed in different types of charts. For more information about the types of charts, see Chart overview.

   • DDoS Access Center: shows the basic website metrics, access trends, request source distribution, and other statistics such as access domain names and client types. The website metrics include PVs, UVs, inbound traffic, and peak bandwidth.

     DDoS Access Center

     Chart name	Section	Default time range	Description	Example
     PV	Single value	1 hour (relative)	The total number of PVs.	100000
     UV	Single value	1 hour (relative)	The total number of UVs.	100000
     Traffic In	Single value	1 hour (relative)	The total volume of inbound traffic of the website. Unit: MB.	300 MB
     Peak network in traffic	Single value	Today (time frame)	The maximum inbound data transmission rate of the website. Unit: byte/s.	100 Bytes/s
     Peak network out traffic	Single value	Today (time frame)	The maximum outbound data transmission rate of the website. Unit: byte/s.	100 Bytes/s
     Traffic network trend	Double line graph	1 week (relative)	The trends of inbound and outbound traffic. Unit: KB/s.	Not supported.
     PV/UV trends	Double line graph	1 week (relative)	The trends of PVs and UVs.	Not supported.
     Access status distribution	Pie chart	1 week (relative)	The distribution of requests with different status codes, such as 400, 304, and 200. Unit: count/minute.	Not supported.
     Access source	World map	1 hour (relative)	The distribution of PVs from different countries.	Not supported.
     Traffic in source (world)	World map	1 hour (relative)	The distribution of inbound traffic from different countries. Unit: MB.	Not supported.
     Traffic in source (China)	China map	1 hour (relative)	The distribution of inbound traffic from different provinces in China. Unit: MB.	Not supported.
     Access heatmap	AMAP	1 hour (relative)	The heat map that shows the geographical locations of visitors.	Not supported.
     Network Ip source Top 10	Donut chart	1 hour (relative)	The distribution of the inbound traffic from different Internet service providers (ISPs), such as China Telecom, China Unicom, China Mobile, and CERNET. Unit: MB.	Not supported.
     Referer	Table chart	1 hour (relative)	The top 100 most used referer URLs, hosts, and the number of redirections.	Not supported.
     Mobile client distribution	Donut chart	1 hour (relative)	The distribution of Anti-DDoS Proxy lines.	Not supported.
     PC client distribution	Donut chart	1 hour (relative)	The top 20 most used user agents, such as iPhone, iPad, Internet Explorer, and Google Chrome.	Not supported.
     Request content type distribution	Donut chart	1 hour (relative)	The top 20 most requested content types, such as HTML, form, JSON, and streaming data.	Not supported.
     Accessed sites	Donut chart	1 hour (relative)	The top 20 most visited domain names of the website.	Not supported.
     Top clients	Table chart	1 hour (relative)	Information about the top 100 clients that initiates the most requests. The information includes the IP addresses, PVs, inbound traffic, number of invalid requests, and number of attacks.	Not supported.
     URL with slowest response	Table chart	1 hour (relative)	Information about the top 100 URLs with the longest response time. The information includes the websites, URLs, response time, and the number of accesses.	Not supported.

   • DDoS Operation Center: shows the overall operations status of the website, including inbound and outbound traffic trends, requests and interception trends, attackers, and visited websites.

   You can also click Subscribe in the upper-right corner of the Log Reports tab to subscribe dashboards and send dashboard data to specific recipients by using emails or DingTalk messages. For more information, see Add a Subscription.

References

Common operations on logs of Alibaba Cloud services