All Products
Search
Document Center

Simple Log Service:Use the log analysis feature

最終更新日:Sep 03, 2024

After you add your website to an Anti-DDoS Proxy instance, you can use the log analysis feature to collect and store the logs of your website, and then query and analyze the collected logs. This topic describes how to use the log analysis feature.

Usage notes

Before you use the log analysis feature, you need to know basic information about the feature, how to calculate the required log storage capacity, and log sampling descriptions. For more information, see Overview.

Prerequisites

  • An Anti-DDoS Proxy instance is purchased and your website is added to the instance. For more information, see Add one or more websites.

  • Simple Log Service is activated. If this is the first time you log on to the Simple Log Service console, you must activate Simple Log Service as prompted.

Step 1: Enable the log analysis feature

  1. Log on to the Anti-DDoS Proxy console.

  2. In the top navigation bar, select the region of your instance.

    • Anti-DDoS Proxy (Chinese Mainland): If your instance is an Anti-DDoS Proxy (Chinese Mainland) instance, select Chinese Mainland.

    • Anti-DDoS Proxy (Outside Chinese Mainland): If your instance is an Anti-DDoS Proxy (Outside Chinese Mainland) instance, select Outside Chinese Mainland.

  3. In the left-side navigation pane, choose Investigation > Log Analysis.

  4. On the Log Analysis page, click Buy Now.

  5. On the DDoS Log Service page, configure the parameters, click Buy Now, and then complete the payment.

    Parameter

    Description

    Applicable Product

    Select the instance type of Anti-DDoS Proxy.

    Log Storage Capacity

    Select the capacity to store logs. Unit: TB.

    If log storage is large enough and within the validity period, logs are stored from the first day the feature is used. The logs that are generated within the following 180 consecutive days are stored. Logs from day 181 overwrite logs from day 1, which indicates that the logs generated only within the last 180 days are stored.

    Important

    After the log storage is exhausted, new logs cannot be stored.

    Duration

    Select a validity period for the feature.

    Important

    If the log analysis feature expires, new logs cannot be stored.

    After you enable the feature, you must authorize Anti-DDoS Proxy to store logs in a dedicated Simple Log Service Logstore based on the instructions on the Log Analysis page.

Step 2: Enable the log collection feature

On the Log Analysis page, enable the log collection feature for the domain name of your website.

  • Enable the log collection feature for a domain name: Select a domain name from the Select Domain Names drop-down list and turn on Status.

  • Enable the log collection feature for multiple domain names at a time: Click Batch Configure in the upper-right corner of the page. In the Batch Configure panel, select multiple domain names and click Batch Enable.

After you enable the log collection feature, Simple Log Service automatically creates a dedicated project for Anti-DDoS Proxy. This dedicated project is used to manage the logs of Anti-DDoS Proxy. You can view the dedicated project on the homepage of the Simple Log Service console.

日志项目

The name of the dedicated project for Anti-DDoS Proxy (Chinese Mainland) starts with ddoscoo-project. The name of the dedicated project for Anti-DDoS Proxy (Outside Chinese Mainland) starts with ddosdip-project. A dedicated project for Anti-DDoS Proxy contains the following resources:

  • A dedicated Logstore that is used to store the logs of Anti-DDoS Proxy. The name of the dedicated Logstore for Anti-DDoS Proxy (Chinese Mainland) starts with ddoscoo-logstore. The name of the dedicated Logstore for Anti-DDoS Proxy (Outside Chinese Mainland) starts with ddosdip-logstore.

  • Two preset log dashboards that are used to display the Log Analysis results in charts. The dashboards are DDoS Access Center and DDoS Operation Center. The information in the dashboards is the same for both Anti-DDoS Proxy (Chinese Mainland) and Anti-DDoS Proxy (Outside Chinese Mainland).

    (Optional) Step 3: Query and analyze logs

    1. On the Log Analysis page, select a domain name from the Select Domain Names drop-down list.

    2. On the Full Logs tab, click 15 Minutes to specify a time range for the query.

      Note
      • Anti-DDoS Proxy logs are retained for 180 days. By default, you can query logs only of the previous 180 days.

      • The query results may contain logs that are generated 1 minute earlier or later than the specified time range.

    3. Enter a query statement in the search box, and then click Search & Analyze.

      Each query statement consists of a search statement and an analytic statement. The search statement and the analytic statement are separated with a vertical bar (|). Format: Search statement|Analytics statement.

      Statement

      Optional

      Description

      Search statement

      Yes

      A search statement specifies search conditions, such as a keyword, a numeric value, a numeric value range, an asterisk (*), or a combination of search conditions.

      If you specify a space or an asterisk (*) as the search statement, no conditions are used for searching, and all logs are returned. For more information, see Search syntax.

      Note

      For more information about log fields, see Fields included in full logs.

      Analytics statement

      Yes

      An analytic statement is used to aggregate and compute the data in search results or all logs.

      If you leave the analytics statement empty, the search results are returned but analysis is not performed. For more information, see Log analysis overview.

      Note
      • In an analytics statement, the from log part is similar to the from <table name> part in a standard SQL statement and can be omitted.

      • By default, the first 100 log entries are returned. If you want to adjust this number, you can execute a LIMIT statement. For more information, see LIMIT clause.

      After a query statement is executed, analysis results are automatically displayed in tables. The analysis results can also be displayed in a variety of charts, such as a line chart, column chart, or pie chart. You can choose a display method based on your business requirements. For more information, see Chart overview

      You can also configure alert rules based on the charts in a dashboard to monitor service status in real time. For more information, see Overview.

      Common query statements

      • Queries the number of visits to a domain name.

        * | SELECT COUNT(*) as times, host GROUP by host ORDER by times desc limit 100
      • Queries the type of attacks that are blocked.

        * | select cc_action,cc_phase,count(*) as t group by cc_action,cc_phase order by t desc limit 10
      • Queries the queries per second (QPS).

        * | select time_series(__time__,'15m','%H:%i','0') as time,count(*)/900 as QPS group by time order by time
      • Queries the domain names that are attacked.

        * and cc_blocks:1 | select cc_action,cc_phase,count(*) as t group by cc_action,cc_phase order by t desc limit 10
      • Queries the URLs that are attacked.

        * and cc_blocks:1 | select count(*) as times,host,request_path group by host,request_path order by times
      • Queries the details of a request.

        * | select date_format(date_trunc('second',__time__),'%H:%i:%s') as time,host,request_uri,request_method,status,upstream_status,querystring limit 10
      • Queries the details of the 5XX status codes.

        * and status>499 | select host,status,upstream_status,count(*)as t group by host,status,upstream_status order by t desc
      • Queries the distribution of request latencies.

        * | SELECT count_if(upstream_response_time<20) as "<20",
        count_if(upstream_response_time<50 and upstream_response_time>20) as "<50",
        count_if(upstream_response_time<100 and upstream_response_time>50) as "<100",
        count_if(upstream_response_time<500 and upstream_response_time>100) as "<500",
        count_if(upstream_response_time<1000 and upstream_response_time>500) as "<1000",
        count_if(upstream_response_time>1000) as ">1000"

    (Optional) Step 4: Query log reports

    Simple Log Service provides dashboards for you to analyze data in real time. After you query and analyze logs by using query and analysis statements, you can save the charts of analysis results to a dashboard. Simple Log Analysis provides two preset dashboards: DDoS Access Center and DDoS Operation Center.

    1. On the Log Analysis page, select a domain name from the Select Domain Names drop-down list.

    2. On the Log Reports tab, click Select Time Range to specify a time range.

      Note

      Each chart on the dashboard is generated based on the statistics within a specific time range. For example, the default time range is 1 hour for a website access chart and 1 week for an access trend chart. After you specify a time range, all charts on the current page are displayed based on the specified time range.

    3. View the preset dashboards.

      The log reports are displayed in different types of charts. For more information about the types of charts, see Chart overview.

      • DDoS Access Center: shows the basic website metrics, access trends, request source distribution, and other statistics such as access domain names and client types. The website metrics include PVs, UVs, inbound traffic, and peak bandwidth.

        DDoS Access Center

        Chart name

        Section

        Default time range

        Description

        Example

        PV

        Single value

        1 hour (relative)

        The total number of PVs.

        100000

        UV

        Single value

        1 hour (relative)

        The total number of UVs.

        100000

        Traffic In

        Single value

        1 hour (relative)

        The total volume of inbound traffic of the website. Unit: MB.

        300 MB

        Peak network in traffic

        Single value

        Today (time frame)

        The maximum inbound data transmission rate of the website. Unit: byte/s.

        100 Bytes/s

        Peak network out traffic

        Single value

        Today (time frame)

        The maximum outbound data transmission rate of the website. Unit: byte/s.

        100 Bytes/s

        Traffic network trend

        Double line graph

        1 week (relative)

        The trends of inbound and outbound traffic. Unit: KB/s.

        Not supported.

        PV/UV trends

        Double line graph

        1 week (relative)

        The trends of PVs and UVs.

        Not supported.

        Access status distribution

        Pie chart

        1 week (relative)

        The distribution of requests with different status codes, such as 400, 304, and 200. Unit: count/minute.

        Not supported.

        Access source

        World map

        1 hour (relative)

        The distribution of PVs from different countries.

        Not supported.

        Traffic in source (world)

        World map

        1 hour (relative)

        The distribution of inbound traffic from different countries. Unit: MB.

        Not supported.

        Traffic in source (China)

        China map

        1 hour (relative)

        The distribution of inbound traffic from different provinces in China. Unit: MB.

        Not supported.

        Access heatmap

        AMAP

        1 hour (relative)

        The heat map that shows the geographical locations of visitors.

        Not supported.

        Network Ip source Top 10

        Donut chart

        1 hour (relative)

        The distribution of the inbound traffic from different Internet service providers (ISPs), such as China Telecom, China Unicom, China Mobile, and CERNET. Unit: MB.

        Not supported.

        Referer

        Table chart

        1 hour (relative)

        The top 100 most used referer URLs, hosts, and the number of redirections.

        Not supported.

        Mobile client distribution

        Donut chart

        1 hour (relative)

        The distribution of Anti-DDoS Proxy lines.

        Not supported.

        PC client distribution

        Donut chart

        1 hour (relative)

        The top 20 most used user agents, such as iPhone, iPad, Internet Explorer, and Google Chrome.

        Not supported.

        Request content type distribution

        Donut chart

        1 hour (relative)

        The top 20 most requested content types, such as HTML, form, JSON, and streaming data.

        Not supported.

        Accessed sites

        Donut chart

        1 hour (relative)

        The top 20 most visited domain names of the website.

        Not supported.

        Top clients

        Table chart

        1 hour (relative)

        Information about the top 100 clients that initiates the most requests. The information includes the IP addresses, PVs, inbound traffic, number of invalid requests, and number of attacks.

        Not supported.

        URL with slowest response

        Table chart

        1 hour (relative)

        Information about the top 100 URLs with the longest response time. The information includes the websites, URLs, response time, and the number of accesses.

        Not supported.

      • DDoS Operation Center: shows the overall operations status of the website, including inbound and outbound traffic trends, requests and interception trends, attackers, and visited websites.

      You can also click Subscribe in the upper-right corner of the Log Reports tab to subscribe dashboards and send dashboard data to specific recipients by using emails or DingTalk messages. For more information, see Add a Subscription.

    References

    Common operations on logs of Alibaba Cloud services