All Products
Search

This Product

    Virtual Private Cloud:Monitor resources of an Express Connect circuit using flow logs

    Document Center

    Virtual Private Cloud:Monitor resources of an Express Connect circuit using flow logs

    Last Updated:Oct 12, 2024

    In a hybrid cloud, services can be deployed in departmental virtual private clouds (VPCs) or in different vSwitches of a VPC, and connect to on-premises data centers to facilitate efficient and secure data transmission. You can leverage flow logs to analyze traffic between the cloud and on-premises instances, track the resource usage of leased lines by a VPC or a vSwitch, and enhance resource planning accordingly to ensure the network performance of key services.

    Overview

    Express Connect circuits

    Express Connect facilitates fast, stable, and secure private network communications between on-premises data centers and VPCs.

    Express Connect circuits ensure that data transmission is reliable and secure, mitigate instability, and avoid data theft.

    Flow logs

    Flow logs in VPCs capture traffic from Elastic Network Interfaces (ENIs), specific VPCs, or all ENIs in a vSwitch.

    Scenarios

    Flow logs can be used to monitor resources usage in Express Connect circuits in the following scenarios:

    • Network enhancement: Flow logs provide insight into how services utilize resources in Express Connect circuits, which is essential for resource planning and performance maintenance of critical services.

    • Security audit: Flow logs help you detect potential security threats, identify abnormalities, and take preventative measures to avoid network failures or remedy the situation by assessing network issues following a security risk.

    • Cost management: Understanding resource usage across services in a hybrid cloud is key to calculating costs with accuracy and finding cost-saving opportunities.

    Example

    A company hosts different services on two VPCs in China (Hangzhou) that share an Express Connect circuit to link with the on-premises data center. The IT department uses flow logs to keep track of how many resources are used by departments, which provides guidance to resource planning and network performance enhancement.

    image

    Prerequisites

    • An Express Connect circuit is established in China (Hangzhou) to link the on-premises data center with the VPC for the data center to access ECS instances in the VPC. For more information, see Connect a data center to ECS by using an Express Connect circuit.

      Note

      In this example, you need to attach the Virtual Border Router (VBR), VPC1, and VPC2 to the Cloud Enterprise Network (CEN). Once attached, the route entries of the VPCs and VBR are automatically advertised to the CEN, which enables route learning between VPCs and the VBR.

    • The following table outlines how CIDR blocks are planned in this example. You can plan CIDR blocks based on your needs. Make sure no overlap occurs.

      Item

      CIDR block

      Server or client IP address

      On-premises IDC

      10.1.1.0/24

      Client address: 10.1.1.70

      VPC1

      192.168.20.0/24

      ECS01 instance: 192.168.20.5

      ECS02 instance: 192.168.20.6

      ECS03 instance: 192.168.20.7

      VPC2

      192.168.10.0/24

      ECS04 instance: 192.168.10.75

      ECS05 instance: 192.168.10.76

      ECS06 instance: 192.168.10.77

      VBR

      VLAN: 1

      Alibaba Cloud Side IPv4 Address: 10.0.0.1

      Data Center Side IPv4 Address: 10.0.0.2

      IPv4 Subnet Mask: 255.255.255.252

      N/A

    • Before creating flow logs, you need to log on to the Simple Log Service product page, and activate the service.

    Procedures

    Step 1: Create flow logs

    Follow these steps to create flow logs for VPC1 and VPC2 and capture ENI traffic:

    1. Log on to the VPC console.

    2. In the left-side navigation pane, click O&M and Monitoring > Flow Log. In the top navigation bar, choose China (Hangzhou).

    3. On the Flow Log page, click Create a Flow Log. Configure the following parameters and click OK.

      Item

      Description

      Resource Type

      Select the type of resource whose traffic you want to capture. In this example, VPC is selected.

      Resource Instance

      Select the instance whose traffic you want to capture. In this example, VPC1 and VPC2 are chosen.

      Data Transfer Type

      Select the type of traffic to capture. In this example, All is chosen.

      Project

      Choose a project to store the captured traffic. Create Project is selected for VPC1. Use the project for VPC2.

      Logstore

      Select a Logstore to keep the captured traffic. Create Logstore is selected for VPC1. The Logstore created for VPC1 is used for VPC2.

      Enable Log Analysis Report

      In this example, the feature is activated.

    Step 2: View flow logs

    1. On the Flow Logs page, find the flow logs you want to manage and click the name of the Logstore.

    2. Perform the steps in the following figure to view the traffic from VPCs to the on-premises data center.flowlog.png

      Number

      Description

      ①

      Enter the following SQL statement to aggregate and sort the traffic to view the percentage of traffic from VPCs to the data center.

      action: ACCEPT and srcaddr: 192.168.* and dstaddr:10.1.* | 
WITH 
    vpc1_traffic AS (
        SELECT 
            date_trunc('minute',__time__) AS minute,
            SUM(bytes*8/(case WHEN "end"-start=0 THEN 1 else "end"-start end)) AS total_vpc1_traffic
        FROM 
            log
        WHERE 
            srcaddr LIKE '192.168.10.%'
        GROUP BY 
            date_trunc('minute',__time__)
    ),
    vpc2_traffic AS (
        SELECT 
            date_trunc('minute',__time__) AS minute,
            SUM(bytes*8/(case WHEN "end"-start=0 THEN 1 else "end"-start end)) AS total_vpc2_traffic
        FROM 
            log
        WHERE 
            srcaddr LIKE '192.168.20.%'
        GROUP BY 
            date_trunc('minute',__time__)
    )
SELECT 
    COALESCE(vpc1_traffic.minute, vpc2_traffic.minute) AS minute,
    (COALESCE(vpc1_traffic.total_vpc1_traffic, 0) * 100/ NULLIF((COALESCE(vpc1_traffic.total_vpc1_traffic, 0) + COALESCE(vpc2_traffic.total_vpc2_traffic, 0)), 0))  AS vpc1_percentage, 
    (COALESCE(vpc2_traffic.total_vpc2_traffic, 0) * 100/ NULLIF((COALESCE(vpc1_traffic.total_vpc1_traffic, 0) + COALESCE(vpc2_traffic.total_vpc2_traffic, 0)), 0)) AS vpc2_percentage
FROM vpc1_traffic FULL OUTER JOIN vpc2_traffic ON vpc1_traffic.minute = vpc2_traffic.minute 
ORDER BY minute

      This SQL statement defines three parameters: minute, VPC1 traffic percentage vpc1_percentage, and VPC2 traffic percentage vpc2_percentage. The latter two parameters are sorted by minute in ascending sequence. Other parameters are as follows:

      • dstaddr: The destination CIDR blocks, which are CIDR block on-premises data center.

      • srcaddr: The source CIDR blocks, which are the CIDR blocks of the VPCs.

      • Enter the remaining fields according to the example values.

      ②

      Choose the time range for which you want to analyze the flow logs.

      ③

      Click the Graph tab and select the FC4CF495-8D94-49A7-855E-787226E92CAF.png icon to choose Line Chart Pro.

      ④

      In the Query and Analysis Configurations area, set the following parameters:

      • X Axis Field: Choose minute.

      • Y Axis Field: Set vpc1_percentage and vpc2_percentage.

      In the Standard Configurations area, set Format to percent (1-100).

      Keep other parameters as their default settings.

      ⑤

      Click Search & Analyze to view the traffic data generated when VPCs communicate with the data center.

    References