After you enable the flow log feature in the Virtual Private Cloud (VPC) console, flow logs are collected and sent to Simple Log Service. You can use Simple Log Service to query and analyze the flow logs. You can also troubleshoot network errors based on the flow logs. This topic describes how to use the data transformation feature of Simple Log Service to filter flow logs for Internet traffic logs.
Prerequisites
The flow log feature is enabled for the VPC that is used. For more information, see Enable the flow log feature.
A project and a Logstore are created to store the Internet traffic logs of VPC flow logs after data transformation. For more information, see Create a project and Create a Logstore.
Background information
VPC provides the flow log feature. The feature records information about inbound and outbound traffic of an elastic network interface (ENI). You can check access control rules, monitor network traffic, and troubleshoot network errors based on the flow logs.
The flow log feature captures traffic information, records the traffic information in logs, and then sends the logs to Simple Log Service. Each log records a specified five-tuple of network traffic that is captured within a specified time window. The time window is approximately 10 minutes. During this time window, the flow log feature aggregates traffic data and sends the traffic data that is recorded as logs to Simple Log Service. If you enable the flow log feature for a VPC or a vSwitch, traffic that is transferred over the ENIs in the VPC or the vSwitch is captured. The ENIs that are created after the flow log feature is enabled are included.
Scenarios
For example, after you enable the flow log feature for your VPC, flow logs are collected and sent to Simple Log Service. Sample log:
{
"vm-id": "i-bp13cg******zs2l",
"srcaddr": "172.16.XX.XX",
"__time__": 1650964251,
"__topic__": "flow_log",
"dstport": "53",
"account-id": "1379******4",
"__source__": "log_service",
"start": "1650862360",
"dstaddr": "100.100.XX.XX",
"vpc-id": "vpc-bp1cznk******vv",
"version": "1",
"packets": "1",
"eni-id": "eni-bp17w******5sfw6m",
"protocol": "17",
"__pack_meta__": "1|MTY1MDk2NDAxOTEyMjczMTQ1NQ==|5|4",
"bytes": "92",
"vswitch-id": "vsw-bp16******wqe6p44",
"srcport": "59986",
"action": "ACCEPT",
"end": "1650862391",
"log-status": "OK",
"direction": "out"
}To analyze Internet traffic when you query and analyze flow logs, you must perform the following operations on the raw logs:
If the srcaddr or dstaddr field does not exist in the raw logs, discard the raw logs.
If the traffic recorded in the raw logs is transferred over internal networks, discard the raw logs.
If you want to analyze Internet traffic in the preceding scenarios, you can use the data transformation feature to transform the collected flow logs.
Procedure
Log on to the Simple Log Service console.
In the Projects section, click the project that you want to manage.
In the left-side navigation pane, click Log Storage. In the Logstores list, click the Logstore that you want to manage.
Click Data Transformation.
In the code editor that appears, enter the following transformation rules:
# If the srcaddr or dstaddr field does not exist in the raw logs, discard the raw logs. e_if(e_not_has("srcaddr"), e_drop()) e_if(e_not_has("dstaddr"), e_drop()) # If the value of the srcaddr or dstaddr field in the raw logs does not conform to the IP address format, discard the raw logs. e_if(op_not(e_match("srcaddr", grok(r'%{IP}'))), e_drop()); e_if(op_not(e_match("dstaddr", grok(r'%{IP}'))), e_drop()); # If the traffic recorded in the raw logs is transferred over internal networks, discard the raw logs. e_if(op_and( op_or(ip_cidrmatch("10.0.0.0/8", v("srcaddr")), ip_cidrmatch("172.16.0.0/12", v("srcaddr")), ip_cidrmatch("192.168.0.0/16", v("srcaddr")) ), op_or(ip_cidrmatch("10.0.0.0/8", v("dstaddr")), ip_cidrmatch("172.16.0.0/12", v("dstaddr")), ip_cidrmatch("192.168.0.0/16", v("dstaddr")) )),e_drop())Use the e_if and e_not_has functions to delete raw logs that do not contain the srcaddr or dstaddr field. For more information, see e_if, e_not_has, and e_drop.
Use the e_if, op_not, and e_match functions to delete raw logs in which the value of the srcaddr or dstaddr field does not conform to the IP address format. For more information, see op_not and e_match.
Use the e_if, op_and, op_or, and ip_cidrmatch functions to delete raw logs in which recorded traffic is transferred over internal networks. For more information, see op_and, op_or, and ip_cidrmatch.
Click Preview Data.
Only the flow logs that record Internet traffic are retained.
Click Save as Transformation Job.
In the Create Data Transformation Job panel, configure the parameters and click OK.
Configure basic information.
Parameter
Description
Job Name
The name of the data transformation job. Example: vpc-flowlog-public.
Authorization Method
The method that is used to authorize Simple Log Service to read data from the source Logstore. Example: Default Role.
Configure a storage destination.
Parameter
Description
Destination Name
The name of the storage destination. Example: target-a.
Destination Region
The region where the destination project resides. Example: China (Hangzhou).
Destination Project
The name of the project that is used to store Internet traffic logs. Example: project-vpc-flowlog-public.
Target Store
The name of the Logstore that is used to store Internet traffic logs. Example: logstore-vpc-flowlog-public.
Authorization Method
The method that is used to authorize Simple Log Service to read data from and write data to the destination Logstore.
Example: Default Role.
Specify a time range for data transformation.
Parameter
Description
Time Range
The time range for data transformation. If you select All, Simple Log Service transforms all data in the source Logstore.
After the data transformation job is created, you can perform the following operations:
View the details and status of the data transformation job. You can also start, stop, modify, or delete the data transformation job. For more information, see Manage a data transformation job.
Go to the destination Logstore to view the Internet traffic logs of the VPC flow logs. The Logstore stores only Internet traffic logs.
For example, you can enter the following query statement to query and analyze Internet traffic by source city and destination city:
*|select ip_to_city(srcaddr) as sourceAddr,ip_to_city(dstaddr) as dstAddr,COUNT(*) as pv group by sourceAddr,dstAddr order by pv limit 10