Elastic Compute Service:How do I query and analyze the network traffic loads of a Linux instance?

Scenarios

When you use a Linux ECS instance, the bandwidth of the instance may be overutilized. You can perform the following steps to troubleshoot the issue:

1. Find the processes that cause high bandwidth utilization.

   You can use utilities such as SAR, iftop, and NetHogs to find the processes. For more information, see the Use SAR, Use iftop, and Use NetHogs sections of this topic.

2. Check whether the processes that cause high bandwidth utilization are running as expected, and handle the issue based on the status of the processes.

   • If the processes are running as expected, you can upgrade the bandwidth of the instance. For more information, see Modify the public bandwidth configurations of an instance associated with an auto-assigned public IP address or Modify the bandwidth configurations of pay-as-you-go instances.

   • If the processes are not running as expected, you can manually check and terminate the processes. You can also use a third-party security tool to check and terminate the processes.

Use SAR

SAR is one of the most comprehensive utilities for system performance analytics in Linux. The utility can report system activities from multiple dimensions. SAR can analyze statistics about sent and received data packets and error messages at the network interface level.

sar command

Syntax of a common command that uses the SAR utility to monitor network traffic:

sar -n DEV [interval] [count]

Example

Run the following command to use SAR to collect activity statistics on network interfaces every second for five consecutive times:

sar -n DEV 1 5

The system displays the average values of the activities on each network interface in the end of the command output based on the collected statistics. You can obtain the name of the network interface that fully utilizes the bandwidth from the command output. For ECS instances, the network interface for the internal network is named eth0 and the network interface for the Internet is named eth1.

A command output similar to the following one is displayed, which indicates that the eth1 network interface has a large amount of traffic. You can perform the following steps to troubleshoot the issue.

Fields in the command output:

• IFACE: the name of the network interface.

• rxpck/s and txpck/s: the number of packets received or sent per second.

• rxkB/s and txkB/s: the number of bytes received or sent per second, in kB/s.

• rxcmp/s and txcmp/s: the number of compressed packets received or sent per second.

• rxmcst/s: the number of multicast packets received per second.

Use iftop

iftop is a free utility in Linux for monitoring traffic on network interfaces in real time. iftop can monitor the real-time traffic, port connection information, and reverse DNS resolution information of a specific network interface.

iftop command

Syntax of a common iftop command:

iftop [-i <interface>]

The <interface> variable specifies the name of a network interface, such as eth0 or eth1. If you do not specify a network interface name by using the -i parameter, the usage of the first network interface is detected by default.

Example

1. Connect to the Linux ECS instance.

   For more information, see Connection method overview.

2. Run the following command to install the iftop utility:

   yum install iftop -y

3. Run the following command to query traffic loads on ports:

   iftop -i eth0 -P

   Note

   To display the port numbers for connections, specify the -P parameter.

   A command output similar to the following one is displayed, which indicates that port 58690 has the largest amount of traffic.

4. Run the following command to obtain the process that uses the port:

   netstat -tunlp |grep [$Port]

   Note

   Set the [$Port] parameter to the port that you obtained in the previous step.

   A command output similar to the following one is displayed.

5. After you find the service corresponding to the port that has the largest amount of traffic, you can stop the service or use the iptables service to handle the relevant addresses.

   For example, you can mask an IP address or throttle traffic for an IP address to ensure that the bandwidth of the instance can be used as expected.

Fields in the command output:

• Section ①: Bandwidth utilization.

• Section ②: External connections, which record the IP addresses that are connected to the instance.

  • The three columns on the right represent the average amounts of real-time traffic within 2 seconds, 10 seconds, and 40 seconds for an IP address connected to the instance.

  • The => symbol represents data transmission and the <= symbol represents data reception.

• Section ③:

  • The first column: TX indicates the sent traffic, RX indicates the received traffic, and TOTAL indicates the total traffic.

  • cum in the second column: indicates the total amount of traffic for each situation in the first column.

  • peak in the third column: indicates the highest amount of traffic for each situation in the first column.

  • rates in the fourth column: indicates the average amounts of traffic within 2 seconds, 10 seconds, and 40 seconds for each situation in the first column.

Common command parameters

In the iftop interface, you can use the following shortcut keys to customize the command output:

• h: specifies whether to display help.

• n: toggles to display the IP address or host name of the local machine.

• s: specifies whether to display the host information of the local machine.

• d: specifies whether to display the host information of the remote destination host.

• t: toggles between display modes, which can be two-line display for sent and received traffic or one-line display for only the sent traffic or only the received traffic.

• N: toggles to display port numbers or port service names.

• S: specifies whether to display the port information of the local machine.

• D: specifies whether to display the port information of the remote destination host.

• p: specifies whether to display port information.

• P: toggles to pause or continue display.

• b: specifies whether to display the average traffic graph bar.

• B: toggles to calculate the average amounts of traffic within 2 seconds, 10 seconds, or 40 seconds.

• T: specifies whether to display the total amount of traffic on each connection.

• L: turns on the screen filtering feature and accepts characters that you enter as a filter condition. For example, after you enter an IP address and press the Enter key, the screen displays only the traffic information related to the IP address.

• L: switches the scale on the top of the display screen. The flow graph bar varies based on the scale.

• J or k: scrolls up or down the connection records displayed on the screen.

• 1 or 2 or 3: sorts the traffic data according to the three columns displayed on the right.

• <: sorts the command output by the local hostname or IP address.

• >: sorts the command output by the hostname or IP address of the remote destination host.

• o: specifies whether to display only the current connection.

• f: edits the filter code.

• ! : calls shell commands.

• q: exits.

Use NetHogs

NetHogs is an open source utility that monitors network traffic. NetHogs can be used to display the bandwidth utilization of each process. This allows you to locate the source of abnormal traffic in a more intuitive manner. NetHogs supports IPv4, IPv6, and connections over local network interfaces and Point-to-Point Protocol (PPP). For more information about how to install the NetHogs utility, see the official help documentation. After the installation is complete, enter nethogs to start the utility. If you do not specify any parameters, NetHogs monitors the eth0 interface by default. You can use commands such as ifconfig to find which network interface (eth1 or eth0) serves traffic over the Internet.

nethogs command

Syntax of a common nethogs command:

nethogs [-i <interface>]

The <interface> variable specifies the name of a network interface, such as eth0 or eth1. If you do not specify a network interface name by using the -i parameter, the usage of the first network interface is detected by default.

Example

1. Connect to the Linux ECS instance.

   For more information, see Connection method overview.

2. Run the following command to install the NetHogs utility:

   yum install nethogs -y

3. Run the following command to query the process-level traffic statistics on network interfaces:

   nethogs eth1

   Note

   The default monitoring interval of NetHogs is 1 second. You can use the -d parameter to set the monitoring interval. For example, run the nethogs eth1 -d 5 command to set the monitoring interval to 5 seconds.

   A command output similar to the following one is displayed.

4. If you determine that a process is malicious, run the following command to terminate the process:

   kill -TERM [$Port1]

   Note

   For a web service program, you can use utilities, such as iftop, to query a specific source IP address and then analyze the web access logs to identify whether the traffic from the IP address is malicious. You can use utilities, such as logwatch and awstats, to analyze logs.

Fields in the command output

On the NetHogs monitoring interface, you can sort processes by the SENT column by pressing the s key and by the RECEIVED column by pressing the r key, and display statistics in different units by pressing the m key. The units include KB/s, KB, B, and MB.

• The PID column indicates the process number of the program associated with the corresponding traffic.

• The USER column indicates the owner of the process.

• The PROGRAM column indicates the execution path of the program.

• The DEV column indicates the name of the monitored network interface.

• The SENT column indicates the amount of data traffic that was sent by the process.

• The RECEIVED column indicates the amount of data traffic that was received by the process.