Configuration guide for TCP-RT - Alibaba Cloud Linux - Alibaba Cloud Documentation Center

TCP-RT is a TCP monitoring tool that uses kernel instrumentation to identify requests and responses in TCP connections and capture related metrics, such as request receipt time and service processing time. TCP-RT is applicable to scenarios where only one concurrent request and response exist on a single connection, such as HTTP/1.1, MySQL, or Redis services. It requires Alibaba Cloud Linux 2 (kernel version 4.19.91-21.al7 or later) or Alibaba Cloud Linux 3. Alibaba Cloud Linux 3 (kernel version 5.10.134-17 or later) provides a new version of TCP-RT that adds support for HTTP/TLS protocols while retaining all features of the original version.

Note

To distinguish between the versions, this document uses tcp_rt.ko to refer to the older version and tcprt for the newer version. The core differences are as follows:

tcp_rt.ko: It is used as a kernel module.
tcprt: It is used with systemctl and a configuration file and provides additional features.

Use tcprt whenever possible.

How it works

Local mode (server perspective)

When a TCP service handles one concurrent request and response, the timeline is shown in the following figure.

The Nth request sent by the client to the server is labeled ReqN and consists of two packets: ReqN-1 and ReqN-2. The server records the arrival time of the first packet as T0 and the arrival time of the second packet as T1. After the server receives the full request, it processes the request and sends two response packets: RspN-1 and RspN-2. The server records the time at which it sends the first response packet as T2. After the client receives the response, it sends an ACK back to the server. The server records the time at which it receives the final ACK as T3.

Based on these timestamps, TCP-RT calculates the following metrics:

upload_time: The time when the user initiated the upload request.
process_time: The server processing time.
This is the interval between the time when the server receives the last request packet and the time when the server starts to send the first response packet.
download_time: The data download time.
This is the interval between the time when the server starts to send the response packets and the time when the server receives the final ACK from the client. This metric is important for large data responses.

Peer mode (client perspective)

When a TCP service handles one concurrent request and response, the timeline is shown in the following figure.

The Nth request (ReqN) sent from the client to the server consists of two data packets: ReqN-1 and ReqN-2. The server records the time it receives the first packet as T0 and the time it receives the second packet as T1. After receiving and processing the request, the server sends two response packets, RspN-1 and RspN-2, to the client. The server records the time it sends the first response packet as T2. After receiving the response packets, the client sends an ACK to the server, and the server records the time it receives the final ACK as T3.

Based on these timestamps, TCP-RT calculates the following metrics:

upload_time: The time when the user makes an upload request.
process_time: The server processing time.
This is the interval between the time when the client sends the last request packet and the time when the client receives the first response packet.
download_time: The data download time.
This is the interval between the time when the client receives the first response packet and the time when the client receives the last response packet.

Output details

TCP-RT collects TCP service metrics in kernel mode and outputs the metrics in the format described in the following table.

File type

Output method

Output timing

Log file

Output via debugfs to the path /sys/kernel/debug/tcp-rt, in files named rt-network-log*. These files have the following characteristics:

The filename suffix is the CPU core ordinal number. For example, on a 32-core server, log files range from rt-network-log0 to rt-network-log31.
Each file has a maximum size of 2 MB. When this limit is exceeded, older data is purged.
Log files output via debugfs are one-time use. After you read the file once, the data you read is cleared.

Timing 1: When the next task (one request/response pair per task) begins on a TCP connection, TCP-RT outputs information about the previous task.
Event 2: Outputs information when a connection is terminated.
After TCP-RT outputs the data, the application layer can read it immediately.

Stats file

Data is aggregated by port (server port or client port) and output to the file rt-network-stats under /sys/kernel/debug/tcp-rt.

Output occurs periodically. The default interval is one minute.

Parsing modes

TCP-RT supports multiple parsing modes that you can configure. For more information, see Using TCP-RT.

default: The default mode.
Parses generic TCP streams that follow a request/response pattern, such as HTTP, Redis, or MySQL.
http: The enhanced HTTP parsing mode.
This mode is built based on the default mode and reports round-trip time (RTT) and congestion window metrics. It also detects the HTTP “Expect: 100-continue” process and logs the 100-continue request and its response as an R.C record. For more information, see Output format.
Enhanced HTTPS
This mode is built based on the http mode and adds support for Transport Layer Security (TLS) message parsing.
- TLS handshake parsing: Detects the TLS 1.2 and TLS 1.3 handshake phases and marks them in the output logs. For more information, see Output format.
  - For TLS 1.2 (four-way handshake), two R.H records are generated.
  - For TLS 1.3 (three-way handshake), one R.H record is generated, including the handshake duration.
- “Close Notify” Alert message parsing: The message is logged as an R.A record. For more information, see Output format.

Note

The parsing mode for tcp_rt.ko is not configurable and supports only the default mode.

Output format

Note

Definitions of TASK and TCP lifecycle.

A TASK represents a complete request and response cycle.
A TCP lifecycle contains multiple TASKs.

Log file format

In log files, each column in a record corresponds to a piece of information. The following figure shows a sample log entry.The columns are described from left to right as follows:

The version number. The value can be V6 or V7.
- V6: The default version.
- V7: This version appears for R-type records that are generated based on specific configurations with extra fields. This includes the following records:
  - R logs in http or https parsing modes use V7 because they include trailing fields.
  - Extended R log formats, such as R.C, R.Z, R.A, R.H, and R.F, use V7. For more information, see the following table.
The record type identifier. The value can be R, E, W, N, or P.
- R: This record is generated when a request reaches the local server and a request and response cycle is complete within a TCP service.
- E: This record is generated when a connection is closed.
- W: This record is generated when a connection is closed during response transmission.
- N: This record is generated when a connection is closed during request reception.
- P: This record is generated when the local server sends a request to a remote server and a request and response cycle is complete within a TCP service.
The seconds part of the TASK start time.
The microseconds part of the TASK start time.
The remote IP address of the TCP connection.
The remote port of the TCP connection.
The local IP address of the TCP connection.
The local port of the TCP connection.

Additional fields are included based on the record type. For more information, see the following table.

Scenario ID	Field description
R	This record indicates normal start and end of a TASK. A single TCP connection can have multiple R records. Data sent by the TASK. Unit: Byte. Total TASK duration. Unit: us. Interval from arrival of the first request segment to acknowledgment of the last response segment. Minimum TCP RTT (Round Trip Time) during the TASK. Unit: us. Number of retransmitted TCP segments during the TASK. TASK sequence number. The first TASK after TCP establishment is numbered 1. TASK service delay. Unit: us. Interval from `arrival of the last request segment` to `sending of the first response segment`. TASK upload delay. Unit: us. Interval from `arrival of the first request segment` to `arrival of the last request segment`. Data received by the TASK. Unit: Byte. Whether receive reordering occurred during the TASK: 1. The issue occurs. 0: No occurrence. Maximum segment size (MSS) used by TCP during the TASK. Unit: Byte. Smoothed RTT at TASK end. Only effective in http/https parsing modes (tcprt only, V7). Unit: us. Congestion window at TASK end. Only effective in http/https parsing modes (tcprt only, V7). Unit: segments.
R.C	Note Supported only by tcprt (V7). May appear in http/https parsing modes. Format matches R records. Indicates the TASK was triggered by an “Expect: 100-continue” request (no application data included).
R.Z	Note Supported only by tcprt (V7). May appear in https parsing mode. Format matches R records. Indicates the TASK belongs to a TLS 1.3 0-RTT handshake.
R.A	Note Supported only by tcprt (V7). May appear in https parsing mode. Format matches R records. Indicates the TASK contains a `Close Notify` Alert message. The TASK does not include extra time between the last response packet and the Alert packet.
R.H	Note Supported only by tcprt (V7). May appear in https parsing mode. R records may include trailing fields. Indicates the TASK belongs to a TLS handshake phase. For TLS 1.3 (three-way handshake), the log reports handshake duration in microseconds. Trailing field format: `H <time>`. Example: `H 151878` means the handshake took 151878 us.
R.F	Note Supported only by tcprt (V7). Appears when `first_frame` is enabled (see Using TCP-RT). R records may include trailing fields. Indicates the TASK triggers completion time statistics by byte thresholds. Based on byte thresholds defined in the configuration file, the response data stream is divided into steps, and data volume plus completion time for each step are appended to the record (units: bytes and us). Trailing field format: `F <n> [bytes time]`. Example: `F 6 7240 214 63698 311 98436 358` means six numbers total: completion times of 214 us, 311 us, and 358 us correspond to data volumes of 7240, 63698, and 98436 Bytes.
P	This record indicates normal start and end of a TASK. A single TCP service can have multiple P records. P records were added in V6 to capture outbound client requests. They appear only if `pports` or `pports_range` is configured. Data sent by the TASK. Unit: Byte. Total TASK duration. From local server sending data to receiving the final remote response. Unit: us. Minimum TCP RTT during the TASK. Unit: us. Number of retransmitted TCP segments during the TASK. TASK sequence number. The first TASK after TCP establishment is numbered 1. TASK service time. From request completion to first response receipt. Unit: us. TASK response receive time. From first to last response packet receipt. Unit: us. Total response size received by the TASK. Unit: Byte. Specifies whether out-of-order reception occurs during the TASK procedure. Valid values: 1. Occurs. 0: No. Maximum segment size (MSS) used by TCP during the TASK. Unit: Byte.
E	This record indicates TCP connection closure. Each TCP connection has exactly one E record. Connections with `pports` or `pports_range` configured also generate this record. Sequence number of the last TASK. Total data sent during the TCP lifecycle. Unit: Byte. Data sent but not acknowledged. Value is 0 if none. Unit: Byte. Total data received during the TCP lifecycle. Unit: Byte. The number of TCP segments retransmitted by the TASK. Minimum TCP RTT during the TCP lifecycle. Unit: us.
N	This record appears when a TCP connection closes during request segment reception. A connection may have zero or one N record. The ordinal number of the last TASK. The duration of the last TASK includes only the receive time because the task was shut down before it could send. This duration is measured in microseconds (us). The amount of data received during the TCP lifecycle. Unit: Byte. Specifies whether out-of-order reception occurred during the TASK process. A value of 1 indicates that an out-of-order reception occurred, while a value of 0 indicates that it did not. Maximum segment size (MSS) used by TCP during the TASK. Unit: Byte.
W	This record appears when a TCP connection closes during response segment transmission. A connection may have zero or one W record. Amount of response data already sent for the last TASK. Unit: Byte. Duration of the last TASK. Since the connection closed prematurely, send time is incomplete. Unit: us. Minimum TCP RTT for the last TASK. Unit: us. Number of retransmitted TCP segments for the last TASK. Sequence number of the last TASK. Service delay for the last TASK. Unit: us. Upload delay for the last TASK. Unit: us. Response data sent but not acknowledged. Value is 0 if none. Unit: Byte. Specifies whether out-of-order reception occurs during the TASK procedure. The valid values are as follows: 1: Occurs. 0: No. Maximum segment size (MSS) used by TCP during the last TASK. Unit: Byte.

Note

Extended log types use the R.x format, where x can be a single character or a combination of characters. For example, R.AF indicates that both A and F apply.
R.A and R.C logs use the response data volume as a criterion and may have minor false positives.
For the metrics in the preceding table that indicate the amount of data sent by a TASK, if the TASK is the last one in a connection, one is subtracted from the snd_nxt value during normal processing to account for the connection closure (FIN). However, if the connection is closed abnormally, for example, if a TASK receives an abnormal termination message (RST) after the TASK is complete and the connection terminates in response to the reset, the actual amount of data sent is one byte greater than the displayed value. Typically, parameter values related to the amount of data sent by a task are correct, and the margin of error is only one byte.

Note

The three time phases are shown in How it works:

R records report the total time (T0 to T3), upload delay (upload_time, T0 to T1), and service delay (process_time, T1 to T2). The download time (download_time) can be calculated using the following formula:

download_time = total_time - upload_time - process_time

Stats file format

The columns are described from left to right as follows:

The timestamp when the record was output.
A reserved field. The value is always all.
The port number.
The average total duration of a TASK that is obtained from R records.
The average service delay of a TASK that is obtained from R records.
The packet loss rate in per mille (‰).
The average RTT in microseconds (us).
The per mille (‰) of TASKs that are closed during request transmission.
The average amount of data sent per TASK.
The average upload delay of a TASK.
The average amount of data received per TASK.
The number of TASKs that are included in the statistics.

tcprt versus tcp_rt.ko version differences

Comparison item	tcprt	tcp_rt.ko
Log file path	Same
Log format	tcprt extends the log format while maintaining backward compatibility.
Feature management	`systemctl`	`modprobe/rmmod`
Feature configuration	Configuration file: `/etc/tcprt-bpf/tcprt.yaml`	Module parameters: `/sys/module/tcp_rt/parameters/*`
System requirements	Alibaba Cloud Linux 3 (kernel version `5.10.134-17` or later).	Alibaba Cloud Linux 2 (kernel version `4.19.91-21.al7` or later). Alibaba Cloud Linux 3.
Features	Supports http and https parsing modes in addition to default. Supports `first_frame` feature.	Supports only default parsing mode.

Feature usage

Choose one of the following methods based on your environment:

tcprt
For Alibaba Cloud Linux 3 (kernel version 5.10.134-17 or later).
tcp_rt.ko
- For Alibaba Cloud Linux 2 (kernel version 4.19.91-21.al7 or later).
- Alibaba Cloud Linux 3.

Note

Because both share the same log file path, you cannot run them at the same time. You can use only one at a time.

tcprt

Installation and configuration.

Install tcprt.
```
sudo yum install -y tcprt
```

Configure tcprt.

Open the /etc/tcprt-bpf/tcprt.yaml file.
```
sudo vim /etc/tcprt/tcprt.yaml
```

Press i to enter the edit mode and configure parameters as required.

The configuration file has three sections: global, per-port, and init. The sections are described in the following tables.

Global

Parameter	Description	Values	Default
peer	Default tcprt port matching mode. false: Match local ports. true: Match remote ports.	true/false	Empty
stats	Enable stats log output.	enable/disable	disable
stats_interval	Stats output interval. Unit: s.	Integer	60
first_frame_bytes	Byte thresholds for completion time statistics. Unit: byte.	List of up to 4 integers	Empty

per port.

This section is a list that is empty by default. Each element defines a port configuration.

Parameter	Description	Values	Default
port	Port number or range.	An integer representing the port number. Two comma-separated integers: closed interval port range.	None
peer	Matching mode for this port (or range).	true/false	Global peering configuration
mode	Protocol parsing mode. default matches tcp_rt.ko behavior.	default/http/https	default
first_frame	Enable completion time statistics by byte thresholds.	enable/disable	disable

Initialize.

Parameter	Description	Values	Default
log_buf_num	Number of sub-buffers for log files.	Integer	8
log_buf_size	Size of each log sub-buffer. Unit: byte. Maximum log file size is `log_buf_num * log_buf_size`.	Integer	262144
stats_buf_num	Number of sub-buffers for stats files.	Integer	8
stats_buf_size	Size of each stats sub-buffer. Unit: byte. Maximum stats file size is `stats_buf_num * stats_buf_size`.	Integer	16384

After you edit the file, press the Esc key, enter :wq, and then press the Enter key to save the file.
Start the service to automatically load the configuration.
```
sudo systemctl start tcprt
```
(Optional) You can enable automatic startup at boot.
```
sudo systemctl enable tcprt
```

Uninstall tcprt.
```
sudo yum remove tcprt
```
Type y and press the Enter key to confirm the uninstallation.

tcprt configuration example

The peer parameter is set to false to match local ports by default. Two port configurations are added:
- Port 443 (exact match): HTTPS parsing is enabled to match the local port. The first_frame feature (completion time by byte thresholds) is enabled with thresholds of 100, 2000, and 20000 bytes.
- Port range 56789 to 56800: The default TCP parsing is used to match remote ports.
The stats feature is disabled.
The default buffer sizes are used for logs and stats.

global:
  peer: false
  stats: disable
  stats_interval: 60
  first_frame_bytes: [100, 2000, 20000, 0]
per-port:
  -
    port: 443
    mode: https
    first_frame: enable
  -
    port: 56789,56800
    mode: default
    peer: true

init:
  log_buf_num: 8
  log_buf_size: 262144
  stats_buf_num: 8
  stats_buf_size: 16384

Note

Byte completion times are tracked using ACK packets. In practice, one ACK packet may acknowledge a large amount of data, such as tens of kilobytes. Therefore, the logged data volumes may exceed the configured byte thresholds.

tcp_rt.ko

Load and configure the module.

You can use one of the following methods.

Configure parameters during module loading.
1. Load the tcp_rt module and configure parameters.
  For example, load the tcp_rt module and configure lports to 80.
```
sudo modprobe tcp_rt lports=80
```
2. Verify the configuration.
```
sudo cat /sys/module/tcp_rt/parameters/lports
```

Load the module and then configure parameters.

Load the module.
```
sudo modprobe tcp_rt
```

After the module is loaded, configure parameters in the /sys/module/tcp_rt/parameters/ directory.

Example: Monitor local port 80.

sudo sh -c 'echo 80 > /sys/module/tcp_rt/parameters/lports'

Verify the configuration.

sudo cat /sys/module/tcp_rt/parameters/lports

The parameters and commands are described in the following table.

Parameter	Description	Default	Command
stats	Enable stats output: 0: No. 1: Yes.	0	`echo 0 > stats`
stats_interval	Stats output interval. Unit: s.	60	`echo 60 > stats_interval`
lports	Local server ports to monitor (up to 6).	None	`echo 80,800,8080 > lports`
pports	Remote ports of TCP connections.	None	`echo 80,800,8080 > pports`
lports_range	Local server port ranges. Two numbers per range. Example command configures 80–100 and 1000–2000.	None	`echo 80,100,1000,2000 >lports_range`
pports_range	Remote port ranges. Two numbers per range. Example command configures 80–100 and 1000–2000.	None	`echo 80,100,1000,2000 >pports_range`
log_buf_num	Maximum log file size is `log_buf_num * 256 k`. Configurable only at module load.	8	`modprobe tcp_rt log_buf_num=10`
stats_buf_num	Maximum stats file size is `stats_buf_num * 16 k`. Configurable only at module load.	8	`modprobe tcp_rt stats_buf_num=10`

Unload the module.
1. Run the following command to deactivate the tcp-rt module. This way, no new connections use TCP-RT.
```
sudo echo 1 > /sys/kernel/debug/tcp-rt/deactivate
```
2. Run the following command to verify that no connections are using the tcp-rt module.
```
lsmod
```
  If the value of Used by for the tcp-rt module is 0, no connections are using tcp-rt.
3. If no connections are using the tcp-rt module, run the following command to uninstall the tcp-rt module.
```
sudo rmmod tcp_rt
```

FAQ

Log file size differs from configuration

Commands such as ls -l may show file sizes that are larger than the configured size. This is normal. Log files are stored in the /sys virtual filesystem. Therefore, the ls -l command does not reflect the actual disk usage because these files do not consume disk space. The displayed size represents the cumulative log volume and increases over time. To check the actual log size, you can redirect the logs to a regular file and inspect the file.

The following figure shows an example of the default settings (2 MB). If you use the cat command to redirect the log content to a file, the actual size of the file is approximately 2 MB, as expected.