TCP-RT configurations - Alibaba Cloud Linux - Alibaba Cloud Documentation Center

TCP-RT is a TCP monitoring tool that uses kernel hooking technology to identify requests and responses on TCP connections and capture information such as request receiving times and service processing times. TCP-RT is suitable for scenarios in which only one concurrent request and response exist on a connection, such as HTTP/1.1 web services, MySQL services, and Redis services, and can run only in Alibaba Cloud Linux 2 with kernel version 4.19.91-21.al7 or later or Alibaba Cloud Linux 3. Alibaba Cloud Linux 3 with kernel version 5.10.134-17 or later provides a new version of TCP-RT, which inherits all features of the old version of TCP-RT and adds support for the HTTP and Transport Layer Security (TLS) protocols.

Note

To provide clarity in the content that involves the functional differences between the old and new versions of TCP-RT, tcp_rt.ko is used to indicate the old version and tcprt is used to indicate the new version. The old and new versions of TCP-RT differ in terms of the usage.

tcp_rt.ko: is used as a kernel module.
tcprt: is used together with configuration files in systemctl and provides new features.

We recommend that you use tcprt.

How TCP-RT works

The following figure shows different phases in which TCP-RT is used for a TCP connection that carries one concurrent request and response.

In the preceding figure, ReqN indicates the Nth request sent by the client to the server. The request consists of the ReqN-1 and ReqN-2 packets. The point in time at which the server receives the first packet is recorded as T0 and the point in time at which the server receives the second packet is recorded as T1. The server processes the received requests and sends the RspN-1 and RspN-2 response packets to the client. The point in time at which the server sends the first response packet is recorded as T2. When the client receives a response packet, the client sends an acknowledgment (ACK) message to the server. The point in time at which the server receives the final ACK message is recorded as T3.

TCP-RT can calculate the following periods of time based on the recorded points in time:

upload_time: the amount of time required to upload the request.
process_time: the amount of time required by the server to process the request.
The value of process_time is a period of time that starts when the server receives the final request packet and ends when the server sends the first response packet to the client.
download_time: the amount of time required to download the data.
The value of download_time is a period of time that starts when the server sends the first response packet to the client and ends when the server receives the final ACK message from the client. This period of time is essential for downloading large amounts of response data.

Information output

TCP-RT collects TCP-related parameters in kernel space and generates parameter files. The following table describes the output modes and output time of the parameter files.

File type

Output mode

Output time

Log file

Log files are the rt-network-log* files that are generated in the /sys/kernel/debug/tcp-rt directory by using debugfs. Log files have the following characteristics:

The suffix of the file name is the serial number of the CPU core. For example, the log files generated by a 32-core server are named from rt-network-log0 to rt-network-log31.
Each log file can be up to 2 MB in size. If a log file exceeds the size limit, the oldest data is deleted from the file to store new data.
Log files that are generated by using debugfs are one-time files. After data is read from a one-time file, the data is deleted.

T1: On a TCP connection, information about a task is generated when the task is completed and the next task starts. A task consists of a request and a response.
T2: Information is generated when the connection is closed.
After data is generated by TCP-RT, the application layer can immediately read the data.

stats file

stats files are the rt-network-stats files that are generated based on the server port-related or client port-related statistics in the /sys/kernel/debug/tcp-rt directory.

stats files are generated at regular intervals. By default, data is generated once every minute.

Parsing modes

TCP-RT supports multiple parsing modes. You can configure the parsing mode of TCP-RT based on your business requirements. For more information, see the Operations section of this topic.

default: the default mode.
In default mode, TCP-RT tracks regular TCP streams, which can be streams of request-response protocol applications, such as HTTP, Redis, and MySQL applications.
http: the HTTP mode,
which is built on the default mode. In HTTP mode, TCP-RT additionally captures round trip times (RTTs) and congestion windows, identifies HTTP requests that include an Expect: 100-continue header and the responses to the requests, and reports each Expect:100-continue request and response pair as an R.C record. For more information, see the Output formats section of this topic.
https: the HTTPS mode,
which is built on the HTTP mode. In HTTPS mode, TCP-RT parses the following TLS packets:
- TLS handshakes: TCP-RT identifies TLS 1.2 handshakes and TLS 1.3 handshakes and generates logs for the handshakes. For more information, see the Output formats section of this topic.
  - For a four-way TLS 1.2 handshake, TCP-RT generates two R.H records.
  - For a three-way TLS 1.3 handshake, TCP-RT generates an R.H record and reports the handshake duration.
- Close Notify Alert packets: TCP-RT generates R.A records in logs for Close Notify Alert packets. For more information, see the Output formats section of this topic.

Note

tcp_rt.ko supports only the default mode. You cannot configure the parsing mode of tcp_rt.ko.

Output formats

Note

The lifecycle of a task and the lifecycle of a TCP connection are different.

The lifecycle of a task consists of a request and a response.
The lifecycle of a TCP connection includes multiple tasks.

Log file format

In log files, each column of a record corresponds to a different set of information. The following figure shows a sample log file. The following section describes the parameters from left to right, as shown in the figure.

The version number, which can be V6 or V7.
- V6: V6 is the default version number.
- V7: R records may include additional information in specific configurations. The version number for the R records is V7.
  - In HTTP or HTTPS parsing mode, R records end with additional fields and the version number of the R records is V7.
  - The version number of extended R records, such as R.C, R.Z, R.A, R.H, and R.F records, is V7. For information about R records in the extended formats, see the following table.
The scenario record identifier. Valid values: R, E, W, N, and P.
- R: An R record is generated for the TCP connection when a request is received by the local server and a response is sent.
- E: An E record is generated when the TCP connection is closed.
- W: A W record is generated when the TCP connection is closed during data transmission.
- N: An N record is generated when the TCP connection is closed during data reception.
- P: A P record is generated for the TCP connection when the local server sends a request to the peer server and a response is sent.
The seconds part of the start time of the task.
The microseconds part of the start time of the task.
The peer IP address of the TCP connection.
The peer port of the TCP connection.
The local IP address of the TCP connection.
The local port of the TCP connection.

The preceding parameters are followed by different parameters in different scenarios. The following table describes the parameters.

Scenario record identifier	Parameter description
R	An R record records the normal start and closure of a task. Each TCP connection can have multiple R records. The amount of data sent by the task. Unit: bytes. The duration of the task. Unit: microseconds. The duration is the interval between the point in time at which the first request segment is received and the point in time at which the final response segment is acknowledged. The minimum TCP RTT of the task. Unit: microseconds. The number of TCP segments retransmitted by the task. The sequence number of the task. After a TCP connection is established, the sequence number of the first task is 1. The latency of the task. Unit: microseconds. The latency is the interval between the point in time at which `the final request segment is received` and the point in time at which `the first response segment is sent`. The latency for sending request segments of the task. Unit: microseconds. The latency is the interval between the point in time at which `the first request segment is received` and the point in time at which `the final request segment is received`. The amount of data received by the task. Unit: bytes. Indicates whether unordered reception occurs during the task. Valid values: 1: Unordered reception occurs during the task. 0: Unordered reception does not occur during the task. The TCP maximum segment length (MSS) used during the task. Unit: bytes. The smoothed RTT (SRTT) at the end of the task. This parameter takes effect in HTTP or HTTPS parsing mode. Only tcprt supports the HTTP and HTTPS parsing modes. The version number of R records in HTTP or HTTPS parsing mode is V7. Unit: microseconds. The congestion window at the end of the task. This parameter takes effect only in HTTP or HTTPS parsing mode. Only tcprt supports the HTTP and HTTPS parsing modes. The version number of R records in HTTP or HTTPS parsing mode is V7. Unit: segment.
R.C	Note Only tcprt supports this type of record. The version number of this type of record is V7. This type of record may be generated in HTTP or HTTPS parsing mode and is in the same format as R records. An R.C record records a task that consists of an Expect: 100-continue request and response pair. The task does not contain application data.
R.Z	Note Only tcprt supports this type of record. The version number of this type of record is V7. This type of record may be generated in HTTPS parsing mode and is in the same format as R records. An R.Z record records a task that is included in a TLS 1.3 0-RTT handshake.
R.A	Note Only tcprt supports this type of record. The version number of this type of record is V7. This type of record may be generated in HTTPS parsing mode and is in the same format as R records. An R.A record records a task that contains a `Close Notify` Alert packet. This task is not counted as an Alert packet to increase the packet processing time, which is the interval between the arrival of an Alert packet and the point in time at which the final response packet is sent.
R.H	Note Only tcprt supports this type of record. The version number of this type of record is V7. This type of record may be generated in HTTPS parsing mode and may end with additional information. An R.H record records a task that is included in a TLS handshake. If an R.H record records a task that is included in a three-way TLS 1.3 handshake, the record ends with the handshake duration in the `H <time>` format. Unit: microseconds. For example, `H 151878` indicates that the handshake requires 151,878 microseconds to complete.
R.F	Note Only tcprt supports this type of record. The version number of this type of record is V7. This type of record is generated after you enable the `first_frame` feature and may end with additional information. For information about how to enable the first_frame feature, see the Operations section of this topic. An R.F record records the bytes-based completion time statistics about a task. TCP-RT divides the response stream of each task based on the bytes-based threshold steps that are defined in the configuration file, calculates the data volume in units of bytes and the completion time in units of microseconds for each step, and then appends the calculation results to the end of an R.F record. The appended calculation results are in the `F <n> [bytes time]` format. For example, `F 6 7240 214 63698 311 98436 358` indicates the data volumes of 7,240 bytes, 63,698 bytes, and 98,436 bytes and the respective completion times of 214 microseconds, 311 microseconds, and 358 microseconds for the data volumes.
P	A P record records the normal start and closure of a task. Each TCP connection can have multiple P records. P is a new record type in V6 and indicates the request information from the client. This record is available only if the `pports` or `pports_range` option is configured. The amount of data sent by the task. Unit: bytes. The duration of the task, which is the interval between the point in time at which the local server sends data and the point in time at which the local server receives the final response from the peer end. Unit: microseconds. The minimum TCP RTT of the task. Unit: microseconds. The number of TCP segments retransmitted by the task. The sequence number of the task. After a TCP connection is established, the sequence number of the first task is 1. The service time of the task, which is the period of time that starts when the request is sent and ends when the first response is received. Unit: microseconds. The amount of time during which the response for the task is received, which is the interval between the point in time at which the first response packet is received and the point in time at which the final response packet is received. Unit: microseconds. The total size of the response packets received by the task. Unit: bytes. Indicates whether unordered reception occurs during the task. Valid values: 1: Unordered reception occurs during the task. 0: Unordered reception does not occur during the task. The TCP MSS used during the task. Unit: bytes.
E	An E record records information about a closed TCP connection. Each closed TCP connection has an E record regardless of whether `pports` or `pports_range` option is configured. The sequence number of the final task. The amount of data sent during the TCP connection lifecycle. Unit: bytes. The amount of data for which the TCP connection has sent a response but has not received the ACK message. A value of 0 indicates that this type of data does not exist. Unit: bytes. The amount of data received during the TCP connection lifecycle. Unit: bytes. The number of TCP segments retransmitted by the task. The minimum TCP RTT in the TCP connection lifecycle. Unit: microseconds.
N	An N record records information about a TCP connection that is closed when the task is requesting response segments. Each TCP connection may have zero or one N record. The sequence number of the final task. The duration of the final task. Only the receiving time is available because the connection is closed. Unit: microseconds. The amount of data received during the TCP connection lifecycle. Unit: bytes. Indicates whether unordered reception occurs during the task. A value of 1 indicates that unordered reception occurs, and a value of 0 indicates that unordered reception does not occur. The TCP MSS used during the task. Unit: bytes.
W	A W record records information about a TCP connection that is closed when the task is responding to request segments. Each TCP connection may have zero or one W record. The amount of data sent in the response of the final task. Unit: bytes. The duration of the final task. The response process of the final task is interrupted and not completed because the connection is closed. Unit: microseconds. The minimum TCP RTT of the final task. Unit: microseconds. The number of TCP segments retransmitted by the final task. The sequence number of the final task. The latency of the final task. Unit: microseconds. The latency for sending request segments of the final task. Unit: microseconds. The amount of data for which the final task has sent a response but has not received the ACK message. A value of 0 indicates that this type of data does not exist. Unit: bytes. Indicates whether unordered reception occurs during the task. Valid values: 1: Unordered reception occurs during the task. 0: Unordered reception does not occur during the task. The TCP MSS used during the final task. Unit: bytes.

Note

R.x records are extended log records. x can be a single character, as shown in the preceding table, or a character combination. Example: R.AF, which indicates R.A and R.F.
R.A and R.C records are based on response data volumes and may contain a few errors.
If the connection is closed as expected, the snd_nxt value is deducted from the amount of data sent by the final task. If the connection is abnormally closed, such as when a reset packet is received after a task is completed and the connection is closed with a reset packet returned, the actual amount of data sent is one byte larger than the displayed value. In most cases, parameter values related to the amount of data sent by a task are valid, and the margin of error is only one byte.

Note

The following periods of time are calculated based on the points in time described in the How TCP-RT works section of this topic:

Total time recorded in an R record (T3 - T0), upload_time (T1 - T0), and process_time (T2 - T1). The value of download_time can be calculated by using the following formula:

download_time = Total time - upload_time - process_time

stats file format

The following section describes the parameters from left to right as recorded:

The timestamp at which the record was generated.
The reserved field. Valid value: all.
The port number.
The average duration of tasks recorded in R records.
The average latency of tasks recorded in R records.
The packet loss rate, which is expressed as a permillage of packets lost with respect to packets sent.
The average value of RTT. Unit: microseconds.
The permillage of tasks closed with respect to all tasks in the data sent by requests.
The average size of data sent by tasks.
The average latency for sending requests of tasks.
The average amount of data received by tasks.
The number of tasks.

Comparison between tcprt and tcp_rt.ko

Item	tcprt	tcp_rt.ko
Path in which log files are stored	Same
Log format	tcprt supports logs in extended formats and is compatible with logs in original formats.
Feature management	`systemctl`	`modprobe/rmmod`
Feature configuration	Configuration file: `/etc/tcprt-bpf/tcprt.yaml`	Module parameters: `/sys/module/tcp_rt/parameters/*`
Supported operating system	Alibaba Cloud Linux 3 with kernel version `5.10.134-17` or later	Alibaba Cloud Linux 2 with kernel version `4.19.91-21.al7` or later Alibaba Cloud Linux 3
Supported feature	Default, HTTP, and HTTPS parsing modes `first_frame` feature	Default parsing mode

Operations

You can perform the following operations to use TCP-RT based on your business scenario.

tcprt
Alibaba Cloud Linux 3 with kernel version 5.10.134-17 or later
tcp_rt.ko
- Alibaba Cloud Linux 2 with kernel version 4.19.91-21.al7 or later
- Alibaba Cloud Linux 3

Note

tcprt and tcp_rt.ko use the same path to store log files. You cannot use tcprt and tcp_rt.ko at the same time.

tcprt

Install and configure tcprt.

Install tcprt.
```
sudo yum install -y tcprt
```

Configure tcprt.

Open the /etc/tcprt-bpf/tcprt.yaml file.
```
sudo vim /etc/tcprt/tcprt.yaml
```

Press the I key to enter Insert mode and configure parameters based on your business requirements.

Configure the parameters in the global, per-port, and init sections. The following tables describe the parameters.

global

Parameter	Description	Valid value	Default value
peer	Specifies whether to match peer ports. false: matches local ports. true: matches peer ports.	true/false	None
stats	Specifies whether to enable the feature of generating stats files.	enable/disable	disable
stats_interval	The interval at which to generate stats files. Unit: seconds.	Integers	60
first_frame_bytes	The threshold steps for bytes-based completion time statistics. Unit: bytes.	A list of up to four integers	None

per-port

The per-port section can contain one or more elements. Each element consists of a list of parameters and specifies a port configuration. By default, the per-port section is empty. The following table describes the parameters in each element.

Parameter	Description	Valid value	Default value
port	The port or port range.	An integer: indicates a single port number. Two integers separated with a comma (,): indicate a port range. Format: <Start port number>,<End port number>.	None
peer	Specifies whether the specified port or port range matches peer ports.	true/false	Value of the peer parameter in the global section
mode	The protocol parsing mode. If you set this parameter to default, tcprt parses TCP streams in the same manner as tcp_rt.ko.	default/http/https	default
first_frame	Specifies whether to enable the first_frame feature that collects bytes-based completion time statistics.	enable/disable	disable

init

Parameter	Description	Valid value	Default value
log_buf_num	The number of sub-buffers for log files.	Integers	8
log_buf_size	The sub-buffer size for log files. Unit: bytes. The maximum size of a log file is calculated by using the following formula: `log_buf_num × log_buf_size`.	Integers	262144
stats_buf_num	The number of sub-buffers for stats files.	Integers	8
stats_buf_size	The sub-buffer size for stats files. Unit: bytes. The maximum size of a stats file is calculated by using the following formula: `stats_buf_num × stats_buf_size`.	Integers	16384

Press the Esc key, enter :wq, and then press the Enter key to save and close the file.
Start tcprt to automatically parse the configuration file.
```
sudo systemctl start tcprt
```
(Optional) Configure tcprt to run on system startup.
```
sudo systemctl enable tcprt
```

Uninstall tcprt.
```
sudo yum remove tcprt
```
Enter y and press the Enter key to uninstall tcprt.

Sample configuration file of tcprt

Set the peer parameter in the global section to false, which specifies that local ports are matched. Two port configurations are added.
- Port 443: Use the peer parameter setting in the global section, enable the HTTPS parsing mode, and then enable the first_frame feature. Set the threshold steps for bytes-based completion time statistics to 100 bytes, 2,000 bytes, and 20,000 bytes.
- Port range of 56789 to 56800: Enable the default parsing mode and specify that the port range matches peer ports.
Disable the feature of generating stats files.
Use the default sub-buffer settings for log files and stats files.

global:
  peer: false
  stats: disable
  stats_interval: 60
  first_frame_bytes: [100, 2000, 20000, 0]
per-port:
  -
    port: 443
    mode: https
    first_frame: enable
  -
    port: 56789,56800
    mode: default
    peer: true

init:
  log_buf_num: 8
  log_buf_size: 262144
  stats_buf_num: 8
  stats_buf_size: 16384

Note

Bytes-based completion time is measured by using ACK packets. In actual data transmission, an ACK packet may correspond to a large amount of data, such as tens of KB of data. The amount of logged data may exceed the bytes threshold configured in a log file.

tcp_rt.ko

Load and configure the tcp_rt.ko module.

You can use one of the following methods to load and configure the module:

Configure the parameters when you load the module.
1. Configure the parameters in the command that you run to load the module.
  For example, set the lports parameter to 80 in the following command that you run to load the tcp_rt.ko module:
```
sudo modprobe tcp_rt lports=80
```
2. Check whether the configurations take effect.
```
sudo cat /sys/module/tcp_rt/parameters/lports
```

Load the module and then configure the parameters.

Run the following command to load the module:
```
sudo modprobe tcp_rt
```

After the module is loaded, run a command to configure the parameters in the /sys/module/tcp_rt/parameters/ file.

For example, run the following command to configure a parameter in the /sys/module/tcp_rt/parameters/ file to listen on local port 80:

sudo sh -c 'echo 80 > /sys/module/tcp_rt/parameters/lports'

Check whether the configurations take effect.

sudo cat /sys/module/tcp_rt/parameters/lports

The following table describes the parameters in the /sys/module/tcp_rt/parameters/ file and the commands that you can run to configure the parameters.

Parameter	Description	Default value	Command
stats	Specifies whether to generate stats files. Valid values: 0: does not generate stats files. 1: generates stats files.	0	`echo 0 > stats`
stats_interval	The interval at which to generate stats files. Unit: seconds.	60	`echo 60 > stats_interval`
lports	The local server ports used to collect data. You can specify up to six ports.	None	`echo 80,800,8080 > lports`
pports	The peer port of the TCP connection.	None	`echo 80,800,8080 > pports`
lports_range	The local server port range. Specify a port range in the following format: <Start port number>,<End port number>. For example, a value of 80,100 specifies the port range of 80 to 100, and a value of 1000,2000 specifies the port range of 1000 to 2000.	None	`echo 80,100,1000,2000 >lports_range`
pports_range	The peer port range of the TCP connection. Specify a port range in the following format: <Start port number>,<End port number>. For example, a value of 80,100 specifies the port range of 80 to 100, and a value of 1000,2000 specifies the port range of 1000 to 2000.	None	`echo 80,100,1000,2000 >pports_range`
log_buf_num	The number of sub-buffers for log files. The maximum size of a log file is calculated by using the following formula: `log_buf_num × 256 k`. You can configure this parameter only when you load the module.	8	`modprobe tcp_rt log_buf_num=10`
stats_buf_num	The number of sub-buffers for stats files. The maximum size of a stats file is calculated by using the following formula: `stats_buf_num × 16 k`. You can configure this parameter only when you load the module.	8	`modprobe tcp_rt stats_buf_num=10`

Uninstall the module.
1. Run the following command to disable the tcp-rt module and ensure that no new connections use TCP-RT:
```
sudo echo 1 > /sys/kernel/debug/tcp-rt/deactivate
```
2. Run the following command to check whether no connections are using the tcp-rt module:
```
lsmod
```
  If the value of Used by in the tcp-rt module is 0 in the command output, no connections are using the tcp-rt module.
3. When no connections are using the tcp-rt module, run the following command to uninstall the tcp-rt module:
```
sudo rmmod tcp_rt
```

FAQ

What do I do if log file sizes are different from the relevant parameter value?

File sizes that are displayed in command outputs, such as the ls -l command output, may be greater than the file size-related parameter values. This is not an issue. Log files are stored in the /sys virtual file system. The size of a log file in the ls -l command output is not the actual volume of logs in the file or the actual amount of disk storage space occupied by the logs. The log file sizes in the ls -l command output are the cumulative volumes of logs, which keep increasing. To view the actual volume of logs in a file, you can redirect the logs to a different file.

The following figure shows an example on how to redirect logs to a different file and view the actual volume of the logs. In this example, the default value of the log file size-related parameter is 2 MB. After you run the cat command to redirect logs in a file to a different file, you can find that the actual volume of the logs is approximately 2 MB as expected.