Container Service for Kubernetes:Use ACK Net Exporter to troubleshoot network issues

Install and configure ACK Net Exporter

Install ACK Net Exporter

1. Log on to the ACK console. In the left-side navigation pane, choose Marketplace > Marketplace.

2. On the Marketplace page, search for ack-net-exporter and click the component.

3. In the upper-right corner of the ack-net-exporter component page, click Deploy in the upper-right corner. The Deploy panel appears.

4. In the Basic Information step, configure the Cluster and Namespace parameters and click Next.

5. In the Parameters step, configure the parameters that are described in the following table and click OK.

   Parameter	Description	Default value
   name	The name of the ACK Net Exporter component.	ack-net-exporter-default
   namespace	The namespace to which ACK Net Exporter belongs.	kube-system
   config.enableEventServer	Specifies whether to enable the event tracing feature. Valid values: false: disables the event tracing feature. true: enables the event tracing feature.	false
   config.enableMetricServer	Specifies whether to enable the metric collection feature. Valid values: false: disables the metric collection feature. true: enables the metric collection feature.	true
   config.remoteLokiAddress	The Grafana Loki service address to which events are pushed.	By default, this parameter is left empty.
   config.metricLabelVerbose	Specifies whether to enable the metric verbose feature. Valid values: false: disables the metric verbose feature. true: enables the metric verbose feature. After you enable this feature, pod IP addresses and labels are saved as the label information of metrics.	false
   config.metricServerPort	The port that is used by the metric service to provide HTTP services.	9102
   config.eventServerPort	The port that is used by the event service to provide gRPC streaming services.	19102
   config.metricProbes	The metric probes that you want to enable. For more information, see the ACK Net Exporter metrics section of this topic.	By default, this parameter is left empty and only the required metric probes are enabled.
   config.eventProbes	The event probes that you want to enable. For more information, see the ACK Net Exporter events section of this topic.	By default, this parameter is left empty and only the required event probes are enabled.

Configure ACK Net Exporter

• You can run the following command to modify the ConfigMap of ACK Net Exporter:

  kubectl edit cm inspector-config -n kube-system

• Alternatively, you can configure ACK Net Exporter in the ACK console.

  1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

  2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, choose Configurations > ConfigMaps.

  3. On the ConfigMap page, set the Namespace parameter to kube-system, search for kubeskoop-config, and then click Edit in the Actions column.

  4. In the Edit panel, configure the parameters and click OK.

     The following table describes the parameters supported by ACK Net Exporter.

     Parameter	Description	Default value
     debugmode	Specifies whether to enable the debugging mode. Valid values: false: disables the debugging mode. true: enables the debugging mode. After you enable this mode, debug-level logs, interface debugging, Go pprof, and gops are supported.	false
     event_config.loki_enable	Specifies whether to enable the feature of pushing events to Grafana Loki. For more information, see the Use Grafana Loki to collect and visualize events section of this topic. Valid values: false: disables the feature of pushing events to Grafana Loki. true: enables the feature of pushing events to Grafana Loki.	false
     event_config.loki_address	The Grafana Loki service address. By default, the system automatically discovers a service named grafana-loki in the specified namespace.	By default, this parameter is left empty.
     event_config.probes	The event probes that you want to enable. For more information, see the ACK Net Exporter events section of this topic.	By default, this parameter is left empty and only the required event probes are enabled.
     event_config.port	The port that is used by the event service to provide gRPC streaming services.	19102
     metric_config.verbose	Specifies whether to enable the metric verbose feature. Valid values: false: disables the metric verbose feature. true: enables the metric verbose feature. After you enable this feature, pod IP addresses and labels are saved as the label information of metrics.	false
     metric_config.port	The port that is used by the metric service to provide HTTP services.	9102
     metric_config.probes	The metric probes that you want to enable. For more information, see the ACK Net Exporter metrics section of this topic.	By default, this parameter is left empty and only the required metric probes are enabled.
     metric_config.interval	The interval at which metrics are collected. Metric collection compromises performance. Therefore, ACK Net Exporter caches the periodically collected metrics in memory.	5

In earlier ACK Net Exporter versions, you need to trigger the system to recreate all ACK Net Exporter containers after you modify the configurations of ACK Net Exporter. The modified configurations take effect after the containers are recreated. You no longer need to perform this operation in ACK Net Exporter 0.2.3 and later versions because these versions support hot updates.

Usage notes for ACK Net Exporter

Use ACK Net Exporter in operating systems other than Alinux

Some key features of ACK Net Exporter rely on eBPF programs to collect information. To meet the requirements of different operating system kernels, ACK Net Exporter uses CO-RE to distribute eBPF programs. When ACK Net Exporter starts, it needs to load the BPF Type Format (BTF) file that is associated with the operating system kernel. The BTF file stores the metadata of the kernel debugging information. If no corresponding BTF file is loaded, the key features become unavailable. Most operating systems of later versions have built-in BTF files. For more information about the operating systems, see BPF Type Format (BTF).

Metrics and metric format supported by ACK Net Exporter

inspector_pod_udprcvbuferrors{namespace="elastic-system",netns="ns402653****",node="iZbp179u0bgzhofjupc****",pod="elastic-operator-0"} 0 1654487977826

Events and event format supported by ACK Net Exporter

ACK Net Exporter can collect events of network exceptions that occur on nodes. This section describes the network exceptions that may occur. These exceptions occasionally occur and are difficult to reproduce. No efficient methods are available to troubleshoot these exceptions.

• Connection failures and request timeouts caused by data packet loss.

• Performance issues caused by time-consuming data processing.

• Business interruptions that occur due to the anomalies of the stateful connection mechanism, such as TCP or connection tracking.

ACK Net Exporter provides eBPF-based context observability for operating system kernels to help you troubleshoot the preceding issues. ACK Net Exporter captures the status of the operating system in real time when an exception occurs, and then generates an event log. For more information about the events and event probes supported by ACK Net Exporter, see the ACK Net Exporter events section of this topic.

type=TCPRESET_NOSOCK pod=storage-monitor-5775dfdc77-fj767 namespace=kube-system protocol=TCP saddr=100.103.42.233 sport=443 daddr=10.1.17.188 dport=33488

For events that require valid operating system kernel stacking information, ACK Net Exporter captures the stacking context in the operating system kernel when these events occur, such as the following event:

type=PACKETLOSS pod=hostNetwork namespace=hostNetwork protocol=TCP saddr=10.1.17.172 sport=6443 daddr=10.1.17.176 dport=43018  stacktrace:skb_release_data+0xA3 __kfree_skb+0xE tcp_recvmsg+0x61D inet_recvmsg+0x58 sock_read_iter+0x92 new_sync_read+0xE8 vfs_read+0x89 ksys_read+0x5A

ACK Net Exporter allows you to view events by using multiple methods. For more information, see the "Collect monitoring data from ACK Net Exporter" section of this topic.

Collect monitoring data from ACK Net Exporter

Scenario 1: Export monitoring data to Prometheus or Grafana and visualize the data

# In the following example, only one endpoint is specified for data collection. 
scrape_configs:
# The job=<job_name> label is added to each time series that is collected based on the configuration. In this example, the job is named net-exporter_sample. 
- job_name: "net-exporter_sample"
static_configs:
  - targets: ["{kubernetes pod ip}:9102"]
                

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
    name: prometheus-server-conf
  namespace: kube-system
data:
  prometheus.yml: |-
          # Add the following configurations to the Prometheus server: 
      - job_name: 'net-exporter'
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_endpoints_name]
          regex: 'net-exporter'
          action: keep

      - job_name: 'kubernetes-pods'

        kubernetes_sd_configs:
        - role: pod

        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name

After you add the configurations, choose Status > Targets in the Prometheus server. The Targets page of the Prometheus server shows the ACK Net Exporter pods that run as expected. You can also enter inspector into the search box on the Graph page of the Prometheus server to view the ACK Net Exporter metrics.

You can configure Grafana to visualize the monitoring data that is collected to Prometheus:

1. In the left-side navigation pane of the Grafana page, choose > Dashboards.

2. On the New dashboard page, click Add a new panel.

3. In the lower part of the Edit Panel page, enter Prometheus in the Data source field. Then, enter the address of the Prometheus server.

4. Click Metric browser and enter inspector. Then, Grafana displays all available ACK Net Exporter metrics. Click Save in the upper-right part. In the dialog box that appears, click Save. Grafana then displays the visualized data, as shown in the following figure.

   

5. Configure how the metrics are displayed on a Grafana dashboard based on the configurations that are displayed in the preceding figure. For example, you can use the following configurations to display the increment trend of the inspector_pod_tcppassiveopens metric. This metric indicates the total number of sockets that are created due to handshake requests sent by clients to establish TCP connections within a network space after the system is started or the container is created. To view the increment trend of this metric, use the following configurations:

   // Use the rate() method provided by PromQL to calculate the increment trend of the metric. 
   rate(inspector_pod_tcppassiveopens[1m])
   
   // Use the labels provided by net-exporter to configure a legend to display the metric. 
   {{namespace}}/{{pod}}/{{node}}

Scenario 2: Export monitoring data to ARMS and visualize the data

To export monitoring data from ACK Net Exporter to Application Real-Time Monitoring Service (ARMS) and visualize the data, perform the following steps.

1. Step 1: Enable Managed Service for Prometheus.

2. Configure custom ACK Net Exporter metrics.

   1. Log on to the ARMS console. In the left-side navigation pane, choose Managed Service for Prometheus > Instances.

   2. Log on to the ARMS console.

   3. In the top navigation bar, select the region in which the ACK cluster is deployed.

   4. In the left-side navigation pane, choose Managed Service for Prometheus > Instances.

   5. Click the name of the Prometheus instance instance that you want to manage to go to the Integration Center page.

   6. On the Instances page, find the instance that you want to manage and click the name of the instance. Then, you are redirected to the instance details page. In most cases, the name of the Prometheus instance is the same as the name of your cluster.

   7. In the left-side navigation pane, click Service Discovery, and then click the Targets tab. In the lower part of the Targets tab, click kubernetes-pods. The information shows that the custom ACK Net Exporter metrics are configured.

      If kubernetes-pods is not displayed, you need to click the Configure tab and turn on the switch on the Default Service Discovery tab.

   8. In the left-side navigation pane, click Dashboards. On the Dashboards page, click a dashboard to log on to Grafana. Click Add panel, select Graph, and then enable the data sources related to your cluster in the Data source section.

   9. Click Metric browser and enter inspector. Then, Grafana automatically displays all available ACK Net Exporter metrics. In the upper-right corner, click Save. In the dialog box that appears, click Save. Grafana then displays the visualized data, as shown in the following figure.

Scenario 3: Export monitoring data to Grafana Loki and visualize the data

You can push anomaly events collected by ACK Net Exporter to your pre-configured Grafana Loki service in real time. This helps you manage these events in a centralized manner. To export monitoring data from ACK Net Exporter to Grafana Loki, perform the following steps.

1. Set up Grafana Loki.

   Note

   Deploy Grafana Loki in a network that the ACK Net Exporter pods can access. ACK Net Exporter can automatically push event logs to Grafana Loki.

2. On the configuration page of ACK Net Exporter, set the enableEventServer parameter to true and the lokiServerAddress parameter to the address of the Grafana Loki service. You can specify the IP address or domain name of the Grafana Loki service as the service address.

3. Run the following command to access the service address and check whether Grafana Loki is available:

   curl http://[Address of Grafana Loki]:3100/ready

4. If Grafana Loki is available, add Grafana Loki as a Grafana data source.

   Open Grafana. In the left-side navigation pane, choose Data source > Loki, enter the address of Grafana Loki, and then click Save & test.

5. In the left-side navigation pane, click Explore. In the top navigation bar, set the data source to Loki and view the events pushed to Grafana Loki.

   You can view the events of a node by selecting the node from the Label filters drop-down list or enter keywords in the Line containers field to search for specific events.

   You can click Add to dashboard in the top navigation bar to add a configured event panel to the dashboard.

   The content of the events provided by ACK Net Exporter varies based on the event type. You can check the event details to view the relevant content.

   For more information about the LogQL query language supported by Grafana Loki, see LogQL: Log query language.

Scenario 4: Use the ACK Net Exporter CLI to collect events

The ACK Net Exporter CLI (inspector-cli) is a scenario-specific troubleshooting and analysis tool developed by the ACK team based on ACK Net Exporter. You can use inspector-cli to collect kernel exception events in real time. inspector-cli can help quickly identify the cause of common exceptions in cloud-native scenarios.

# Start a temporary container to run inspector-cli. You can replace the image with a later version to update inspector-cli. 
docker run -it --name=inspector-cli --network=host registry.cn-hangzhou.aliyuncs.com/acs/inspector:v0.0.1-12-gff0558c-aliyun

which inspector
# /bin/inspector is the working path of inspector-cli. You can directly run inspector-cli in the container.

# Set the -e parameter to the address of the event service of ACK Net Exporter. 
inspector watch -e 10.1.16.255

# Expected results: 
 INFO  TCP_RCV_RST_ESTAB Namespace=kube-system Pod=kube-proxy-worker-tbv5s Node=iZbp1jesgumdx66l8ym8j8Z Netns=4026531993 10.1.16.255:43186 -> 100.100.27.15:3128
...

You can also log on to the inspector container of ACK Net Exporter to troubleshoot issues.

# When you run the following command, set the -n parameter to the namespace of ACK Net Exporter and specify the ACK Net Exporter pod that you want to access. 
kubectl exec -it -n kube-system -c inspector net-exporter-2rvfh -- sh

# Run the following command to view the distribution of network entities on the current node. 
inspector list entity

# Run the following command to listen for network exception events and other relevant information in the local network. 
inspector watch -d -v
#{"time":"2023-02-03T09:01:03.402118044Z","level":"INFO","source":"/go/src/net-exporter/cmd/watch.go:63","msg":"TCPRESET_PROCESS","meta":"hostNetwork/hostNetwork node=izbp1dnsn1bwv9oyu2gaupz netns=ns0 ","event":"protocol=TCP saddr=10.1.17.113 sport=6443 daddr=10.1.17.113 dport=44226  state:TCP_OTHER "}

# You can also specify multiple ACK Net Exporter nodes to view the time when the event occurs on these nodes. 
inspector watch -s 10.1.17.113 -s 10.1.18.14 -d -v
            

Use ACK Net Exporter to troubleshoot occasional container network issues

This section describes how to troubleshoot occasional network issues in cloud-native scenarios. ACK Net Exporter can help you quickly obtain information that is required for fixing these issues.

DNS timeout issues

A large number of services in cloud-native environments rely on the DNS resolution service provided by CoreDNS. If a DNS issue correlated to CoreDNS occurs, you need to check the metrics of the CoreDNS pod.

Nginx Ingress 499, 502, 503, and 504 issues

You can check the changes of the preceding metrics at the point in time when the issue occurs to narrow down the scope. If you still cannot locate the cause, submit a ticket and include the preceding information in your ticket to request technical support.

TCP reset issues

Occasional network latency and jitter issues

Case study

The following cases show how to use ACK Net Exporter to troubleshoot container network issues.

Case 1: Occasional DNS resolution timeout

Symptom

Customer A submitted a ticket to request technical support to handle DNS resolution timeouts that occasionally occur. The application of Customer A is written in PHP. CoreDNS is configured to perform DNS resolution.

Troubleshooting

1. Obtain DNS metrics from the monitoring system of Customer A.

2. Analyze the metrics at the moments of reported timeouts. The analysis reveals the following issues:

   • Each time a DNS resolution timeout occurs, the value of inspector_pod_udpnoports increases by 1. The value of this metric is small.

   • The number of __udp4_lib_rcv packet drops indicated by the inspector_pod_packetloss metric increases by 1. However, the change in the number of packet drops is minor.

3. Customer A specifies that the IP address of the DNS server is a public IP address provided by an Internet service provider (ISP). Based on the obtained metrics, the DNS timeouts occurred because the amount of time required to send the response to the client is large. The response is received after the DNS query times out in user mode.

Case 2: Occasional Java application connection failure

Symptom

Customer B submitted a ticket to request technical support to resolve the following issue: Tomcat occasionally becomes unavailable and the issue lasts 5 to 10 seconds each time.

Troubleshooting

1. The log analysis result shows that the Java runtime was performing a garbage collection (GC) operation when the issue occurred.

2. Customer B deployed ACK Net Exporter and analyzed the monitoring data. Customer B found that the value of the inspector_pod_tcpextlistendrops metric increased significantly at the time when the issue occurred.

3. The analysis results indicate that request processing was slowed down when the Java runtime performed the GC operation. However, new requests are not throttled. As a result, a large number of connections are created and the backlog of the LISTEN socket is overflowed. This causes the value of the inspector_pod_tcpextlistendrops metric to increase.

4. The issue of TCP connections piling up is transient and is not attributed to the request processing capability of TCP. In this scenario, Customer B modified the Tomcat parameters as we recommended and resolved the issue.

Case 3: Occasional network jitter

Symptom

Customer C submitted a ticket to request technical support to resolve the following issue: The round-trip time between the Redis instance and application significantly increases. As a result, timeout errors occur. The issue cannot be reproduced.

Troubleshooting

1. After Customer C analyzed the log, Customer C identified that the response time of Redis requests occasionally exceeds 300 milliseconds.

2. Customer C also identified that the value of the inspector_node_virtsendcmdlat metric increased at the time when the issue occurred. The affected monitoring levels in Prometheus service are 15 and 18. After calculation, Customer C identified two virtual calls with a long response time. The response time of the call with a monitoring level of 15 exceeds 36 milliseconds and the response time of the call with a monitoring level of 18 exceeds 200 milliseconds.

3. The kernel occupies the CPU when processing virtual calls. In this case, CPU resources cannot be preempted by other operations. As a result, the execution of virtual calls is slowed down when pods are added or deleted in batches, which further causes the response time to increase.

Case 4: NGINX Ingress occasionally fails to pass health checks

Symptom

Customer D submitted a ticket to request technical support to resolve the following issue: The NGINX Ingress occasionally fails to pass health checks. As a result, request failures occur.

Troubleshooting

1. After Customer D deployed ACK Net Exporter, Customer D identified that the following metrics are abnormal:

   1. The values of the inspector_pod_tcpsummarytcprxqueue and inspector_pod_tcpsummarytcptxqueue metrics increased.

   2. The value of the inspector_pod_tcpexttcptimeouts metric increased.

   3. The value of the inspector_pod_tcpsummarytcptimewaitconn metric decreased and the value of the inspector_pod_tcpsummarytcpestablishedconn metric increased.

2. The analysis results indicate that the kernel ran as expected when the issue occurred. Connections are created as expected. However, exceptions occurred when the user process handled the packets in the receive socket and sent packets. In this scenario, the health check failure may be caused by a scheduling or rate limiting issue.

3. Customer D checked the monitoring data of the cgroups as we recommended and identified CPU throttling at the point in time when the health check failure occurred. The results indicate that the user process occasionally failed to schedule CPU resources due to a cgroup issue.

4. To resolve this issue, refer to CPU Burst and configure CPU Burst for the NGINX Ingress.

Appendix

ACK Net Exporter metrics

The metrics supported by ACK Net Exporter are constantly updated. For more information, see the instructions on the Marketplace page of the ACK console. All metrics and events provide pod-specific information, except net_softirq and virtcmdlat, which are not related to pods.

ACK Net Exporter events

The following table describes the operating system network-related events that can be captured by using the latest ACK Net Exporter version.

Recommended Grafana configuration file

• If you use a Grafana version later than 8.4.0, click ACK Net Exporter-0.2.9.json to download the Grafana configuration file.

• If you use Grafana 8.4.0 or earlier, click ACK Net Exporter-legacy.json to download the Grafana configuration file.