eBPF-based Network Observability on ACK: Zero-instrumentation Visibility into Kubernetes Networking

This article introduces KubeSkoop, an eBPF-based ACK tool offering zero-instrumentation, pod-level Kubernetes network visibility for diagnosing DNS timeouts and TCP resets.

Kubernetes networking failures are some of the hardest problems to diagnose in production. A DNS timeout, an intermittent TCP reset, or a 502 from an ingress controller rarely leaves a clean trail, and by the time someone starts investigating, the packet that caused it is long gone. Traditional troubleshooting means SSHing into nodes, running tcpdump, and manually correlating timestamps across pods that may no longer exist. eBPF changes this by observing the kernel's network stack directly, without modifying application code or injecting sidecars. This article looks at how eBPF-based tooling, specifically KubeSkoop, provides pod-level network observability on Container Service for Kubernetes (ACK), and walks through deploying it and using it to diagnose a real failure.

Why zero-instrumentation matters for network visibility

Most observability approaches require some form of instrumentation: an SDK in the application, a sidecar proxy, or a service mesh control plane. That works well for application-level metrics like request latency or error rate, but Kubernetes networking problems often originate below the application layer entirely, in the kernel's socket buffers, netfilter rules, or the CNI plugin's packet handling. A dropped packet at the veth pair or a socket stuck in a retransmit loop is invisible to application instrumentation because the application never sees it as anything other than a slow response.

eBPF (extended Berkeley Packet Filter) solves this by running small, verified programs directly inside the Linux kernel, attached to specific hook points in the network stack: socket operations, netfilter chains, network device drivers, and more. Because these programs observe kernel state directly, they see every pod's network activity without requiring any change to the pod itself. This is what "zero-instrumentation" means in practice: the workload never sees any injected code, and no developer has to add tracing libraries to their application to get network-level visibility.

KubeSkoop: eBPF observability purpose-built for Kubernetes

KubeSkoop is an open source network diagnosis suite that applies eBPF to the specific structure of a Kubernetes cluster. Rather than exposing raw kernel events, it maps them back to Kubernetes objects, so a kernel-level TCP retransmit event gets attributed to the specific pod, namespace, and node it occurred on. KubeSkoop runs as a DaemonSet, placing one agent per node, and each agent uses eBPF probes to collect metrics across the full network path: socket layer, bridge, veth pairs, netfilter, and sysctl-tunable kernel parameters.

The suite covers three main capabilities:

● Continuous monitoring: each node agent exports Prometheus metrics and anomaly events covering DNS resolution failures, TCP resets, retransmissions, and connection tracking table exhaustion.

● On-demand diagnosis: given a source and destination address, KubeSkoop can trace the exact path a packet takes through the cluster's network stack and identify where it was dropped or delayed.

● Packet capture and latency detection: targeted packet capture and per-hop latency measurement between specific pods or nodes, without needing to attach a debugger to any container.

On ACK specifically, this capability is available as a managed feature built on the open source KubeSkoop project, or as a self-managed deployment for clusters that want more control over configuration.

Deploying KubeSkoop on an ACK cluster

The fastest path on ACK is the managed feature: enable it from the console's Diagnostics section under Operations > Prometheus Monitoring, which installs the KubeSkoop agent DaemonSet on every node and wires it into Alibaba Cloud Prometheus Service automatically, without needing to manage the Prometheus, Grafana, and Loki stack separately.

For clusters that need a self-managed deployment instead, confirm the agents are running on every node once installed:

kubectl get pod -n kubeskoop -o wide

In self-managed environments, the underlying infrastructure can vary, but the same need for node-level network visibility applies. For workloads hosted on a Python VPS, for example, understanding the underlying network behavior can be important when troubleshooting connectivity and performance issues.

Each agent exposes Prometheus-format metrics on port 9102. To pull metrics directly from a single agent for inspection, without going through the full Prometheus stack:

AGENT_IP=$(kubectl get pod -n kubeskoop -l app=kubeskoop-agent \
  -o jsonpath='{.items[0].status.podIP}')

curl http://$AGENT_IP:9102/metrics | grep kubeskoop_

This returns per-pod counters covering categories like TCP retransmissions, connection resets, DNS query latency, and socket buffer errors, labeled with pod name, namespace, and node.

Diagnosing a real failure: intermittent DNS timeouts

DNS timeouts inside Kubernetes clusters are a common and frustrating class of bug, because they are usually intermittent and hard to reproduce on demand. A typical symptom looks like an application logging occasional ETIMEDOUT errors on service-to-service calls, with no clear pattern.

With KubeSkoop's metrics already flowing into Prometheus, the first step is checking whether DNS queries are timing out at the kernel level rather than assuming it is an application bug:

promql
sum(rate(kubeskoop_dns_timeout_total[5m])) by (pod, namespace)

If this query shows a nonzero rate for a specific pod or namespace, the next step is correlating it against connection tracking table pressure, since a full conntrack table is one of the most common root causes of dropped DNS packets in high-churn clusters:

promql
kubeskoop_conntrack_table_usage_ratio > 0.8

A usage ratio consistently above 0.8 on the node hosting the affected pods is a strong signal that the conntrack table is the bottleneck, not the application or the DNS server itself. From here, KubeSkoop's on-demand diagnosis can confirm the exact drop point by tracing a live connection between the affected pod and the cluster DNS service, specifying source, destination, and port through the diagnosis interface, which walks the packet through every layer of the stack, from the pod's network namespace through the bridge or ENI, to netfilter, and out to the destination, flagging the first point of failure. This type of detailed network context can also be valuable for AI agents in software engineering, which can correlate infrastructure signals such as DNS failures, connection tracking pressure, and packet drops with application telemetry to help identify likely root causes faster.

ACK's network diagnostics console feature exposes this same connectivity trace without requiring kernel or CNI expertise from the operator running it.

A minimal example: what an eBPF probe actually observes

To make the mechanism concrete, here is a simplified example of the kind of kernel-level tracing KubeSkoop's agents perform under the hood, using bpftrace, a higher-level frontend for writing eBPF programs. This snippet traces TCP retransmission events system-wide and prints the source and destination:

// trace_retransmits.bt
kprobe:tcp_retransmit_skb
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)$sk;

    printf("TCP retransmit: pid=%d comm=%s saddr=%s daddr=%s\n",
        pid, comm,
        ntop($inet->inet_saddr),
        ntop($inet->inet_daddr));
}

Running this with bpftrace trace_retransmits.bt attaches a probe to the kernel's tcp_retransmit_skb function and prints an event every time the kernel retransmits a TCP segment, with no changes to any application and no packet capture overhead beyond the probe itself. KubeSkoop's agents do the same thing at a larger scale across dozens of hook points, then aggregate and attribute the results back to Kubernetes pod identities rather than raw kernel addresses, which is the layer of translation that makes the data usable for cluster operators rather than kernel engineers.

eBPF at the CNI layer

Network observability is one half of the picture; the other is the CNI plugin itself, since that is what determines how packets actually move between pods and nodes. Terway, Alibaba Cloud's CNI plugin for ACK, uses eBPF for traffic acceleration at the data path level (its DataPath V2 mode), reducing the latency overhead that traditional iptables-based packet forwarding introduces at scale. This matters for observability too, since a CNI that already exposes eBPF hook points makes it easier for tools like KubeSkoop to attach probes without conflicting with the plugin's own packet handling.

Closing thoughts

eBPF-based observability closes a gap that application-level monitoring cannot reach: the kernel's own view of what happened to a packet, independent of what the application believed happened. For Kubernetes clusters where network failures are the hardest category of incident to reproduce and diagnose, tools like KubeSkoop turn that kernel-level data into something attributable to a specific pod, namespace, and node, without requiring any instrumentation inside the workload itself. Combined with a CNI plugin that already operates at the eBPF layer, this gives operators a complete, low-overhead path from a vague symptom like "intermittent timeouts" down to the exact kernel event that caused it.

Disclaimer: The views expressed herein are for reference only and don't necessarily represent the official views of Alibaba Cloud.

Community

eBPF-based Network Observability on ACK: Zero-instrumentation Visibility into Kubernetes Networking

Why zero-instrumentation matters for network visibility

KubeSkoop: eBPF observability purpose-built for Kubernetes

Deploying KubeSkoop on an ACK cluster

Diagnosing a real failure: intermittent DNS timeouts

A minimal example: what an eBPF probe actually observes

eBPF at the CNI layer

Closing thoughts

Read previous post:

Ila Bandhiya

You may also like

Comments

Ila Bandhiya

Related Products

Container Service for Kubernetes

ACK One

Managed Service for Prometheus

Cloud-Native Applications Management Solution