BPF tracepoints peek inside the running kernel

Mon, 06 Mar 2028 00:00:00 +0000

BPF and kernel tracepoints let you observe behavior at the source with a tiny footprint: no agent, no restart, no stopping the workload. Traditional profiling means installing something, restarting with flags, or pausing the process — each of which perturbs the system under study or is forbidden in production. A BPF probe attaches to an already-running kernel, collects its answer, and detaches, leaving the workload exactly as it found it.

For the questions “what is slow” and “what is happening right now,” a minute with the right one-liner beats an hour of log archaeology.

Probe without pausing

A small eBPF program attaches to an existing tracepoint and accumulates counts or histograms while the work keeps running. Because it runs in kernel context, the probe rarely becomes the bottleneck. The program is verified before loading — the kernel checks memory safety and termination — so a buggy probe is rejected rather than crashing the machine. Aggregation happens in kernel maps, meaning millions of events collapse into a small histogram before anything crosses to userspace; the common case costs single-digit percent overhead, often less than the logging the team already runs.

Start from questions, not tools. “Which process opens the most files” maps to one tracing shape, “p99 disk latency by device” to another, and the bpftrace one-liner gallery covers both without writing C. Prefer high-level frontends — bpftrace, bcc tools — over raw BPF unless the measurement truly needs custom logic; the one-liner you can write during an incident beats the perfect program you finish next week.

Tracepoints are the stable names

Kernel tracepoints mark events such as block I/O, scheduling, syscall entry/exit that are less brittle than raw kprobes, so probes survive kernel updates. A kprobe pins a kernel function’s address, which shifts between versions; a tracepoint names a semantic event the kernel promises to keep, so the same probe works after Tuesday’s patch reboot. Attach to the ones closest to your behavior: block tracepoints for disk questions, scheduler tracepoints for latency-off-CPU questions, syscall tracepoints for “who calls what” questions.

List what exists before guessing names — tracepoints are enumerated under the kernel’s tracing interface, and every standard tool can print them. When no tracepoint covers the event, fall back to kprobes or uprobes deliberately and pin the probe to kernel versions in comments, accepting the maintenance cost with open eyes. Stability is a spectrum: tracepoints first, well-known kprobes second, raw addresses never in anything you intend to keep.

Read a histogram, not a firehose

Aggregate a noisy event stream into a latency or count histogram to expose the top offenders without dumping each event. Use that shape to find the culprit, then drill in. A per-event dump of a million syscalls answers nothing a human can read; a log-scale latency histogram shows the bimodal split — fast path versus the slow one percent — in a single screen. The histogram is the diagnosis; individual events are only the confirmation, sampled after the shape tells you where to look.

Keep the aggregation key tight: by process, by file, by syscall — one dimension at a time. Two-dimensional breakdowns explode cardinality and bury the signal, and full-argument dumps reintroduce the firehose through the back door. When the histogram fingers a suspect, switch to targeted tracing on that process or file alone, with a time bound, and stop once the confirming sample exists.

When it breaks

The classic failure is the probe that never attaches: wrong tracepoint name, missing kernel headers, or a locked-down environment blocking BPF loading — and the tool exits quietly or with a cryptic verifier message. Verify the environment first (tracepoints listed, loader permissions present) before doubting the probe logic; most BPF failures are environmental, not logical. In containers, remember the probe usually needs host-level privileges the container runtime does not grant by default.

The second failure is observer effect at scale: an unfiltered probe on a hot path — every syscall on a 100k-rps box — burns whole cores and skews the measurement it was taking. Always aggregate in-kernel, filter to the suspect early, and time-box production probes. If overhead looks suspicious, halve the sampling or narrow the filter and check the answer stays stable; a measurement that changes shape when observed more cheaply was never trustworthy.

Check it quickly

Confirm the tooling works and take a first histogram:

bpftrace -l "tracepoint:syscalls:*" | head -n 5
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @opens[comm] = count(); }' &
BPF_PID=$!; sleep 10; kill -INT $BPF_PID; wait

The first proves tracepoints are visible, and the ten-second sample ranks processes by file opens — the shape of every BPF investigation in miniature.

Sources

Kernel tracing docs on bpftrace, Official bpftrace reference, checked 2026-06-01.
eBPF introduction, Official eBPF documentation, checked 2026-06-01.