Google uses eBPF for faster production diagnostics

Google has outlined how it uses open-source eBPF for real-time lock contention analysis and fleet-wide diagnostics, cutting the time needed to deploy diagnostic changes across its infrastructure.

Changes to observability and diagnostic tooling that once took months to roll out can now be deployed across the fleet in hours or days through independent eBPF packages. Site reliability engineering teams can also identify the cause of transient performance problems in hours or days rather than weeks.

eBPF, short for extended Berkeley Packet Filter, allows software to run inside the Linux kernel for monitoring, tracing and other tasks. Google described it as a way to add targeted diagnostic logic directly to kernel subsystems without the disruption of custom debug kernels or repeated attempts to reproduce faults in test environments.

The challenge involved localised anomalies that are hard to catch in large production systems, including sporadic I/O latency spikes and misconfigured control groups, or cgroups. According to the case study, traditional profiling and tracing tools can add enough CPU and memory overhead to make them unattractive in active, latency-sensitive clusters.

Operational shift

Google's framework uses lightweight eBPF tracing programs to observe lock contention in real time. These programs attach to lock-related tracepoints and kernel functions to capture stack traces and monitor mutexes and semaphores, while filtering telemetry within the kernel so only events above defined wait-time thresholds are surfaced.

This design is intended to limit the amount of data exported from running systems and reduce performance impact. The same framework also supports broader infrastructure health monitoring through tracing tools that export metrics using eBPF iterators pinned to the eBPF filesystem.

For more targeted investigations, engineers can deploy intermittent tracing programs to specific production machines through an internal diagnostics platform linked to Google's cluster management agent. The setup gives operators direct visibility into behaviours such as block I/O latency and scheduler delays at the point an anomaly appears.

Engineering hurdles

Running this tooling across a large, varied fleet created several technical constraints. Google said detailed tracing programs often ran into kernel verifier limits on stack usage and instruction count as they grew more complex.

To address that, engineers reworked code layouts, controlled which functions were inlined, bounded loops and moved large stack variables into eBPF maps. They also had to deal with differences between kernel versions across the infrastructure, where certain tracepoints or internal functions are not available everywhere.

The tooling was therefore designed to tolerate those differences and degrade gracefully when required structures were absent. That is significant for operators with mixed estates, because compatibility problems can slow or block the use of low-level observability tools at scale.

Security was another concern. Rather than grant developers broad CAP_SYS_BPF permissions in production environments, Google built a centralised eBPF management daemon that validates and authorises tracing scripts through an approval workflow.

Production impact

The system has already been used to identify and resolve multi-core kernel lock bottlenecks in live production workloads. Engineers have also investigated hardware-specific I/O latency issues and cgroup configuration problems without switching kernels or waiting weeks for a suitable reproduction.

That marks a change in how diagnostics are deployed. In the older model Google described, root-causing an active issue often depended on compiling and distributing custom debug kernels, a process that could take weeks and still fail to reproduce a short-lived anomaly under the same conditions.

By contrast, eBPF-based tracing can be attached when an issue appears and removed when it is no longer needed. Google said the in-kernel filtering model, combined with the safety checks enforced by the eBPF verifier, allows deep inspection with negligible system overhead.

The case also illustrates the growing use of eBPF beyond networking and security into day-to-day production diagnostics for large infrastructure operators. For companies managing distributed systems at scale, the appeal lies in using one mechanism both for continuous telemetry and for short-notice investigations into faults that standard metrics do not explain.

Google said the verifier provides assurance that deployed tracing tools cannot crash the system, hang the CPU or corrupt kernel memory.