bpftrace: one-liners that replace strace in production

7 September 2026 In Posts 844 words 4 min

Lead DevOps

strace halts a process on every syscall. On a live server at 2000 RPS, that means timeouts and alerts. bpftrace runs through eBPF in the kernel — tracing happens in parallel, without stopping anything. The overhead difference is orders of magnitude.

Installation

# Debian / Ubuntu
sudo apt install bpftrace

# RHEL / CentOS / Fedora
sudo dnf install bpftrace

# Arch
sudo pacman -S bpftrace

# Verify
sudo bpftrace -V

Full probe coverage requires debug symbols:

# Debian
sudo apt install linux-image-$(uname -r)-dbg
sudo apt install systemtap-sdt-dev

Check available probes:

sudo bpftrace -l | grep sched_process
# sched:sched_process_exec
# sched:sched_process_fork
# sched:sched_process_exit

bpftrace Syntax in 60 Seconds

One-liner format:

sudo bpftrace -e 'probe { action }'

Structure: what (probe) and what to do (action). Probe types:

Type	Example	Description
kprobe	`kprobe:do_sys_openat2`	kernel function entry
kretprobe	`kretprobe:do_sys_openat2`	kernel function return
tracepoint	`syscalls:sys_enter_openat`	stable kernel tracepoint
usdt	`usdt:/bin/python3:probe`	user-level static trace
profile	`profile:hz:99`	timer-based sampling

Built-in variables available in actions:

pid          # process ID
tid          # thread ID
comm         # process name
nsecs        # nanoseconds timestamp
curtask      # current task_struct
args         # probe arguments (if available)

Example — all execve calls:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { join(args->argv); }'

exec: Who Is Launching Processes

Want to understand which process triggers fork/exec across the system:

sudo bpftrace -e '
    tracepoint:syscalls:sys_enter_execve {
        time("%H:%M:%S ");
        printf("%s (PID %d) exec: %s\n", comm, pid, args->argv[0]);
    }
'

Output during 10 seconds of monitoring:

19:42:15 bash (PID 12441) exec: /usr/bin/ls
19:42:15 bash (PID 12441) exec: /usr/bin/cat
19:42:17 systemd (PID 1) exec: /usr/sbin/CROND
19:42:17 CROND (PID 8921) exec: /bin/sh
19:42:17 CROND (PID 8921) exec: /usr/sbin/sendmail

Track a specific process and its children:

sudo bpftrace -e '
    tracepoint:syscalls:sys_enter_execve /pid == 1234/ {
        printf("child exec: %s\n", args->argv[0]);
    }
'

The filter /pid == 1234/ uses standard syntax. Without it, bpftrace catches everything.

open: Which Files a Process Opens

sudo bpftrace -e '
    tracepoint:syscalls:sys_enter_open,
    tracepoint:syscalls:sys_enter_openat {
        printf("%s (PID %d) -> %s\n", comm, pid, str(args->filename));
    }
'

Filter by process name:

sudo bpftrace -e '
    tracepoint:syscalls:sys_enter_openat /comm == "nginx"/ {
        @[str(args->filename)] = count();
    }
'

This is aggregation — counting how many times each file was opened. @ is the built-in variable for maps. Output after Ctrl+C shows a sorted table.

Monitor open failures (ENOENT, EACCES):

sudo bpftrace -e '
    tracepoint:syscalls:sys_exit_openat {
        if (args->ret < 0) {
            printf("%s error %d on %s\n", comm, args->ret, str(args->filename));
        }
    }
'

Network: Connections and Dropped Packets

Monitor outbound connections:

sudo bpftrace -e '
    tracepoint:syscalls:sys_enter_connect {
        printf("%s (PID %d) connect to port %d\n", comm, pid, args->uservaddr->sin_port >> 8);
    }
'

Dropped iptables packets:

sudo bpftrace -e '
    kprobe:nf_hook_slow {
        @drops[comm] = count();
    }
'

Note

Not all kprobes are available on every kernel. Verify with sudo bpftrace -l | grep nf_hook.

Aggregation by port — a common task:

sudo bpftrace -e '
    tracepoint:syscalls:sys_enter_connect {
        @port = count();
    }
' 2>/dev/null | sort -rn | head -20

Errors: getpid Does Not Exist

Familiar functions may be absent in bpftrace. This is not bash — different rules apply.

Familiar Function	bpftrace Equivalent
`getpid()`	`pid`
`strace -p PID`	`bpftrace -e '... /pid == N/ {...}'`
`readlink /proc/PID/fd/N`	`nsecs`, `curtask`

Calling getpid() inside bpftrace produces a compilation error — the BPF program has no access to libc.

Warning

bpftrace cannot trace a process already running under active strace. They conflict at the ptrace level.

Error output — use strerror():

sudo bpftrace -e '
    tracepoint:syscalls:sys_exit_openat {
        if (args->ret < 0) {
            printf("%s: %s\n", str(args->filename), strerror(-args->ret));
        }
    }
'

bpftrace vs strace: Overhead Comparison

strace uses ptrace(PTRACE_SYSCALL). On every syscall, the kernel stops the process, copies data to userspace, then resumes. This is synchronous.

bpftrace compiles a BPF program and loads it into the kernel. Tracing happens in kernel context without stopping the process. Data accumulates in a ring buffer and is read asynchronously.

Comparison on nginx, 5000 RPS:

Method	p99 Latency	CPU Overhead	Observability
No tracing	12ms	—	—
strace -p PID	340ms	18%	syscalls
bpftrace one-liner	14ms	0.3%	syscalls + aggregation

Tip

For a quick check: strace -c -p PID gives a summary table of syscalls. bpftrace does the same via count() and hist().

When bpftrace Is Enough vs When You Need strace

bpftrace — for a system-wide view. Monitoring all processes, aggregation, heat maps, catching anomalies without affecting production.

strace — for deep-diving a specific request. Detailed log of every syscall with arguments and returns to reproduce a problem.

# bpftrace: aggregation — who opens the most files
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }'

# strace: detailed log of a single request
strace -f -e openat -s 200 curl localhost/api/endpoint

Three rules:

Don’t know the process — use bpftrace.
Know the PID and need detailed logs — use strace -p PID.
On production under load — bpftrace only.

One-liners as aliases:

echo 'alias bt="sudo bpftrace"' >> ~/.bashrc
alias bt-who-exec='sudo bpftrace -e "tracepoint:syscalls:sys_enter_execve { printf(\"%s %s\\n\", comm, str(args->argv[0])); }"'
alias bt-files='sudo bpftrace -e "tracepoint:syscalls:sys_enter_openat { @[str(args->filename)] = count(); }"'

bpftrace goes further — kernel memory, CPU profiling, allocator debugging. For a basic start, these four one-liners cover most of what you need.