Using eBPF (bcc/bpftrace) for Deep Linux Performance Analysis

Why eBPF Feels Like Having X-Ray Vision For Your Linux Box

Ever watched a server grind to a crawl and thought, *what on Earth is it doing in there?*
Old tools like top or strace give you the **what**, but rarely the **why**.
I hit this wall last year when a customer’s database started stalling every few minutes.
perf said *kernel time – 78 %*. Nice, but where inside the kernel?
A friend nudged me toward eBPF. Two hours later I was staring at the exact line of kernel code that held a spin-lock too long. **Problem fixed before dinner.**

eBPF is basically a tiny, super-fast virtual machine that lives inside the Linux kernel.
It lets us drop little probes in there **while the machine is running**.
Two easy ways to talk to it are:

• BCC – big toolbox written in Python/C: loads of ready-made commands.
• bpftrace – mini scripting language for “explain this weird blip **now**”.

Traditional vs. eBPF – A 60-second Comparison

Classic profiler
Collect stack traces → dump 5 MB/s to disk → crunch for ten minutes → maybe find the bottleneck.

eBPF one-liner
Count how many times every process hits a slow path function, **live in RAM**, zero disk IO. Ctrl-C to print a table, done.

That order-of-magnitude reduction in effort? It changes how you think about debugging.

My 15-minute Start-Up Routine Anytime Something Feels Sluggish

1. Run sudo biolatency-bpfcc 1
Shows a histogram of disk latency every second. Quick eyeball test for “disk is trashing”.
2. If disk is clean, try:
   sudo execsnoop-bpfcc
   Tells me which new commands just spawn. Often it’s a rogue cron job or healthcheck script.
3. Still no clue?
   sudo bpftrace -e 'profile:hz:99 { @[kstack] = count(); }'
   Samples every running CPU 99 times a second and prints the hottest kernel stacks.
   I see a wall of *spinlock*, notice it’s the same file system code line every time – SMR disk firmware bug.

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat /container_id != 0/
                 { @[comm, container_id] = count(); }'

sudo bpftrace -e 'kprobe:tcp_retransmit_skb {
  @retransmits[comm] = count();
  @total = count();
}'