Detecting Memory Leaks in systemd Services with perf and heaptrack

Your server feels slow? It might be tiny leaks, not traffic spikes

Picture this. You’re sipping coffee on a Tuesday morning. Your Slack pings. “App is crawling.” You check the metrics—CPU looks fine, disk is quiet. But RAM? It keeps climbing. One restart later, it feels okay… until tomorrow.

That’s a memory leak. And if your app runs under systemd, the leak is hiding in plain sight.

Why leaks love systemd (and why you won’t see them coming)

Classic monitors look for sudden jumps. Leaks move like snails. They add a few kilobytes every minute. By the time your graphs look scary, the damage is already done.

What you’ll notice first:

• Page loads get chunky at 3 p.m. every day
• Systemd restarts the service “randomly”
• Your cloud bill creeps up because you keep upsizing the box

A startup I helped last year lost fifty grand before they spotted the loop that never freed a struct. One line patch. Two days of profiling. Painful lesson.

Step-by-step: Catch the leak before it bites

1. Watch with systemd-cgtop (takes 30 seconds)

systemd-cgtop -m

Look for any service whose memory column climbs and never drops. That’s your suspect.

2. Grab a quick perf trace (2 minutes)

sudo perf record -e brk,mmap -g -p $(systemctl show -p MainPID myapp.service | cut -d= -f2)

Let it run while you brew another coffee. Then:

perf report --sort comm,dso,symbol

Scroll to the top. If you see malloc or strdup with no matching free, you’ve found the smoking gun.

3. Zoom in with heaptrack (5–10 minutes)

Temporarily edit the unit file:

[Service]
ExecStart=/usr/bin/heaptrack /usr/local/bin/myapp

Reload and restart:

sudo systemctl daemon-reload && sudo systemctl restart myapp.service

Reproduce the leak, stop the service, then open the trace:

heaptrack_gui heaptrack.myapp.*.gz

The flame graph shows exactly which line keeps allocating but never frees. I once found a JSON parser holding 2 GB of tiny strings—one forgotten json_decref call.

Fix, ship, and prove it’s gone

1. Patch the code—every malloc gets a free, every new gets a delete
2. Run the same heaptrack steps again
3. Compare the new graph: flat line = victory

Hiccups you’ll hit and how to dodge them

• Missing symbols? Install debug packages: sudo apt install myapp-dbgsym
• Can’t attach? Add --cap-add=SYS_PTRACE in containers or drop ProtectKernelTunables=yes in the unit
• Logs exploding? Limit trace time to the exact scenario that leaks

Make it part of your routine

Every Friday at 4 p.m. I run a 15-minute “memory check” script in CI. It spins up the service under heaptrack, hits the API with synthetic traffic, and fails the build if the heap grows more than 5 %. Caught three fresh leaks last month—before they ever reached prod.

Start small. Pick one service, profile it for a day, and see what surprises you. Once you’ve done it once, catching leaks feels like brushing your teeth—quick, boring, and absolutely worth it.

FAQ

How do I know if my systemd service has a memory leak?

Use systemd-cgtop -m to watch memory climb, then confirm with heaptrack or perf. If the heap keeps growing under steady load, you’ve got a leak.