Investigating Performance

Overview

Performance work fails in a specific way: the engineer reads the code, forms an intuition about what must be slow, optimizes that, and ships a change that measurably does nothing. The bottleneck is somewhere else, because the bottleneck is almost always somewhere else.

Core principle: Measure, localize, change one thing, measure again. Intuition selects what to measure; it never selects what to fix.

Companion skills: systematic-debugging — same discipline applied to wrongness rather than slowness. Both refuse to act before the cause is located.

The Iron Law

NO OPTIMIZATION WITHOUT A MEASUREMENT THAT NAMES THE BOTTLENECK

You must be able to say "X accounts for N% of the time" before you change X. Not "X looks expensive." Not "X is O(n²)." A measurement, with a number.

No exceptions:

• Not for "this is obviously the slow part" — obvious is what makes it worth measuring
• Not for "the fix is cheap anyway" — a cheap fix to a non-bottleneck is pure risk with zero payoff
• Not for changes you would make on style grounds regardless. Make them; just don't call it performance work

The corollary: a change without an after-measurement is not an optimization, it is a guess that shipped.

Step 0: Define the Target

"Slow" is not a bug report. Write down all four before touching anything:

Without a target you cannot tell when you are done, and performance work has no natural stopping point. Without a statistic you will optimize the average and never touch the tail that generated the complaint.

Step 1: Reproduce It

You need a harness that produces the slow behavior on demand, repeatably.

• Real data shape. A thousand dev rows will not reproduce a missing index; ten million will.
• Warm up first, then measure. JIT compilation, connection pools, and page cache all make the first run a different program.
• Repeat and take the distribution. A single timing is noise. Report min/p50/p99, not one number.
• Pin the environment. Same machine, same build flags, no other load. Especially: measure a production build — a dev-mode framework build can be 10× slower and is not the thing you are shipping.

If you cannot reproduce it locally, measure in the environment where it happens. Production profiling at low sample rates is safe and is better than a faithful reproduction you do not have.

Step 2: Characterize Before Profiling

Two minutes here saves an hour of profiling the wrong dimension.

A quick sanity check: concurrency ≈ throughput × latency (Little's Law). If you serve 100 req/s at 2s each, roughly 200 requests are in flight. If the pool holds 20 connections, the pool is the bottleneck and no amount of query tuning will help.

Step 3: Profile the Right Dimension

Profile the dimension the characterization implicated. A CPU profile of an I/O-bound program shows an idle process and teaches you nothing.

Exact invocations per stack — Go, Node, Python, browser, SQL, system-level: references/profiling-tools.md.

Read the profile top-down by cumulative time, and be suspicious of flat profiles. A profile with no peak usually means the cost is spread across a call made far too many times — count the calls, don't just time them.

Step 4: The Usual Suspects

Once localized, the cause is usually on this list.

Before optimizing, ask whether the work can be avoided entirely. Caching, batching, pagination, doing it asynchronously, or not doing it are all bigger wins than making it faster, and usually simpler.

Step 5: Change One Thing, Then Measure Again

one change → re-measure → keep it or revert it → repeat

• One variable at a time. Two changes at once and you cannot attribute the result — including the case where one helped and the other hurt by the same amount.
• Revert what does not measurably help. Every optimization costs readability; unpaid-for complexity is a permanent tax.
• Confirm the improvement is real, not noise. Compare distributions across repeated runs (benchstat and equivalents do this properly), not two single numbers.
• Commit the benchmark with the fix. It is the only thing that will notice when the regression comes back.
• Re-profile after each win. The bottleneck moves. The second-biggest cost is now the biggest, and it is often somewhere entirely new.

Stop when you hit the target from Step 0. Continuing past it trades real maintainability for a number nobody asked for.

Tail Latency

If the complaint is about p99, the mean is actively misleading — a fix that improves the mean often makes the tail worse.

Common causes, in rough order of frequency:

• Queueing. Utilization above ~70% makes wait time rise sharply. The fix is capacity or backpressure, not faster code.
• GC pauses. Look at pause distribution, not total GC time. Reduce allocation rate or retained set.
• Lock convoy. One slow holder makes every waiter slow; latency spikes correlate with contention, not load.
• Retries. A timeout plus a retry turns one slow request into three, exactly when the system is least able to serve them.
• Cold paths. Cache misses, first-request-after-idle, connection re-establishment.
• Head-of-line blocking. One slow item stalls everything behind it in the same connection, batch, or partition.

Watch for coordinated omission in your load generator. A harness that waits for each response before sending the next request stops sending load exactly when the system is slowest — which erases the tail you are trying to measure. Use a generator that sends at a fixed rate, or one that explicitly corrects for it.

Red Flags

• "This is obviously the slow part" — with no profile
• Optimizing a function that a profile shows at 0.4% of runtime
• Benchmarking a dev build, or with an empty database
• A single timing used as evidence
• Two changes in one commit, both labeled performance
• A micro-benchmark improving while the end-to-end metric does not move
• Caching added before the cost of the uncached path was measured
• "It should be faster now"

All of these mean: go back and measure.

Common Mistakes

Optimizing the part you understand. Familiarity is not evidence. The bottleneck is disproportionately in code you did not write.

Micro-benchmarks that do not reflect real use. Amdahl's law is unforgiving: making 5% of the runtime twice as fast buys 2.5%.

Confusing allocation with retention. High allocation rate is a GC-pressure problem. Growing in-use heap is a leak. They need different profiles and different fixes.

Caching to hide a bug. A cache in front of an accidentally-quadratic function makes the problem invisible until the cache misses under exactly the conditions where you need it most.

Ignoring variance. A change that improves p50 and degrades p99 is usually a regression, because the tail is what users report.

No after-measurement. Without it you have shipped a diff, not a fix — and you have no idea which it was.