The Tail of Two Metrics

When a colleague casually remarked, “Let’s use average response time to show how much faster the query API got,” I felt a sudden spike in my own latency.

I managed to contain myself, though my voice still came out cold. “Why use the average? Just because it makes our optimization look good?”

He stared at me blankly. “Why shouldn’t I?”

“Because P99 degraded!” I said. “Do you even know what P99 measures?”

“What’s the big deal?” he shrugged. “The average shows an improvement. Isn’t that enough?”

I was stunned. Measuring performance with percentiles like P99 felt like engineering 101. But shortly after the argument, a unsettling realization hit me: I couldn’t actually articulate why percentiles were superior beyond a gut feeling.

I knew percentiles were more realistic than averages. If your P99 response time is 10ms, 99 out of 100 requests finish within 10ms. Meanwhile, an average of 1ms can be dangerously misleading if a few requests take 5 seconds. But why does software behave this way? Why does our intuition crave the average while reality demands the tail?

I decided to dig deeper.

Percentiles weren’t invented for computer systems. In the 19th century, Sir Francis Galton used them to analyze human traits—heights, weights, and test scores. But human heights follow a neat, symmetrical Bell curve. Software response times do not. Software performance lives in a world of heavy tails.

Why is latency long-tailed? It comes down to two inherent layers of modern software:

The In-Process Tax: Modern runtimes rely on garbage collection, dynamic memory allocation, and CPU scheduling. Most of the time, code executes in nanoseconds. But occasionally, a thread hits a GC pause, lock contention, or thread pool exhaustion.
The Distributed Tax: Microservices don’t exist in isolation. Service A calls Service B, which calls Service C. At each hop, network latency multiplies.

This is where our intuition breaks down. We tend to view software linearly, assuming components act independently. But modern systems are networks of multiplicative dependencies.

Consider a typical dashboard request that fans out to 10 microservices, each with a 99% success rate (P99 of acceptable latency). The probability of the entire dashboard loading without hitting a tail-end lag isn’t 99%—it drops to $(0.99)^{10} \approx 90%$. Add 50 downstream calls, and your chance of hitting a smooth response falls below 60%.

The tail isn’t an anomaly or an edge case. In a distributed network, the tail is the statistical destination of compounding probability.

Averages assume a linear world where bad luck is evenly smoothed out. They offer the comforting illusion that everything is mostly fine. But complex software isn’t linear. Systems don’t break because the median got slower. They break when minor glitches snowball into total failure.

My colleague wasn’t being lazy; he was falling for a classic psychological trap: choosing the metric that offers comfort over the metric that exposes risk.

Averages tell you how your system performs when everything goes right. Percentiles tell you how your system behaves when reality happens. And in software engineering, surviving reality is the only metric that counts.