Tail latency and why p99 becomes typical

Key points

• If one call has a p99 of 100ms, a request that makes one call is slow 1% of the time. A request that makes 100 calls is slow about 63% of the time - because it only takes one.
• That is tail latency amplification, and it is why large fan-out systems care about p99.9 while a single service can reasonably stop at p99.
• The average is actively misleading here. A service with a 20ms mean and a 2s p99 will feel broken to a meaningful share of users while every dashboard looks healthy.
• The usual causes of a tail are not the code: garbage collection, a cold cache, queueing behind a slow neighbour, a retry, or an unlucky disk. They are largely independent of the request itself.
• Hedged requests exploit that independence: send a duplicate after the p95 elapses and take the first answer. A few percent extra load can cut the tail dramatically.
• The structural fixes are to reduce fan-out, make optional dependencies genuinely optional with a fast timeout, and return partial results rather than waiting for the slowest component.

Example

/*
 * THE ARITHMETIC.
 *
 *   P(request is fast) = P(one call is fast) ^ number_of_calls
 *
 *   one call at p99 = 100ms:
 *     1 call     0.99^1   = 99.0%  fast   ->  1% slow
 *     10 calls   0.99^10  = 90.4%         -> 10% slow
 *     100 calls  0.99^100 = 36.6%         -> 63% SLOW
 *
 * At 100 calls the p99 case is the MAJORITY experience. This is why a
 * fan-out architecture must care about p99.9, not p99.
 */

// HEDGING - send a second request after the p95, take whichever answers.
public Result fetchHedged(String key) {
    CompletableFuture<Result> first = CompletableFuture
            .supplyAsync(() -> client.get(key), pool);

    CompletableFuture<Result> hedge = new CompletableFuture<>();
    // Only fires if the first has not answered within the p95.
    scheduler.schedule(() -> {
        if (!first.isDone()) {
            CompletableFuture.supplyAsync(() -> client.get(key), pool)
                    .whenComplete((r, e) -> { if (e == null) hedge.complete(r); });
        }
    }, p95Millis, TimeUnit.MILLISECONDS);

    return CompletableFuture.anyOf(first, hedge)
            .thenApply(o -> (Result) o).join();
}
// Cost: a few percent more load, since only the slow tail is duplicated.
// Cap it - hedging everything doubles traffic and makes the tail worse.

/*
 * STRUCTURAL FIXES, in order of effectiveness:
 *
 *   1. fan out to fewer things        the exponent is the problem
 *   2. return partial results         do not wait for the optional 20%
 *   3. aggressive timeouts on         a 50ms budget beats a 5s one that
 *      optional calls                  is technically "successful"
 *   4. hedge the tail                 cheap, bounded, effective
 *   5. make each service faster       LAST - it barely moves the exponent
 */

Fan-out turns the p99 into the common case - reducing the number of calls beats making each call faster.