A mean can make an uncomfortable service look fast. If ninety-five requests take 50 milliseconds and five take two seconds, the average is only 147.5 milliseconds. A latency histogram retains more information about that slow minority without storing every request. What it preserves is a bucketed approximation of the distribution, not the complete original observations or an automatically exact percentile.

Know what the buckets contain

In a Prometheus classic histogram, a bucket’s le boundary includes every observation less than or equal to that boundary. Counts are cumulative: a 50-millisecond request contributes to the 100-millisecond bucket and every larger one. The positive-infinity bucket covers all observations, while the count and sum components provide the sample population and total observed duration. Adding cumulative bucket counts together double-counts observations.

Subtracting neighboring cumulative counts recovers the counts in disjoint intervals. You still do not know where observations lie inside each interval. Compatible histograms can be aggregated by adding corresponding bucket counts before estimating a percentile. Averaging independently calculated instance percentiles does not recover the combined distribution, even when every individual percentile was accurately calculated.

Resolution determines the answer

For an ordinary positive classic bucket, histogram_quantile interpolates inside the bucket containing the requested rank, assuming observations are distributed uniformly within that interval. Real latency may cluster near either edge. The returned number is therefore an estimate tied to the bucket width, not a newly recovered measurement. Place useful boundaries near important operating thresholds instead of spreading every boundary across irrelevant ranges.

A bucket exactly at a 300-millisecond objective can directly count the observations meeting that threshold. This can answer an SLO question more faithfully than asking whether an interpolated p99 happened to land below it. Choosing more boundaries increases the number of classic bucket series per label combination. Modern Prometheus also supports native histograms, which its current guidance prefers when the instrumentation and surrounding system support them. Standard exponential native buckets use a different interpolation model; the exercise here deliberately isolates classic buckets.

When it breaks

An upper bucket that is too wide makes a small movement of observations produce an implausibly large movement of the estimated tail. Missing the required positive-infinity bucket can make the classic quantile calculation invalid. If the requested rank falls into that final infinite bucket, Prometheus returns the preceding finite boundary rather than inventing an infinite latency estimate. None of these behaviors identifies the exact slowest request.

Empty populations have no observed percentile. Mismatched boundaries across instances, mixed observation units, or invalid nonmonotonic cumulative counts also undermine interpretation. When querying cumulative counters over time, calculate appropriate increases or rates before aggregation so resets are handled. A distribution over all requests additionally needs explicit outcome semantics: fast failures can improve the apparent latency while the service becomes less useful.

Check it quickly

Save this as histogram-shape.py and run Python 3. It calculates exact nearest-rank p99 from a synthetic raw sample, then independently estimates p99 from two classic bucket layouts. The compact interpolation function is restricted to this positive, nonempty fixture and asserts that the selected bucket is finite; it is not a replacement for Prometheus’s complete quantile implementation.

from math import ceil, inf
from statistics import mean

values = [0.05] * 95 + [2.0] * 5
coarse = [0.1, 0.3, 1.0, 5.0, inf]
fine = [0.1, 0.3, 1.0, 2.0, 5.0, inf]

def cumulative(bounds):
    return [sum(value <= bound for value in values) for bound in bounds]

def estimate_p99(bounds):
    counts = cumulative(bounds)
    rank = 0.99 * counts[-1]
    lower, previous = 0.0, 0
    for upper, count in zip(bounds, counts):
        if count >= rank:
            assert upper != inf and count > previous
            return lower + (upper - lower) * (rank - previous) / (count - previous)
        lower, previous = upper, count
    raise AssertionError('rank not covered')

exact = sorted(values)[ceil(0.99 * len(values)) - 1]
counts = cumulative(coarse)
increments = [counts[0]] + [right - left for left, right in zip(counts, counts[1:])]
assert counts == [95, 95, 95, 100, 100]
assert increments == [95, 0, 0, 5, 0] and sum(increments) == 100
assert exact == 2.0 and abs(estimate_p99(coarse) - 4.2) < 1e-12
assert abs(estimate_p99(fine) - 1.8) < 1e-12
print(f"mean={mean(values):.4f}s; raw nearest-rank p99={exact:.1f}s")
print(f"coarse cumulative counts={counts}; sum_of_buckets={sum(counts)}")
print(f"classic linear p99: coarse={estimate_p99(coarse):.1f}s; fine={estimate_p99(fine):.1f}s")
print(f"at or below 0.3s={counts[1]}/{counts[-1]}")

The raw p99 is two seconds, but the coarse buckets estimate 4.2 seconds because the relevant observations are somewhere between one and five seconds. Adding a two-second boundary changes the estimate to 1.8 seconds. The narrower bucket improves the error in this sample, yet the estimate still differs from the exact raw percentile; extra resolution does not recreate the original values. Both estimates also match histogram_quantile in Prometheus 3.5.1 within a numerical tolerance of 1e-12.

The cumulative counts sum to 485 although only one hundred requests exist. Their disjoint increments sum to one hundred, and the 0.3-second bucket gives the exact threshold result of 95 out of 100 for this dataset. Read those outputs together: histogram counts can answer aligned threshold questions precisely while percentile interpolation remains approximate.

Sources