A latency mean can conceal a slow subset of requests. A histogram retains more distribution information by counting observations into buckets, but it still discards detail within each bucket. Prometheus can estimate quantiles from those counts without storing every observation. The estimate depends on bucket boundaries and aggregation; it is not a recovery of the complete original latency distribution.

A classic histogram exports cumulative _bucket counters, a _count, and a _sum. The le label identifies each inclusive bucket upper bound. An observation of 1.5 seconds increments the two-second bucket and every larger bucket, so bucket values must not be summed as though they were disjoint counts. The positive-infinity bucket includes all observations and corresponds to the count.

For nonnegative latency observations, applying rate to the sum and count gives total seconds accumulated per second and observations per second. Their ratio is mean latency in seconds. Bucket rates describe the distribution over the selected time range. Apply the counter rate before aggregating instances so each series’ observed reset is handled independently.

Aggregate buckets before estimating a quantile

For one fleet percentile, sum the bucket rates while retaining le, then call histogram_quantile. Keep any grouping dimensions required by the question, such as service. Compatible bucket boundaries and matching event definitions are necessary for meaningful aggregation. Removing le destroys the bucket identity the classic quantile function needs.

Do not average instance percentiles. A percentile is a position in a distribution, not an additive count, and even equal request volumes do not make percentile averaging valid. The combined bucket curve can have a very different shape from either instance. If the question is the fraction meeting a two-second objective, a bucket exactly at two seconds divided by the total count answers that threshold question directly.

A summary may export client-computed quantiles. Those values cannot be merged into a fleet quantile by summing or averaging them. Its count and sum can still contribute to a combined mean when event definitions match. Select the metric type from the aggregation you need, not solely from a panel’s preferred display.

When it breaks

A wide bucket near the desired percentile can make the estimate look more precise than the measurements justify. Classic quantile calculation interpolates within the selected bucket; it does not know where the individual observations occurred inside it. Keep boundaries useful around service objectives, and review them before a rollout that mixes different schemas across replicas. Refining future buckets cannot recover detail already discarded from historical observations.

A missing bucket or exporter is also different from zero observations. An apparently better percentile can result from losing the slow instance’s metrics. Check scrape completeness, contributing series, and sample counts before interpreting improvement. Sparse or absent observations can yield no useful quantile; replacing such results with zero latency creates a misleading success signal.

Check it quickly

Save the program as histogram-query.py and run python3 histogram-query.py. It requires Docker and the Prometheus 3.5.1 image, which may be fetched initially. Real promtool evaluates classic histogram queries against minute-spaced synthetic counters. Each instance contributes one hundred observations per minute, but their bucket shapes and accumulated latency differ substantially.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"1\"}"
        values: "0+95x5"
      - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"2\"}"
        values: "0+100x5"
      - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"5\"}"
        values: "0+100x5"
      - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"+Inf\"}"
        values: "0+100x5"
      - series: "api_duration_seconds_count{instance=\"fast\"}"
        values: "0+100x5"
      - series: "api_duration_seconds_sum{instance=\"fast\"}"
        values: "0+17x5"
      - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"1\"}"
        values: "0+0x5"
      - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"2\"}"
        values: "0+0x5"
      - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"5\"}"
        values: "0+100x5"
      - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"+Inf\"}"
        values: "0+100x5"
      - series: "api_duration_seconds_count{instance=\"slow\"}"
        values: "0+100x5"
      - series: "api_duration_seconds_sum{instance=\"slow\"}"
        values: "0+400x5"
    promql_expr_test:
      - expr: "abs(histogram_quantile(0.99, sum by (le) (rate(api_duration_seconds_bucket[2m]))) - 4.94) < bool 1e-12"
        eval_time: 5m
        exp_samples:
          - labels: "{}"
            value: 1
      - expr: "abs(avg(histogram_quantile(0.99, rate(api_duration_seconds_bucket[2m]))) - 3.385) < bool 1e-12"
        eval_time: 5m
        exp_samples:
          - labels: "{}"
            value: 1
      - expr: "sum(rate(api_duration_seconds_bucket{le=\"2\"}[2m])) / sum(rate(api_duration_seconds_count[2m]))"
        eval_time: 5m
        exp_samples:
          - labels: "{}"
            value: 0.5
      - expr: "abs(sum(rate(api_duration_seconds_sum[2m])) / sum(rate(api_duration_seconds_count[2m])) - 2.085) < bool 1e-12"
        eval_time: 5m
        exp_samples:
          - labels: "{}"
            value: 1
"""
rules = r""""""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
    script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
    result = subprocess.run(
        ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
         "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
        input=script, text=True, capture_output=True, timeout=45,
    )
finally:
    subprocess.run(["docker", "rm", "-f", name],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
    print(result.stderr, end="")
    raise SystemExit(result.returncode)

The expected output is SUCCESS. The correctly combined p99 estimate is 4.94 seconds. Averaging the two instance p99 estimates produces 3.385 seconds, which is not the fleet p99. Half the observations meet the two-second threshold, and the combined mean is 2.085 seconds. Numerical assertions allow a tiny floating-point tolerance rather than requiring an identical binary representation of decimal fractions.

The fixture validates PromQL arithmetic and aggregation, not actual request instrumentation or the distribution within a bucket. It uses no external network or host mounts and removes its disposable container. Current Prometheus guidance also discusses native histograms, which encode buckets differently and can simplify queries when the instrumentation and storage path support them. This example deliberately targets classic histograms: retain le, aggregate their counter rates correctly, and describe the returned percentile as an estimate.

Sources