Metrics · XB Field Notes

Prometheus rate estimates counter growth per second

Fri, 24 Dec 2027 00:00:00 +0000

A request counter answers how many events an instrumented process has accumulated. A rate answers how quickly those events are arriving. Prometheus rate estimates average growth per second over a selected range, adjusts observed counter resets, and extrapolates toward the range boundaries. The result is useful precisely because it has a time unit, but it is not an exact reconstruction of every event between scrapes.

Sampling determines what can be inferred

An ordinary counter increases until it resets, commonly after a restart. A decrease between observed samples is treated as a reset by the counter functions. A restart and recovery entirely between two scrapes may be invisible if the new value already exceeds the previous sample. More frequent scraping improves observation resolution; no query can recover an unobserved history with certainty.

Apply rate to individual counter series before aggregating them. Summing first can let one instance’s growth hide another instance’s reset. Labels define series identity, so a replacement instance with different labels starts a different series rather than continuing the old one. A gauge such as queue depth can fall normally and should not be interpreted as a resetting counter.

The selected range is part of the meaning

A range selector is open at its left boundary and closed at its right boundary. At minute five, [5m] excludes a sample exactly at minute zero. The float-counter calculation needs at least two usable samples. A range containing one sample produces no rate value; it does not establish zero traffic. Leave enough room for the configured cadence and realistic missed scrapes.

Longer windows generally smooth short bursts and delay their disappearance from the average. Shorter windows respond sooner but offer fewer observations and can become empty. Choose the window from the event duration and detection objective, then label it explicitly. irate uses the last two samples, whereas rate uses the selected range. Both can be useful on the same metric if panels explain their different questions.

The query evaluation interval controls when a rule asks the question, not which samples were scraped. Refreshing a dashboard every second cannot create new observations from a target scraped every minute. Inspect these two cadences separately when assessing alert responsiveness.

When it breaks

A newly created series may vanish from a rate panel until a second sample arrives. Filling every missing result with zero can conceal collection failure. Check the raw series, timestamps, and scrape status before treating a blank panel as idle traffic. A very smooth long-window line can also hide a brief overload that matters to users.

Reset adjustment is not immunity to gaps. Missing samples can hide bursts or multiple restarts, and boundary extrapolation introduces estimation. increase applies the corresponding range estimate in event-count units; it may return a fractional count even for integer counters. Use application accounting records when exact totals are required, rather than promising that a dashboard reconstructs them.

Check it quickly

Save this program as rate.py and run python3 rate.py. Docker and the Prometheus 3.5.1 image are required; Docker can download the image if needed. The program invokes the real promtool engine against synthetic samples spaced one minute apart. The isolated container has no network access or host mounts and is removed on exit.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
tests:
 - interval: 1m
 input_series:
 - series: "demo_requests_total{instance=\"steady\"}"
 values: "0+60x5"
 - series: "demo_requests_total{instance=\"restart\"}"
 values: "100 160 220 20 80 140"
 - series: "demo_requests_total{instance=\"gaps\"}"
 values: "0 60 _ 180 _ 300"
 - series: "demo_requests_total{instance=\"one\"}"
 values: "_ _ _ _ _ 7"
 promql_expr_test:
 - expr: "rate(demo_requests_total{instance=\"steady\"}[5m])"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"steady\"}"
 value: 1
 - expr: "abs(rate(demo_requests_total{instance=\"restart\"}[5m]) - 5/6) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"restart\"}"
 value: 1
 - expr: "increase(demo_requests_total{instance=\"restart\"}[5m])"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"restart\"}"
 value: 250
 - expr: "irate(demo_requests_total{instance=\"restart\"}[5m])"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"restart\"}"
 value: 1
 - expr: "rate(demo_requests_total{instance=\"gaps\"}[5m])"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"gaps\"}"
 value: 1
 - expr: "rate(demo_requests_total{instance=\"one\"}[5m])"
 eval_time: 5m
 exp_samples: []
"""
rules = r""""""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
 script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

Successful assertions print SUCCESS. The steady series grows at one request per second. For samples 100 160 220 20 80 140, the five-minute rate is 5/6, increase is 250, and irate is one. These differ because the selected range excludes the first sample, corrects the observed reset, and extrapolates its average across the requested duration.

The deliberately linear series with gaps still yields one per second; that controlled result does not prove arbitrary gaps harmless. The one-sample series yields an empty vector. The test checks values and absence separately, using numerical tolerance for the repeating fraction. Before deploying a rule, add representative restart, sparse-traffic, and missing-data cases from the service whose counter you will monitor.

Sources

Prometheus: query functions, rate, increase, irate and reset adjustment, checked 2026-09-13.
Prometheus: querying basics, range boundaries and sample selection, checked 2026-09-13.
Prometheus: unit testing rules, synthetic samples and expression assertions, checked 2026-09-13.

Cardinality is a metric budget

Wed, 21 Jul 2027 00:00:00 +0000

A metric costs more than its short name suggests. Prometheus identifies a time series by its metric name and complete label set. A new observed label combination creates a different series, which must be ingested, indexed, retained, and queried. Cardinality budgeting asks which distinctions are worth paying for and whether their number stays bounded as the service grows.

Count combinations that actually exist

Label dimensions multiply the possible combinations, but the product of their distinct values is an upper bound unless every combination occurs. Two routes, two methods, and two status categories allow eight combinations. A service that only accepts one method on each route may emit fewer. Count observed combinations as well as theoretical limits, and include target labels such as instance when estimating what the server will ingest.

The value set matters more than the label’s friendly name. A normalized route such as /orders/{order_id} groups operations; a raw path containing each order identifier creates a growing dimension. Request IDs, timestamps, user identifiers, and arbitrary query strings can grow with event volume. A label named route is not bounded merely because the developer intended it to describe a route. Check the values produced on real traffic, including errors and uncommon paths.

Include the metric representation

A counter usually contributes one series for each observed label combination, before optional companion metrics. A classic histogram expands the same combination into its bucket, sum, and count series. With six finite bucket boundaries, one positive-infinity bucket, a sum, and a count, that is nine core series per combination. Some clients expose an additional creation-time series; include it in a real inventory rather than assuming this simplified count is complete.

Replicas add their own instance dimension. Scrape interval changes samples per series, while retention changes how long historical samples remain; neither is the same as the number of distinct series. Rapidly changing identifiers also create churn: yesterday’s short-lived combinations may remain in storage even after their producers disappear. Monitor active series, creation rate, ingestion volume, memory, and query behavior instead of using one fixed bytes-per-series estimate for every deployment.

When it breaks

An innocent-looking diagnostic label can turn an aggregate metric into a per-request event store. Removing that label from a dashboard query does not undo the cost already paid to ingest and retain it. Hashing each identifier still preserves its distinctness, and truncating a random identifier generally only postpones the problem while introducing collisions. Choose a deliberately bounded category when aggregation is the actual requirement.

Do not collapse away distinctions needed to detect customer impact. Keep useful bounded outcome or service dimensions, and move request-level investigation to appropriately controlled logs or traces. If unwanted labels are removed during collection, check for duplicate resulting series within a scrape and for lost diagnostic meaning. Prefer fixing the producer where possible, then verify the new label set and plan how historical series age out.

Check it quickly

Save this as cardinality.py and run Python 3. The synthetic events exercise every route, method, and status combination fifty times. One version retains only those bounded dimensions; another also includes a unique request ID. The calculation assumes twenty replicas each expose the same combinations as classic histograms with six finite boundaries.

from itertools import product

routes = ["/orders/{order_id}", "/products/{product_id}"]
methods = ["GET", "POST"]
statuses = ["200", "500"]
events = list(product(routes, methods, statuses, range(50)))
bounded = {(route, method, status) for route, method, status, _ in events}
with_request_id = {(route, method, status, f"request-{i}")
 for i, (route, method, status, _) in enumerate(events)}

finite_buckets = 6
# Core classic-histogram series: finite buckets, +Inf, _sum, _count.
# Optional _created series are deliberately excluded from this calculation.
series_per_combination = finite_buckets + 3
replicas = 20
bounded_series = len(bounded) * series_per_combination * replicas
request_id_series = len(with_request_id) * series_per_combination * replicas

assert len(events) == 400 and len(bounded) == 8
assert len(with_request_id) == 400
assert bounded_series == 1440 and request_id_series == 72000
print(f"observed label sets: bounded={len(bounded)}, request_id={len(with_request_id)}")
print(f"core classic-histogram series: bounded={bounded_series}, request_id={request_id_series}")
print(f"multiplier={request_id_series // bounded_series}x")

Four hundred events collapse into eight bounded combinations. Adding request IDs keeps four hundred distinct combinations, giving 1,440 versus 72,000 core histogram series across the twenty replicas: a fiftyfold increase. Optional creation-time series and other metric families are excluded explicitly, so these are modeled series counts rather than a measurement of Prometheus memory or storage use.

The comparison demonstrates why aggregating events at instrumentation time changes the cost profile. Before approving another label, state the operational question it answers, its maximum expected value set, the combinations actually emitted, and the effect on each metric family. Then validate that estimate against the collector and server as the workload changes.

Sources

Prometheus: Data Model, series identity and changes in label sets, checked 2026-09-13.
Prometheus: Metric and Label Naming, bounded labels and high-cardinality dimensions, checked 2026-09-13.
Prometheus: Instrumentation, cardinality costs and instrumentation choices, checked 2026-09-13.

Pushgateway retains batch metrics until their lifecycle is handled

Thu, 13 May 2027 00:00:00 +0000

A short-lived job can finish before Prometheus next scrapes it. Pushgateway offers an intermediary that retains pushed metrics for later scraping, but it is not the default answer for every short process. Prometheus recommends it primarily for suitable service-level batch outcomes. The retained data has a lifecycle independent of the job, so successful scraping does not prove the job recently ran or succeeded.

Identify a stable batch outcome

A service-level job, such as a nightly reconciliation, can publish its last result under a stable grouping key. Putting a new run identifier, timestamp, or machine identity in that key creates additional groups instead of updating one result. Pushgateway has no automatic per-group TTL that removes obsolete runs. Owners must replace or delete the relevant data deliberately.

Machine-specific batch results can often use a node exporter’s textfile collector instead. A gateway also introduces a shared availability dependency. Its up value describes whether Prometheus can scrape the gateway, not whether each original pusher is healthy. Choose the collection model around the monitored event and ownership of cleanup, rather than treating pushing as a universal way around discovery or network policy.

Replacement is defined by the HTTP method

The URL path defines a grouping key, including the job and any additional label pairs. HTTP PUT replaces the metrics belonging to that exact group. POST replaces metric families with the names included in the request while preserving other families in the group. POST does not mean increment the previous value or append an event to a durable log.

DELETE removes the exact group identified by its URL. Deleting a job-only grouping key does not delete every group that also has that job plus additional labels. The gateway’s documented method ordering allows a later successful push to follow an earlier queued deletion. This matters when verifying cleanup instead of assuming that an accepted deletion has already affected every matching-looking series.

Scrape time is not last success time

The gateway exports push_time_seconds for the last successful push to a group. Reading the metrics endpoint does not advance that value. Prometheus assigns scrape timestamps to the samples it collects, so recently scraped samples can still describe a very old job result. Publish a separate last-success timestamp as a gauge value when the job’s successful completion time is the actual question.

Do not attach an explicit sample timestamp to pushed exposition: Pushgateway rejects those timestamps. A Unix timestamp stored as a gauge’s value is different and is supported. A successful push also establishes neither business success nor immediate durable persistence. A job can report failure successfully, and a gateway without persistence can lose held data when it restarts.

When it breaks

A failed or retired job can leave its old success metric available indefinitely. An alert checking only that the series exists will miss the missing execution. Compare the last-success value with the expected schedule, and keep job removal synchronized with group cleanup. A changing grouping label can make old and new runs appear side by side forever.

Using POST when a complete snapshot was intended can retain a metric the new run no longer publishes. Using PUT with only one metric can erase other results needed by dashboards. Review which component owns the entire group and which method it uses. There is no --replace option that substitutes for these HTTP semantics.

Check it quickly

Save this as pushgateway.py and run python3 pushgateway.py. Docker and the Pushgateway 1.11.1 image are required; an uncached image may be downloaded. The disposable server accepts requests only through container loopback. No host port is published, no host directory is mounted, and cleanup removes only the uniquely named test container.

import http.client
import io
import json
import subprocess
import time
import uuid

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


port = 9091
script = r"""exec /bin/pushgateway --web.listen-address=127.0.0.1:9091
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/pushgateway:v1.11.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: request("GET", "/metrics")[0] == 200)
 group = "/metrics/job/nightly_sync/team/data"
 first = "# TYPE items_processed gauge\nitems_processed 7\n# TYPE last_success_unixtime gauge\nlast_success_unixtime 1234\n"
 assert request("PUT", group, first, "text/plain; version=0.0.4")[0] == 200
 def snapshot():
 status, body = request("GET", "/metrics")
 assert status == 200
 return {line.split("{", 1)[0]: float(line.rsplit(" ", 1)[1])
 for line in body.splitlines()
 if not line.startswith("#") and 'job="nightly_sync"' in line}
 def first_snapshot():
 data = snapshot()
 return data if data.get("items_processed") == 7 else None
 initial = eventually(first_snapshot)
 assert initial["last_success_unixtime"] == 1234
 time.sleep(0.2)
 assert snapshot()["push_time_seconds"] == initial["push_time_seconds"]
 partial = "# TYPE items_processed gauge\nitems_processed 9\n"
 assert request("POST", group, partial, "text/plain; version=0.0.4")[0] == 200
 after_post = snapshot()
 assert after_post["items_processed"] == 9 and after_post["last_success_unixtime"] == 1234
 replacement = "# TYPE items_processed gauge\nitems_processed 11\n"
 assert request("PUT", group, replacement, "text/plain; version=0.0.4")[0] == 200
 after_put = snapshot()
 assert after_put["items_processed"] == 11 and "last_success_unixtime" not in after_put
 assert request("DELETE", "/metrics/job/nightly_sync")[0] == 202
 # A successful later POST orders the preceding queued deletion before this check.
 assert request("POST", group, "", "text/plain; version=0.0.4")[0] == 200
 assert snapshot()["items_processed"] == 11
 assert request("DELETE", group)[0] == 202
 eventually(lambda: not snapshot())
 print(json.dumps({"initial_items": 7, "post_items": 9, "put_items": 11,
 "post_preserved_other_metric": True, "put_removed_other_metric": True,
 "scrapes_did_not_refresh_push_time": True,
 "parent_group_delete_did_not_match": True, "group_deleted": True}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

Assertions show seven processed items after the initial PUT, nine after POST while the separate last-success gauge remains, and eleven after a replacement PUT that removes that other gauge. Repeated reads leave push time unchanged. Deleting the job-only group leaves the team-qualified group intact; deleting the exact group removes its metrics. The fixture tests gateway API behavior, not job scheduling, Prometheus integration or durable recovery.

Sources

Prometheus: when to use the Pushgateway, limited service-level batch use cases and retained series, checked 2026-09-13.
Pushgateway 1.11.1: README, PUT, POST, DELETE, grouping keys and push timestamps, checked 2026-09-13.

Prometheus histograms retain a bucketed distribution

Wed, 06 Jan 2027 00:00:00 +0000

A latency mean can conceal a slow subset of requests. A histogram retains more distribution information by counting observations into buckets, but it still discards detail within each bucket. Prometheus can estimate quantiles from those counts without storing every observation. The estimate depends on bucket boundaries and aggregation; it is not a recovery of the complete original latency distribution.

A classic histogram exports cumulative _bucket counters, a _count, and a _sum. The le label identifies each inclusive bucket upper bound. An observation of 1.5 seconds increments the two-second bucket and every larger bucket, so bucket values must not be summed as though they were disjoint counts. The positive-infinity bucket includes all observations and corresponds to the count.

For nonnegative latency observations, applying rate to the sum and count gives total seconds accumulated per second and observations per second. Their ratio is mean latency in seconds. Bucket rates describe the distribution over the selected time range. Apply the counter rate before aggregating instances so each series’ observed reset is handled independently.

Aggregate buckets before estimating a quantile

For one fleet percentile, sum the bucket rates while retaining le, then call histogram_quantile. Keep any grouping dimensions required by the question, such as service. Compatible bucket boundaries and matching event definitions are necessary for meaningful aggregation. Removing le destroys the bucket identity the classic quantile function needs.

Do not average instance percentiles. A percentile is a position in a distribution, not an additive count, and even equal request volumes do not make percentile averaging valid. The combined bucket curve can have a very different shape from either instance. If the question is the fraction meeting a two-second objective, a bucket exactly at two seconds divided by the total count answers that threshold question directly.

A summary may export client-computed quantiles. Those values cannot be merged into a fleet quantile by summing or averaging them. Its count and sum can still contribute to a combined mean when event definitions match. Select the metric type from the aggregation you need, not solely from a panel’s preferred display.

When it breaks

A wide bucket near the desired percentile can make the estimate look more precise than the measurements justify. Classic quantile calculation interpolates within the selected bucket; it does not know where the individual observations occurred inside it. Keep boundaries useful around service objectives, and review them before a rollout that mixes different schemas across replicas. Refining future buckets cannot recover detail already discarded from historical observations.

A missing bucket or exporter is also different from zero observations. An apparently better percentile can result from losing the slow instance’s metrics. Check scrape completeness, contributing series, and sample counts before interpreting improvement. Sparse or absent observations can yield no useful quantile; replacing such results with zero latency creates a misleading success signal.

Check it quickly

Save the program as histogram-query.py and run python3 histogram-query.py. It requires Docker and the Prometheus 3.5.1 image, which may be fetched initially. Real promtool evaluates classic histogram queries against minute-spaced synthetic counters. Each instance contributes one hundred observations per minute, but their bucket shapes and accumulated latency differ substantially.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
tests:
 - interval: 1m
 input_series:
 - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"1\"}"
 values: "0+95x5"
 - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"2\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"5\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"+Inf\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_count{instance=\"fast\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_sum{instance=\"fast\"}"
 values: "0+17x5"
 - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"1\"}"
 values: "0+0x5"
 - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"2\"}"
 values: "0+0x5"
 - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"5\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"+Inf\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_count{instance=\"slow\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_sum{instance=\"slow\"}"
 values: "0+400x5"
 promql_expr_test:
 - expr: "abs(histogram_quantile(0.99, sum by (le) (rate(api_duration_seconds_bucket[2m]))) - 4.94) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "abs(avg(histogram_quantile(0.99, rate(api_duration_seconds_bucket[2m]))) - 3.385) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "sum(rate(api_duration_seconds_bucket{le=\"2\"}[2m])) / sum(rate(api_duration_seconds_count[2m]))"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 0.5
 - expr: "abs(sum(rate(api_duration_seconds_sum[2m])) / sum(rate(api_duration_seconds_count[2m])) - 2.085) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
"""
rules = r""""""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
 script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

The expected output is SUCCESS. The correctly combined p99 estimate is 4.94 seconds. Averaging the two instance p99 estimates produces 3.385 seconds, which is not the fleet p99. Half the observations meet the two-second threshold, and the combined mean is 2.085 seconds. Numerical assertions allow a tiny floating-point tolerance rather than requiring an identical binary representation of decimal fractions.

The fixture validates PromQL arithmetic and aggregation, not actual request instrumentation or the distribution within a bucket. It uses no external network or host mounts and removes its disposable container. Current Prometheus guidance also discusses native histograms, which encode buckets differently and can simplify queries when the instrumentation and storage path support them. This example deliberately targets classic histograms: retain le, aggregate their counter rates correctly, and describe the returned percentile as an estimate.

Sources

Prometheus: histograms and summaries, bucketed observations, aggregation and metric choices, checked 2026-09-13.
Prometheus: query functions, classic histogram_quantile interpolation and rate ordering, checked 2026-09-13.
Prometheus: unit testing rules, executing expected PromQL values, checked 2026-09-13.

Put the unit in the metric name

Fri, 30 Oct 2026 00:00:00 +0000

A metric named request_duration leaves readers guessing whether 250 means seconds, milliseconds, or something else. A Prometheus-facing name such as http_request_duration_seconds carries that contract into queries, recording rules, and alerts. The contract still needs a correct implementation. Appending _seconds to measurements collected in milliseconds changes the spelling, not the magnitude, and can create a thousandfold monitoring error.

Convert at a defined boundary

Choose one quantity and one unit for each metric family. Prometheus recommends base units such as seconds and bytes, with a unit suffix where applicable. A duration and an absolute timestamp can both use seconds, but they are different quantities: last_success_timestamp_seconds should not be combined with request_duration_seconds. Describe the measurement and boundary in its help text so that a useful name has an equally clear definition.

If a library reports milliseconds, convert once before observation or export. Preserve sufficient precision instead of truncating a short duration to zero. Configure the dashboard to display the base unit appropriately; a formatter may show 0.250 seconds as 250 milliseconds without changing stored samples. Do not add another conversion in a recording rule because the panel happens to use milliseconds on its axis.

Unit and type answer different questions

An accumulating byte count conventionally ends in _bytes_total; a current memory usage gauge ends in _bytes. Both measure bytes, but only the former represents accumulated work and is suitable for a counter rate. Counts of requests conventionally use _total without inventing a physical unit. Histogram families add their own component suffixes, such as _seconds_bucket, _seconds_sum, and _seconds_count, to the duration family name.

A ratio commonly represents a fraction between zero and one; percentages use a different numeric scale. State which one a panel expects. A counter may reset when its process restarts, and Prometheus counter-rate functions account for decreases within the observed series. An additional _created series is not required for that behavior. Apply rate calculations before aggregating independent counters so that their individual resets remain visible.

When it breaks

A correctly named metric can still contain incorrect values. Mixing seconds and milliseconds under the same identity contaminates history, so changing only a label or help string cannot repair old samples. Treat a unit migration as a contract change: update the producer, queries, alerts, and dashboards together, and keep historical comparisons explicit. A new identity may be appropriate when the old series has an incompatible meaning.

The suffix also cannot enforce counter behavior. A gauge named _total remains a bad input to a counter-rate interpretation if it moves down during ordinary operation. Conversely, dividing an already calculated per-second rate by the interval again invents a different quantity. These are dimensional and semantic errors, not display problems. Other telemetry conventions may carry units in metadata rather than names; follow the documented exporter mapping instead of renaming every upstream instrument blindly.

Check it quickly

Save the following as units.py and run Python 3. It converts three synthetic latency measurements and emits a small Prometheus text exposition with help and type declarations. Decimal arithmetic makes the millisecond-to-second conversion explicit. The comparison demonstrates why a renamed but unconverted producer cannot share the same latency threshold.

from decimal import Decimal

latency_ms = [Decimal('120'), Decimal('250'), Decimal('630')]
latency_seconds = [value / Decimal('1000') for value in latency_ms]
threshold_seconds = Decimal('0.300')
correct_good = sum(value <= threshold_seconds for value in latency_seconds)
rename_only_good = sum(value <= threshold_seconds for value in latency_ms)
transferred_bytes = 1024 * 1024
used_ratio = Decimal('25') / Decimal('100')

assert correct_good == 2 and rename_only_good == 0
assert sum(latency_seconds) == Decimal('1.000')
assert used_ratio == Decimal('0.25')
print(f"# within 0.300s: converted={correct_good}/3, rename_only={rename_only_good}/3")
print('# HELP demo_last_request_duration_seconds Duration of the most recent completed request.')
print('# TYPE demo_last_request_duration_seconds gauge')
print(f'demo_last_request_duration_seconds {latency_seconds[-1]:.3f}')
print('# HELP demo_transferred_bytes_total Bytes transferred since this process started.')
print('# TYPE demo_transferred_bytes_total counter')
print(f'demo_transferred_bytes_total {transferred_bytes}')
print('# HELP demo_capacity_used_ratio Fraction of configured capacity currently used.')
print('# TYPE demo_capacity_used_ratio gauge')
print(f'demo_capacity_used_ratio {used_ratio}')

The first comment reports two of three observations within 0.300 seconds after conversion, compared with zero of three when millisecond values are merely treated as seconds. The exported last-request gauge is 0.630 seconds, the byte counter is 1,048,576, and the capacity ratio is 0.25, corresponding to 25% in a percentage display.

The complete output passed promtool check metrics using Prometheus 3.5.1. That parser check establishes valid exposition and naming lint for this sample; it cannot infer whether real instrumentation measured the promised quantity. Verify a known input through collection, export, query, and display before trusting a renamed production metric.

Sources

Prometheus: Metric and Label Naming, base units, suffixes, and conventions at the Prometheus boundary, checked 2026-09-13.
Prometheus: Data Model, metric identity and labels, checked 2026-09-13.
Prometheus: Query Functions, counter rates and reset handling, checked 2026-09-13.

Histograms preserve the latency shape

Sun, 20 Sep 2026 00:00:00 +0000

A mean can make an uncomfortable service look fast. If ninety-five requests take 50 milliseconds and five take two seconds, the average is only 147.5 milliseconds. A latency histogram retains more information about that slow minority without storing every request. What it preserves is a bucketed approximation of the distribution, not the complete original observations or an automatically exact percentile.

Know what the buckets contain

In a Prometheus classic histogram, a bucket’s le boundary includes every observation less than or equal to that boundary. Counts are cumulative: a 50-millisecond request contributes to the 100-millisecond bucket and every larger one. The positive-infinity bucket covers all observations, while the count and sum components provide the sample population and total observed duration. Adding cumulative bucket counts together double-counts observations.

Subtracting neighboring cumulative counts recovers the counts in disjoint intervals. You still do not know where observations lie inside each interval. Compatible histograms can be aggregated by adding corresponding bucket counts before estimating a percentile. Averaging independently calculated instance percentiles does not recover the combined distribution, even when every individual percentile was accurately calculated.

Resolution determines the answer

For an ordinary positive classic bucket, histogram_quantile interpolates inside the bucket containing the requested rank, assuming observations are distributed uniformly within that interval. Real latency may cluster near either edge. The returned number is therefore an estimate tied to the bucket width, not a newly recovered measurement. Place useful boundaries near important operating thresholds instead of spreading every boundary across irrelevant ranges.

A bucket exactly at a 300-millisecond objective can directly count the observations meeting that threshold. This can answer an SLO question more faithfully than asking whether an interpolated p99 happened to land below it. Choosing more boundaries increases the number of classic bucket series per label combination. Modern Prometheus also supports native histograms, which its current guidance prefers when the instrumentation and surrounding system support them. Standard exponential native buckets use a different interpolation model; the exercise here deliberately isolates classic buckets.

When it breaks

An upper bucket that is too wide makes a small movement of observations produce an implausibly large movement of the estimated tail. Missing the required positive-infinity bucket can make the classic quantile calculation invalid. If the requested rank falls into that final infinite bucket, Prometheus returns the preceding finite boundary rather than inventing an infinite latency estimate. None of these behaviors identifies the exact slowest request.

Empty populations have no observed percentile. Mismatched boundaries across instances, mixed observation units, or invalid nonmonotonic cumulative counts also undermine interpretation. When querying cumulative counters over time, calculate appropriate increases or rates before aggregation so resets are handled. A distribution over all requests additionally needs explicit outcome semantics: fast failures can improve the apparent latency while the service becomes less useful.

Check it quickly

Save this as histogram-shape.py and run Python 3. It calculates exact nearest-rank p99 from a synthetic raw sample, then independently estimates p99 from two classic bucket layouts. The compact interpolation function is restricted to this positive, nonempty fixture and asserts that the selected bucket is finite; it is not a replacement for Prometheus’s complete quantile implementation.

from math import ceil, inf
from statistics import mean

values = [0.05] * 95 + [2.0] * 5
coarse = [0.1, 0.3, 1.0, 5.0, inf]
fine = [0.1, 0.3, 1.0, 2.0, 5.0, inf]

def cumulative(bounds):
 return [sum(value <= bound for value in values) for bound in bounds]

def estimate_p99(bounds):
 counts = cumulative(bounds)
 rank = 0.99 * counts[-1]
 lower, previous = 0.0, 0
 for upper, count in zip(bounds, counts):
 if count >= rank:
 assert upper != inf and count > previous
 return lower + (upper - lower) * (rank - previous) / (count - previous)
 lower, previous = upper, count
 raise AssertionError('rank not covered')

exact = sorted(values)[ceil(0.99 * len(values)) - 1]
counts = cumulative(coarse)
increments = [counts[0]] + [right - left for left, right in zip(counts, counts[1:])]
assert counts == [95, 95, 95, 100, 100]
assert increments == [95, 0, 0, 5, 0] and sum(increments) == 100
assert exact == 2.0 and abs(estimate_p99(coarse) - 4.2) < 1e-12
assert abs(estimate_p99(fine) - 1.8) < 1e-12
print(f"mean={mean(values):.4f}s; raw nearest-rank p99={exact:.1f}s")
print(f"coarse cumulative counts={counts}; sum_of_buckets={sum(counts)}")
print(f"classic linear p99: coarse={estimate_p99(coarse):.1f}s; fine={estimate_p99(fine):.1f}s")
print(f"at or below 0.3s={counts[1]}/{counts[-1]}")

The raw p99 is two seconds, but the coarse buckets estimate 4.2 seconds because the relevant observations are somewhere between one and five seconds. Adding a two-second boundary changes the estimate to 1.8 seconds. The narrower bucket improves the error in this sample, yet the estimate still differs from the exact raw percentile; extra resolution does not recreate the original values. Both estimates also match histogram_quantile in Prometheus 3.5.1 within a numerical tolerance of 1e-12.

The cumulative counts sum to 485 although only one hundred requests exist. Their disjoint increments sum to one hundred, and the 0.3-second bucket gives the exact threshold result of 95 out of 100 for this dataset. Read those outputs together: histogram counts can answer aligned threshold questions precisely while percentile interpolation remains approximate.

Sources

Prometheus: Histograms and Summaries, distribution representations, aggregation, and native histograms, checked 2026-09-13.
Prometheus: histogram_quantile, classic bucket interpolation and boundary cases, checked 2026-09-13.
Google SRE: Monitoring Distributed Systems, latency tails and meaningful monitoring, checked 2026-09-13.

Metrics · XB Field Notes

Prometheus rate estimates counter growth per second

Sampling determines what can be inferred

The selected range is part of the meaning

When it breaks

Check it quickly

Sources

Cardinality is a metric budget

Count combinations that actually exist

Include the metric representation

When it breaks

Check it quickly

Sources

Pushgateway retains batch metrics until their lifecycle is handled

Identify a stable batch outcome

Replacement is defined by the HTTP method

Scrape time is not last success time

When it breaks

Check it quickly

Sources

Prometheus histograms retain a bucketed distribution

Classic histograms are several related series

Aggregate buckets before estimating a quantile

When it breaks

Check it quickly

Sources

Put the unit in the metric name

Convert at a defined boundary

Unit and type answer different questions

When it breaks

Check it quickly

Sources

Histograms preserve the latency shape

Know what the buckets contain

Resolution determines the answer

When it breaks

Check it quickly

Sources