Prometheus · XB Field Notes

Annotations explain an alert without changing its identity

Tue, 22 Feb 2028 00:00:00 +0000

Alert labels identify which condition is being tracked and provide fields for routing and grouping. Annotations explain that condition: a current value, a concise summary, or a runbook link. Putting changing prose in labels can split one ongoing problem into many identities. Putting routing fields only in annotations can leave a route unable to recognize the alert’s intended owner.

Separate identity from the current explanation

A queue alert might keep alertname, service, and instance as stable labels while its summary changes from six waiting items to eight. Alertmanager can update the annotation for the same alert fingerprint rather than adding another alert. A fingerprint is derived from identity labels; it is not a hash of the rendered notification text.

This distinction also matters before Alertmanager receives the alert. Prometheus tracks its pending duration for an alert label set. A label containing a changing current value can keep creating new identities and prevent the intended waiting period from completing. Put that changing value in an annotation unless it deliberately defines a different alert condition. Review templates and rule labels together when diagnosing repeated pending alerts.

Notification templates receive several scopes of data. GroupLabels contains the labels chosen for grouping. CommonLabels contains label pairs common to all alerts in the notification. CommonAnnotations similarly contains annotations shared by the group. The Alerts list retains the labels and annotations of individual members, allowing a notification to explain their different states.

If two replicas have summaries describing different queue sizes, there is no shared summary to put in CommonAnnotations. A template that prints only that field can omit the very information an operator needs. Use common context for the heading, then iterate over individual alerts for changing summaries. A shared runbook link can remain common even when the individual values differ.

Do not assume a grouping label such as instance exists in every group. Grouping by service and alert name intentionally combines instances. A template should reflect the selected grouping contract and handle an absent common field deliberately. Representative fixtures need multiple alerts, unequal annotations, and the relevant firing or resolved states, not just one idealized alert where every field happens to be common.

When it breaks

A dynamic label creates a new fingerprint for every value change, increasing alert churn and potentially restarting pending state. A route matching team cannot use a team annotation in place of a label. An overly broad notification heading can also describe one instance while listing alerts from several services if grouping and template assumptions drift apart.

A template that parses successfully can still render an empty or misleading message for actual group data. Check the rendered text, not only syntax. Conversely, rendering successfully does not prove the integration accepts the message or that a notification was delivered. Keep credentials and sensitive request details out of annotations because they can be copied to every receiver selected by routing.

Check it quickly

Save this as annotations.py and run python3 annotations.py with Docker available. It uses Alertmanager 0.28.1, downloading the image if needed. The isolated server has a receiver with no delivery integrations. Two synthetic alerts are submitted through its actual API; one annotation is then updated without changing any labels.

import http.client
import io
import json
import subprocess
import time
import uuid

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


port = 9093
script = r"""set -eu
cat > /tmp/alertmanager.yml <<'ALERT_CONFIG'
route:
 receiver: preview
 group_by: [alertname, service]
 group_wait: 1h
 group_interval: 5m
 repeat_interval: 4h
receivers:
 - name: preview
ALERT_CONFIG
exec /bin/alertmanager --config.file=/tmp/alertmanager.yml --storage.path=/tmp/data \
 --web.listen-address=127.0.0.1:9093 --cluster.listen-address=
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: request("GET", "/-/ready")[0] == 200)
 alerts = [{"labels": {"alertname": "QueueHigh", "service": "checkout", "instance": instance},
 "annotations": {"summary": "queue=" + str(depth), "runbook": "https://example.invalid/queue"}}
 for instance, depth in [("api-a", 6), ("api-b", 9)]]
 assert request("POST", "/api/v2/alerts", json.dumps(alerts))[0] == 200
 before = eventually(lambda: (a if len(a := get_json("/api/v2/alerts")) == 2 else False))
 fingerprints = {a["labels"]["instance"]: a["fingerprint"] for a in before}
 alerts[0]["annotations"]["summary"] = "queue=8"
 assert request("POST", "/api/v2/alerts", json.dumps([alerts[0]]))[0] == 200
 def updated_annotations():
 items = get_json("/api/v2/alerts")
 changed = any(x["labels"]["instance"] == "api-a" and
 x["annotations"]["summary"] == "queue=8" for x in items)
 return items if changed else None
 updated = eventually(updated_annotations)
 assert len(updated) == 2
 assert {a["labels"]["instance"]: a["fingerprint"] for a in updated} == fingerprints
 # Supply the documented notification Data shape to the real template renderer.
 data = {"receiver": "preview", "status": "firing",
 "groupLabels": {"alertname": "QueueHigh", "service": "checkout"},
 "commonLabels": {"alertname": "QueueHigh", "service": "checkout"},
 "commonAnnotations": {"runbook": "https://example.invalid/queue"},
 "alerts": [{"status": "firing", "labels": a["labels"], "annotations": a["annotations"]}
 for a in sorted(updated, key=lambda a: a["labels"]["instance"])]}
 template = '{{define "preview"}}{{.GroupLabels.service}}|common-summary={{with .CommonAnnotations.summary}}{{.}}{{else}}none{{end}}{{range .Alerts}}\n{{.Labels.instance}}:{{.Annotations.summary}}{{end}}{{end}}'
 for path, content in [("/tmp/preview.tmpl", template), ("/tmp/data.json", json.dumps(data))]:
 subprocess.run(["docker", "exec", "-i", name, "/bin/sh", "-c", "cat > " + path],
 input=content, text=True, capture_output=True, check=True, timeout=5)
 rendered = subprocess.run(["docker", "exec", name, "/bin/amtool", "template", "render",
 "--template.glob=/tmp/preview.tmpl", "--template.data=/tmp/data.json",
 '--template.text={{ template "preview" . }}'],
 text=True, capture_output=True, check=True, timeout=5).stdout.strip()
 assert rendered == "checkout|common-summary=none\napi-a:queue=8\napi-b:queue=9", rendered
 print(json.dumps({"alert_count": 2, "fingerprints_unchanged": True, "rendered": rendered}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The result confirms that two alerts remain and their fingerprints are unchanged after the update. The program then supplies the documented notification data structure to the real amtool template renderer. Its common annotations intentionally include the shared runbook but exclude the unequal summaries. This explicit template input tests rendering, not the server’s notification assembly or delivery pipeline.

The rendered heading says common-summary=none, followed by api-a:queue=8 and api-b:queue=9. Assertions verify the updated value and the separate per-alert details. All HTTP traffic remains on container loopback, with no host mounts or published ports, and the container is removed on exit. A production notification still needs validation using its actual route, template data, and receiver integration.

Sources

Alertmanager: notification template reference, group labels, common annotations and individual alert data, checked 2026-09-13.
Prometheus: alerting rules, labels identify alerts while annotations carry explanatory content, checked 2026-09-13.
Alertmanager 0.28.1: README, local template rendering with amtool, checked 2026-09-13.

Recording rules store scheduled query results

Wed, 26 Jan 2028 00:00:00 +0000

A recording rule runs a PromQL expression at scheduled evaluation times and stores the result as a new metric. Repeated dashboard queries can then read that metric instead of recomputing the expression. Calling this a cache can obscure an important difference: the recorded series has its own sample history, cadence, and labels. It is not a transparent cache of every possible query request.

Three clocks influence the displayed value

Scraping determines when raw observations enter storage. Rule evaluation determines when an expression produces recorded samples. A dashboard chooses when and at what step to query either series. Refreshing a panel more frequently does not make a recording rule evaluate sooner. Reading the recorded metric between evaluations can therefore return an older value than evaluating the raw expression at that moment.

A rule group has an interval, defaulting to the global evaluation interval when not specified. Rules within a group run in order with the same evaluation timestamp. That permits a later rule to consume an earlier rule’s output. Separate groups do not provide that same sequencing guarantee merely because their interval strings match. Design dependencies explicitly rather than relying on coincidental execution order.

Precomputation changes the data you retain

Aggregation determines the dimensions of the output. If a rule sums away instance, dashboards cannot later recover an instance breakdown from that recorded series alone. Keep the raw metric when that investigation matters, and choose the recorded dimensions for a known consumer. Each remaining label combination creates another stored series, so recording every conceivable aggregation can increase ingestion and storage substantially.

Changing an expression affects subsequent evaluations. A normal rule reload does not recalculate all earlier samples under the new definition or fill history from before the rule existed. A semantic change under an unchanged metric name can place incompatible definitions on one graph. Document the definition and window, and use a deliberate migration when consumers require historical comparability.

When it breaks

A dashboard may appear frozen even though raw samples keep arriving. Compare the last recorded sample with the rule group’s last evaluation, duration, and errors. If a group is still evaluating when its next iteration is due, subsequent evaluations are skipped until the current one finishes or times out. Recording a costly query moves work into a scheduler; it does not make that work free.

A query can also be quick because it answers a narrower question after aggregation. Check that the removed dimensions were intentional before celebrating the speedup. A trivial expression rarely needs a recording rule solely for performance. Measure the repeated query cost and expected reuse, then account for the extra series, rule execution, and freshness delay in the decision.

Check it quickly

Save the program as recording-cadence.py and run python3 recording-cadence.py. It requires Docker and the public Prometheus 3.5.1 image, which Docker can fetch if missing. The real promtool engine receives queue values one minute apart. Both the unit-test evaluation step and the rule group’s interval are two minutes, deliberately separating observations from precomputation.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 2m
rule_files:
 - /tmp/rules.yml
tests:
 - interval: 1m
 input_series:
 - series: "queue_depth{queue=\"orders\"}"
 values: "1 4 9 16"
 promql_expr_test:
 - expr: "sum(queue_depth)"
 eval_time: 1m
 exp_samples:
 - labels: "{}"
 value: 4
 - expr: "queue:depth:sum"
 eval_time: 1m
 exp_samples:
 - labels: "{__name__=\"queue:depth:sum\"}"
 value: 1
 - expr: "queue:depth:sum"
 eval_time: 2m
 exp_samples:
 - labels: "{__name__=\"queue:depth:sum\"}"
 value: 9
"""
rules = r"""groups:
 - name: slow-recording
 interval: 2m
 rules:
 - record: queue:depth:sum
 expr: sum(queue_depth)
"""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
 script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

The expected output is SUCCESS. At minute one, the raw sum is four while the recorded value is still one from minute zero. At minute two, the recorded series becomes nine. The expression is intentionally cheap so the example isolates cadence; it is not a recommendation to precompute every queue gauge, and it does not benchmark a live rule scheduler.

The container has no external network, no host mounts, and a unique cleanup name. These assertions establish that recorded and on-demand values can legitimately differ between evaluation times. For an actual deployment, verify rule loading, execution duration, output cardinality, and consumer queries. A successful file reload alone does not establish that a new metric has accumulated the history needed by a long-range dashboard.

Sources

Prometheus: recording rules, stored evaluations, group intervals and missed iterations, checked 2026-09-13.
Prometheus: configuration, separate scrape and evaluation settings, checked 2026-09-13.
Prometheus: unit testing rules, test evaluation cadence and input intervals, checked 2026-09-13.

High-cardinality labels increase stored series before anyone queries them

Mon, 10 Jan 2028 00:00:00 +0000

A Prometheus series is identified by a metric name and its complete label set. Adding customer or request identity to a counter can create many series even when every dashboard immediately sums those identities away. Query aggregation changes the returned result; it does not remove the raw series that were ingested. Storage and ingestion costs arise before a user opens the dashboard.

Count observed combinations, not isolated values

One new label value is not automatically one new series across the entire database. It creates a new identity wherever the metric name and other labels form a distinct combination. Multiplying possible value counts gives a Cartesian upper bound for a given metric and target scope; it equals the observed count only when every combination is present. Dependencies between dimensions can make the bound much larger than reality. Actual series count also depends on target labels and metric families.

A route label based on a small set of route templates can be bounded. A full URL containing identifiers and query strings often is not. Customer identifiers, request IDs, arbitrary error messages and timestamps also need deliberate scrutiny. A histogram with multiple bucket series amplifies the impact of each label combination further. Budget from the actual exposition and expected workload rather than one dimension in isolation.

Querying fewer rows does not store fewer rows

An expression summing requests by route can return three results after reading twelve customer-specific series. Those twelve input series still exist. If route totals are the required metric contract, instrumentation can accumulate a route-level counter directly and export three series. That reduces the identities emitted at the source instead of relying on every consumer to hide the extra dimension.

Removing a label during ingestion is not equivalent to this accumulation. Distinct samples can acquire the same identity, and relabeling will not decide how to sum them. Prefer a deliberate instrumentation change or an explicitly defined recording query, understanding that recording adds derived series while retaining raw ones under the storage policy. Preserve detailed correlation in logs or traces when needed, with their own access, retention and cost limits.

Diagnose growth with the right inventory

The TSDB status API exposes head statistics, including series counts by metric name. Those values help identify large families but are not a complete cost model. Sample interval, label length, churn, chunk handling, retention and storage behavior also matter. Do not translate a small fixture’s series reduction into an identical percentage reduction in production memory or disk.

Instant query counts and head-series statistics can differ when some head series no longer have recent selectable samples. Historical blocks introduce another scope again. Record the measurement time and population when comparing reports. A release that changes label values may create churn while older identities remain in storage, so a graph returning fewer current results does not prove that all earlier cost has vanished.

When it breaks

A newly added unbounded label can increase ingestion and memory pressure until scrapes or queries suffer. Start by identifying the metric family and instrumentation change, then stop unnecessary new identities at their source or through a carefully reviewed sample filter. Immediately deleting historical data is not a universal recovery requirement, and changing retention does not remove every resource cost instantly.

A subtler failure is a successful aggregation that makes a dashboard look small while the raw metric continues growing. Review both exported identities and query output. Give new dimensions an expected value range, useful consumer and owner. For counters, preserve reset handling when creating production rate queries; a static sum used to compare fixture totals is not a replacement for rate-before-aggregation semantics.

Check it quickly

Save this as cardinality.py and run python3 cardinality.py. It requires Docker and Prometheus 3.5.1, downloading the image if needed. A local text endpoint exports three routes with four synthetic customer identities each, plus independently accumulated route totals. The real server scrapes those samples and exposes both query and TSDB statistics.

import http.client
import io
import json
import subprocess
import time
import uuid
from urllib.parse import quote

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


def query(expression):
 response = get_json("/api/v1/query?query=" + quote(expression, safe=""))
 assert response["status"] == "success"
 return response["data"]["result"]


def value(expression):
 result = query(expression)
 assert len(result) == 1, (expression, result)
 return float(result[0]["value"][1])

port = 9090
# Requires Docker and the Prometheus 3.5.1 image.
script = r"""set -eu
mkdir -p /tmp/www /tmp/targets
cat > /tmp/prom.yml <<'PROM_CONFIG'
global:
 scrape_interval: 1s
 scrape_timeout: 500ms
scrape_configs:
 - job_name: cardinality
 metrics_path: /metrics.txt
 static_configs:
 - targets: ['127.0.0.1:8080']

PROM_CONFIG
cat > /tmp/www/metrics.txt <<'METRIC_DATA'
# TYPE raw_requests_total counter
# TYPE bounded_requests_total counter
raw_requests_total{route="/orders",customer_id="synthetic-1"} 1
raw_requests_total{route="/orders",customer_id="synthetic-2"} 2
raw_requests_total{route="/orders",customer_id="synthetic-3"} 3
raw_requests_total{route="/orders",customer_id="synthetic-4"} 4
bounded_requests_total{route="/orders"} 10
raw_requests_total{route="/cart",customer_id="synthetic-1"} 1
raw_requests_total{route="/cart",customer_id="synthetic-2"} 2
raw_requests_total{route="/cart",customer_id="synthetic-3"} 3
raw_requests_total{route="/cart",customer_id="synthetic-4"} 4
bounded_requests_total{route="/cart"} 10
raw_requests_total{route="/search",customer_id="synthetic-1"} 1
raw_requests_total{route="/search",customer_id="synthetic-2"} 2
raw_requests_total{route="/search",customer_id="synthetic-3"} 3
raw_requests_total{route="/search",customer_id="synthetic-4"} 4
bounded_requests_total{route="/search"} 10

METRIC_DATA
/bin/httpd -f -p 127.0.0.1:8080 -h /tmp/www > /tmp/httpd.log 2>&1 &
exec /bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data --web.listen-address=127.0.0.1:9090
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: len(query('raw_requests_total')) == 12)
 assert value('count(raw_requests_total)') == 12
 assert value('count(bounded_requests_total)') == 3
 assert value('sum(raw_requests_total)') == value('sum(bounded_requests_total)') == 30
 raw_by_route = {x["metric"]["route"]: float(x["value"][1])
 for x in query('sum by (route) (raw_requests_total)')}
 bounded_by_route = {x["metric"]["route"]: float(x["value"][1])
 for x in query('bounded_requests_total')}
 assert raw_by_route == bounded_by_route
 statistics = get_json('/api/v1/status/tsdb')["data"]
 counts = {x["name"]: x["value"] for x in statistics["seriesCountByMetricName"]}
 assert counts["raw_requests_total"] == 12 and counts["bounded_requests_total"] == 3
 print(json.dumps({"raw_series": 12, "bounded_series": 3, "snapshot_total": 30,
 "per_route": raw_by_route, "tsdb_counts_confirmed": True}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The assertions confirm twelve raw series, three bounded series, and a total of thirty in both representations. Each route totals ten. After query aggregation, the TSDB still reports twelve raw series, demonstrating that a smaller result has not erased its inputs. The isolated container uses no external network, host mounts or published ports and is removed on exit. This validates identities and arithmetic, not a production capacity benchmark.

Sources

Prometheus: data model, series identity is the metric name plus its label set, checked 2026-09-13.
Prometheus: metric and label naming, bounded dimensions and high-cardinality label guidance, checked 2026-09-13.
Prometheus: storage, head data, blocks and retention costs, checked 2026-09-13.
Prometheus: HTTP API, head-series statistics by metric name, checked 2026-09-13.

Prometheus rate estimates counter growth per second

Fri, 24 Dec 2027 00:00:00 +0000

A request counter answers how many events an instrumented process has accumulated. A rate answers how quickly those events are arriving. Prometheus rate estimates average growth per second over a selected range, adjusts observed counter resets, and extrapolates toward the range boundaries. The result is useful precisely because it has a time unit, but it is not an exact reconstruction of every event between scrapes.

Sampling determines what can be inferred

An ordinary counter increases until it resets, commonly after a restart. A decrease between observed samples is treated as a reset by the counter functions. A restart and recovery entirely between two scrapes may be invisible if the new value already exceeds the previous sample. More frequent scraping improves observation resolution; no query can recover an unobserved history with certainty.

Apply rate to individual counter series before aggregating them. Summing first can let one instance’s growth hide another instance’s reset. Labels define series identity, so a replacement instance with different labels starts a different series rather than continuing the old one. A gauge such as queue depth can fall normally and should not be interpreted as a resetting counter.

The selected range is part of the meaning

A range selector is open at its left boundary and closed at its right boundary. At minute five, [5m] excludes a sample exactly at minute zero. The float-counter calculation needs at least two usable samples. A range containing one sample produces no rate value; it does not establish zero traffic. Leave enough room for the configured cadence and realistic missed scrapes.

Longer windows generally smooth short bursts and delay their disappearance from the average. Shorter windows respond sooner but offer fewer observations and can become empty. Choose the window from the event duration and detection objective, then label it explicitly. irate uses the last two samples, whereas rate uses the selected range. Both can be useful on the same metric if panels explain their different questions.

The query evaluation interval controls when a rule asks the question, not which samples were scraped. Refreshing a dashboard every second cannot create new observations from a target scraped every minute. Inspect these two cadences separately when assessing alert responsiveness.

When it breaks

A newly created series may vanish from a rate panel until a second sample arrives. Filling every missing result with zero can conceal collection failure. Check the raw series, timestamps, and scrape status before treating a blank panel as idle traffic. A very smooth long-window line can also hide a brief overload that matters to users.

Reset adjustment is not immunity to gaps. Missing samples can hide bursts or multiple restarts, and boundary extrapolation introduces estimation. increase applies the corresponding range estimate in event-count units; it may return a fractional count even for integer counters. Use application accounting records when exact totals are required, rather than promising that a dashboard reconstructs them.

Check it quickly

Save this program as rate.py and run python3 rate.py. Docker and the Prometheus 3.5.1 image are required; Docker can download the image if needed. The program invokes the real promtool engine against synthetic samples spaced one minute apart. The isolated container has no network access or host mounts and is removed on exit.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
tests:
 - interval: 1m
 input_series:
 - series: "demo_requests_total{instance=\"steady\"}"
 values: "0+60x5"
 - series: "demo_requests_total{instance=\"restart\"}"
 values: "100 160 220 20 80 140"
 - series: "demo_requests_total{instance=\"gaps\"}"
 values: "0 60 _ 180 _ 300"
 - series: "demo_requests_total{instance=\"one\"}"
 values: "_ _ _ _ _ 7"
 promql_expr_test:
 - expr: "rate(demo_requests_total{instance=\"steady\"}[5m])"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"steady\"}"
 value: 1
 - expr: "abs(rate(demo_requests_total{instance=\"restart\"}[5m]) - 5/6) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"restart\"}"
 value: 1
 - expr: "increase(demo_requests_total{instance=\"restart\"}[5m])"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"restart\"}"
 value: 250
 - expr: "irate(demo_requests_total{instance=\"restart\"}[5m])"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"restart\"}"
 value: 1
 - expr: "rate(demo_requests_total{instance=\"gaps\"}[5m])"
 eval_time: 5m
 exp_samples:
 - labels: "{instance=\"gaps\"}"
 value: 1
 - expr: "rate(demo_requests_total{instance=\"one\"}[5m])"
 eval_time: 5m
 exp_samples: []
"""
rules = r""""""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
 script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

Successful assertions print SUCCESS. The steady series grows at one request per second. For samples 100 160 220 20 80 140, the five-minute rate is 5/6, increase is 250, and irate is one. These differ because the selected range excludes the first sample, corrects the observed reset, and extrapolates its average across the requested duration.

The deliberately linear series with gaps still yields one per second; that controlled result does not prove arbitrary gaps harmless. The one-sample series yields an empty vector. The test checks values and absence separately, using numerical tolerance for the repeating fraction. Before deploying a rule, add representative restart, sparse-traffic, and missing-data cases from the service whose counter you will monitor.

Sources

Prometheus: query functions, rate, increase, irate and reset adjustment, checked 2026-09-13.
Prometheus: querying basics, range boundaries and sample selection, checked 2026-09-13.
Prometheus: unit testing rules, synthetic samples and expression assertions, checked 2026-09-13.

Alertmanager routes alerts through a label-matching tree

Wed, 08 Dec 2027 00:00:00 +0000

Alertmanager selects notification receivers by walking a routing tree against an alert’s labels. The tree expresses ownership and escalation policy, while receivers describe notification integrations. A configuration can parse correctly and still send the wrong class of alert to the wrong destination. Verify the label contract and tree behavior before treating a successful reload as evidence that routing works.

Follow the tree, including its stopping rule

Every alert enters the root route, which must match all alerts. Child routes inherit settings they do not override. If no child matches, the current node handles the alert. Otherwise, traversal normally stops after the first matching child. Setting continue: true on a matching child allows subsequent siblings to be considered. It does not mean that every receiver in the configuration receives the alert.

One route node names one receiver. A receiver can contain multiple notification integrations, and multiple matching routes can also select different receivers. These are different ways to deliver to several destinations. A route cannot simply replace its receiver field with a list and expect the same behavior. Review the chosen approach for duplicate notifications and intentionally different grouping or escalation settings.

Ownership is a label contract

A route matching team="payments" needs that exact label value on the alert entering Alertmanager. An annotation named team does not satisfy the matcher. Aggregation in a Prometheus expression can remove a needed dimension before the alert reaches routing. Inspect the final alert label set, including alert relabeling, rather than assuming the raw metric’s labels survived unchanged.

Keep the root receiver useful for alerts that miss more specific children. An unmatched alert still needs an observable destination and an owner who can correct its classification. Test absent labels as well as recognized teams. A missing team on a warning can take the root route, while a critical alert without that team can still match a separate severity route.

Route order is part of behavior. A broad early route can prevent a narrower later route from being reached unless continuation is enabled. Moving a block of YAML can therefore change recipients without changing any matcher text. Review ordering changes like policy changes, and keep representative label fixtures beside the configuration so the intended selection remains explicit.

When it breaks

A typo or a label renamed by an exporter can send alerts to the root receiver. A new catch-all child can capture alerts intended for a later team. Excessive continuation can create multiple notifications for one event. Diagnose these failures with the actual labels and the entire matching path, not by reading one route in isolation.

Correct receiver selection still does not establish delivery. Grouping delays, silence or inhibition, template errors, transport failures, and destination behavior can intervene afterward. Inspect routing first, then follow the selected receiver through the notification pipeline. Increasing the rule’s waiting duration will not fix an incorrect team matcher, and reducing grouping delay will not repair missing credentials at a receiver.

Check it quickly

Save this as routing.py and run python3 routing.py. It requires Docker and the public Alertmanager 0.28.1 image, which may be downloaded if absent. The real amtool validates the configuration and checks seven label cases. Receivers deliberately have names without delivery integrations, so the experiment evaluates routing without contacting any person or external service.

import subprocess
import uuid

# Requires Docker and Alertmanager 0.28.1. No receiver has a delivery integration.
script = r"""set -eu
cat > /tmp/routes.yml <<'ROUTES'
route:
 receiver: default
 group_by: [alertname, service]
 routes:
 - receiver: payments
 matchers: ['team="payments"']
 continue: true
 - receiver: pager
 matchers: ['severity="critical"']
receivers:
 - name: default
 - name: payments
 - name: pager

ROUTES
/bin/amtool check-config /tmp/routes.yml
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=payments,pager team=payments severity=critical
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=payments team=payments severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=pager team=data severity=critical
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default team=data severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default team=payments-preview severity=warning
sed 's/continue: true/continue: false/' /tmp/routes.yml > /tmp/stop.yml
/bin/amtool config routes test --config.file=/tmp/stop.yml --verify.receivers=payments team=payments severity=critical
"""
name = "xbcatm-route-test-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-s"],
 input=script, text=True, capture_output=True, timeout=30,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 raise RuntimeError(result.stderr or result.stdout)
print("Verified seven route cases, including continuation and unmatched labels.")

The first case resolves to payments,pager: the payments route matches and continues, then the critical route also matches. A payments warning selects only payments; a critical data-team alert selects pager. Unmatched and missing team values select the root receiver. The payments-preview value does not match the exact payments value.

The final check changes continuation to false and confirms that the payments critical alert now selects only payments. Each expected receiver set is enforced by --verify.receivers, so a mismatch fails execution. The container has no external network or host mounts and is removed on exit. Notification delivery requires a separate controlled test of the real receiver integration.

Sources

Alertmanager: configuration, route inheritance, sibling continuation and receiver definitions, checked 2026-09-13.
Alertmanager 0.28.1: README, amtool configuration and routing checks, checked 2026-09-13.
Prometheus: Alertmanager, separation of routing, grouping and suppression, checked 2026-09-13.

Prometheus rules track alert state before notification delivery

Wed, 01 Sep 2027 00:00:00 +0000

An alerting rule evaluates a PromQL expression and tracks the returned label sets over time. Prometheus decides whether each resulting alert is pending or firing. Alertmanager receives alert information and handles notification routing, grouping, silencing, and inhibition. A reliable investigation follows those stages separately: a firing alert is evidence of rule state, not proof that a person received a message.

Vector membership activates the alert

Each element returned by the expression represents an active alert candidate. A comparison such as queue_depth > 5 filters out values that do not meet the threshold. Adding the bool modifier changes comparison behavior: false comparisons remain as elements with value zero. An alert rule tests the presence of elements, so using that modifier can activate an alert even when the number is zero.

Labels identify alert instances. Keep stable routing dimensions, such as service, queue, and severity, in labels. Put changing explanatory values in annotations. If a label includes the current queue size, every change can create a different identity and restart the pending period. An annotation can include that size without redefining which alert is being tracked.

Waiting is measured at evaluations

With for: 2m, a newly active label set is pending until it has remained active for two minutes of evaluated time. A rule evaluated every minute first seeing the condition at minute one can fire at minute three. The engine observes evaluations, not continuous physical reality between samples. The timing therefore includes the wait until the next evaluation as well as the configured duration.

Omitting for permits an active expression to fire on its first evaluation. That may suit an unequivocal failure, but it also makes a single bad sample sufficient; choose deliberately rather than relying on a forgotten default.

If the expression stops returning that label set before firing, pending state clears. With the default behavior used here, a firing alert also clears when its condition disappears. An explicitly configured keep_firing_for can extend firing after the condition clears; that option is not present in this example. Do not describe all alerts as latched incidents that require manual acknowledgement to resolve.

When it breaks

A threshold over a missing series usually returns nothing. That can clear an alert even though the application has stopped exporting data. Decide whether absence needs its own rule rather than blindly treating it as recovery. Test label changes and missing samples alongside threshold transitions. A syntax check alone cannot reveal these semantic failures.

A firing alert can produce no visible page because a silence, an inhibition rule, routing, grouping delay, or delivery failure intervened. These mechanisms belong to the notification path and do not change Prometheus’s threshold evaluation. Inspect the active alert labels and then the matching Alertmanager route and delivery evidence. Increasing for does not repair a wrong receiver or a missing route label.

Check it quickly

Save this as alert-state.py and run python3 alert-state.py with Docker available. It uses Prometheus 3.5.1; Docker may fetch the image if it is not cached. Real promtool evaluates the rule against queue values 0 6 6 6 0 6 0 0, one sample per minute. No Alertmanager or external notification destination is involved.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
rule_files:
 - /tmp/rules.yml
tests:
 - interval: 1m
 input_series:
 - series: "queue_depth{queue=\"orders\"}"
 values: "0 6 6 6 0 6 0 0"
 promql_expr_test:
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\",alertstate=\"pending\"}) or vector(0)"
 eval_time: 1m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\",alertstate=\"firing\"}) or vector(0)"
 eval_time: 2m
 exp_samples:
 - labels: "{}"
 value: 0
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\",alertstate=\"firing\"}) or vector(0)"
 eval_time: 3m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\"}) or vector(0)"
 eval_time: 4m
 exp_samples:
 - labels: "{}"
 value: 0
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\",alertstate=\"pending\"}) or vector(0)"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\"}) or vector(0)"
 eval_time: 6m
 exp_samples:
 - labels: "{}"
 value: 0
"""
rules = r"""groups:
 - name: queue-alerts
 interval: 1m
 rules:
 - alert: QueueBacklog
 expr: queue_depth{queue="orders"} > 5
 for: 2m
 labels:
 severity: page
 annotations:
 summary: Orders queue remains above five
"""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
 script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

The expected output is SUCCESS. At minute one the alert is pending, at minute two it is not firing, and at minute three it fires. At minute four the condition recovers and no active ALERTS element remains. A second spike starts pending at minute five but clears at minute six, so it never reaches firing.

The assertions inspect the synthetic ALERTS series; their or vector(0) only makes absence countable inside the test. It is not part of the production alert expression. The disposable container has no network access or host mounts. This verifies state transitions for the supplied timeline. Receiver matching, grouping delays, and actual message delivery require separate tests with the intended Alertmanager configuration.

Sources

Prometheus: alerting rules, active label sets, for and synthetic ALERTS series, checked 2026-09-13.
Prometheus: unit testing rules, evaluation timelines and assertion fixtures, checked 2026-09-13.
Prometheus: Alertmanager, notification grouping, routing, silences and inhibition, checked 2026-09-13.

Recording rules should preserve the components of a ratio

Tue, 31 Aug 2027 00:00:00 +0000

Repeated service-level queries are good candidates for recording rules when their cost and reuse justify precomputation. The design question is what to store. Saving only an instance’s error percentage can discard the traffic volume needed for later aggregation. Saving error rates and request rates preserves the numerator and denominator, allowing consumers to calculate a correctly weighted service fraction.

A percentage needs its population

Suppose a busy instance handles ten requests per second with 0.1 errors per second. A quiet instance handles one request per second with 0.5 errors per second. Their individual error fractions are one percent and fifty percent. Averaging those fractions gives 25.5 percent, which weights instances equally even though their request volumes differ by a factor of ten.

The service sees eleven requests and 0.6 errors per second. Its error fraction is 0.6/11, approximately 5.45 percent. This is the ratio of aggregate errors to aggregate requests. Both measurements must describe the same eligible traffic population and time window. A faster dashboard is still wrong if its numerator excludes failures that its denominator includes, or the two sides aggregate different services.

Build the rule dependency in order

Start with rate on each counter series so observed resets are handled before aggregation. Then sum by the dimensions consumers actually need. The example first records request rates by service and instance, then service request rates, service error rates, and finally the service error fraction. The intermediate instance rate is included to demonstrate an explicit dependency consumed by a later rule.

All four rules share a group and run in declaration order with the same evaluation time. Matching service labels on numerator and denominator make their division pair the intended series. A denominator retaining an extra instance label would change vector matching and could remove results or require a different join. Test the actual output labels as well as the numeric answer.

Keep names descriptive about aggregation level and range. A two-minute rate and a thirty-minute rate are different quantities even when their current values happen to match. Record the eligible event definition beside the rule and identify which dashboards or alerts consume it. Changing the formula under a stable name changes future samples, not the meaning of already stored history.

When it breaks

An average of percentages can make a small failing replica dominate a fleet report or conceal a busy failing replica. Increasing the evaluation frequency does not repair that arithmetic. Recover the underlying counts or rates and divide after aggregation. Once only percentages remain, the original traffic weights generally cannot be reconstructed from those percentages alone.

Zero traffic is another distinct case. A zero denominator does not define a zero error fraction, and a missing numerator may mean absent instrumentation rather than no errors. Decide those semantics explicitly with the metric contract. Avoid automatically filling every empty or undefined result with a healthy zero. Validate that errors are exported consistently, including during healthy operation.

Check it quickly

Save this as recording-weighted.py and run python3 recording-weighted.py. Docker and the Prometheus 3.5.1 image are required; the image may be downloaded on first use. Real promtool evaluates four recording rules over synthetic minute-spaced counter samples. The fixture deliberately gives its two instances very different traffic weights and error fractions.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
rule_files:
 - /tmp/rules.yml
tests:
 - interval: 1m
 input_series:
 - series: "http_requests_total{service=\"checkout\",instance=\"busy\"}"
 values: "0+600x5"
 - series: "http_errors_total{service=\"checkout\",instance=\"busy\"}"
 values: "0+6x5"
 - series: "http_requests_total{service=\"checkout\",instance=\"quiet\"}"
 values: "0+60x5"
 - series: "http_errors_total{service=\"checkout\",instance=\"quiet\"}"
 values: "0+30x5"
 promql_expr_test:
 - expr: "service:http_requests:rate2m"
 eval_time: 5m
 exp_samples:
 - labels: "{__name__=\"service:http_requests:rate2m\",service=\"checkout\"}"
 value: 11
 - expr: "abs(service:http_errors:rate2m - 0.6) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{service=\"checkout\"}"
 value: 1
 - expr: "abs(service:http_errors:ratio_rate2m - 0.6/11) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{service=\"checkout\"}"
 value: 1
 - expr: "abs(avg(rate(http_errors_total[2m]) / rate(http_requests_total[2m])) - 0.255) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
"""
rules = r"""groups:
 - name: service-rates
 interval: 1m
 rules:
 - record: instance:http_requests:rate2m
 expr: sum by (service, instance) (rate(http_requests_total[2m]))
 - record: service:http_requests:rate2m
 expr: sum by (service) (instance:http_requests:rate2m)
 - record: service:http_errors:rate2m
 expr: sum by (service) (rate(http_errors_total[2m]))
 - record: service:http_errors:ratio_rate2m
 expr: service:http_errors:rate2m / service:http_requests:rate2m
"""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
 script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

The expected output is SUCCESS. The recorded request rate is eleven, the error rate is 0.6, and the resulting fraction is 0.6/11. A separate assertion demonstrates that the incorrect average of instance fractions is 0.255. Numerical comparisons use a small tolerance where floating-point division is involved, and label assertions retain the checkout service identity.

The disposable container has no external network or host mounts. This is an arithmetic and dependency test, not a performance benchmark or proof of production instrumentation completeness. Before deploying the rules, add empty-traffic, counter-reset, and missing-series cases reflecting the actual service. Measure query cost and rule execution time to establish whether precomputation delivers the intended operational benefit.

Sources

Prometheus: recording rules, ordered rule dependencies and stored output, checked 2026-09-13.
Prometheus: query functions, rate before aggregation for counter resets, checked 2026-09-13.
Prometheus: unit testing rules, testing recorded series and PromQL results, checked 2026-09-13.

Prometheus pulls metrics and records scrape outcomes

Mon, 30 Aug 2027 00:00:00 +0000

Prometheus collects metrics by requesting an HTTP endpoint on each configured target. The target exposes observations; the collector chooses when to scrape, parses the response, and stores samples. This arrangement makes collection failures observable from the collector’s perspective. It does not guarantee that the endpoint is reachable, that every expected target was discovered, or that the application’s user journeys work.

A job contains independently scraped instances

A job groups targets with a shared purpose, such as several API replicas. Each target is an instance with its own scrape outcome. The usual job and instance labels identify that context, although relabeling can change the final labels. Service discovery supplies candidates; scrape configuration determines their endpoint, protocol, interval, timeout, and authentication. Discovery and collection therefore have separate failure modes.

The collector must be able to connect to the target. An exporter needs an inbound path from authorized collectors, even though it does not initiate metric uploads to them. Network policy, TLS identity, credentials, and routing remain deployment responsibilities. A pull design changes who initiates the connection; it does not remove network boundaries or make arbitrary exporters safe to expose publicly.

Read the generated metric precisely

For a configured instance, Prometheus generates up, with value one for a successful scrape and zero for a failed scrape. It is a metric, not an up label. Scrape duration and sample counts provide additional collection evidence. An exporter can return a valid response while the database behind it is unavailable, or while an important business metric is missing. Such a scrape can still succeed.

Use application outcomes and exporter-specific dependency metrics to answer those further questions. Conversely, up=0 describes a collection failure, which might be caused by the monitoring path while users remain unaffected. Keep that distinction in alert wording so responders investigate the correct surface instead of assuming every collection problem is a customer outage.

When it breaks

A connection refusal, timeout, certificate error, or invalid exposition can prevent a scrape from succeeding. Inspect the target’s last scrape error before changing the interval. A slower interval does not repair a wrong port, and a larger timeout can conceal a slow exporter without removing its cost. Review emitted series and response size when collection becomes expensive. Keep scrape timeouts within the configured interval, and evaluate exporter load before shortening the schedule. Faster polling can multiply database queries performed by an exporter without making those queries cheaper.

A target removed from discovery is different from a configured target returning zero. Its absence can eventually remove the corresponding series from instant queries. Therefore, a fleet-wide sum of up cannot prove that every expected instance still exists. Compare discovery inventory with the intended deployment when completeness matters. Likewise, a successful /metrics request is not a substitute for testing a checkout or login operation.

Check it quickly

Save this as pull.py and run python3 pull.py with Docker available. It uses prom/prometheus:v3.5.1; an uncached image requires an initial registry download. One disposable container scrapes its own loopback endpoint and an unused loopback port. No host port is published, and the container has no external network access.

import json
import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
# All HTTP traffic stays on loopback inside one disposable container.
script = r"""set -eu
cat > /tmp/prom.yml <<'CONFIG'
global:
 scrape_interval: 1s
 scrape_timeout: 500ms
scrape_configs:
 - job_name: self
 static_configs:
 - targets: ['127.0.0.1:9090']
 - job_name: refused
 static_configs:
 - targets: ['127.0.0.1:1']
CONFIG
/bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data \
 --web.listen-address=127.0.0.1:9090 > /tmp/prom.log 2>&1 &
for attempt in $(seq 1 30); do
 if wget -qO /tmp/ready.json 'http://127.0.0.1:9090/api/v1/query?query=count%28up%29%20%3D%3D%202%20and%20sum%28up%29%20%3D%3D%201'; then
 if grep -q '"status":"success"' /tmp/ready.json && ! grep -q '"result":\[\]' /tmp/ready.json; then
 wget -qO- 'http://127.0.0.1:9090/api/v1/query?query=up'
 exit 0
 fi
 fi
 sleep 1
done
cat /tmp/prom.log >&2
exit 1
"""
name = "xbcatm-prom-live-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if result.returncode:
 raise RuntimeError(result.stderr or result.stdout)
payload = json.loads(result.stdout)
assert payload["status"] == "success"
values = {item["metric"]["job"]: float(item["value"][1])
 for item in payload["data"]["result"]}
assert values == {"self": 1.0, "refused": 0.0}, values
print(json.dumps({"up_by_job": values, "scrape_reachability_only": True}, indent=2))

The result must contain self: 1.0 and refused: 0.0. Assertions check actual query results after both targets have been evaluated; the startup loop avoids interpreting an initially empty database as a failure. The final cleanup removes only the uniquely named test container. This verifies collection outcomes, not business availability or discovery completeness.

Turn the outcome into a useful investigation

In a real incident, identify the failing instance, read its scrape error, and test connectivity from the collector’s network context. Compare the response with the expected exporter format and check its dependency status. Keep secrets out of copied diagnostic output. Once collection recovers, confirm both up and the specific business series used by dashboards. Restoring one does not automatically establish the other.

Sources

Prometheus: jobs and instances, generated scrape metrics and target identity, checked 2026-09-13.
Prometheus: configuration, scrape intervals, timeouts and target configuration, checked 2026-09-13.
Prometheus: HTTP API, instant query response format, checked 2026-09-13.

Metric relabeling filters samples before ingestion

Wed, 04 Aug 2027 00:00:00 +0000

Prometheus has several relabeling stages, and their similar syntax can hide different effects. Target relabeling transforms discovered targets before collection. Metric relabeling transforms scraped samples just before ingestion. A drop action at the first stage can prevent an endpoint from being scraped; the same action at the second stage discards matching samples from an endpoint that was already contacted.

Decide whether to remove data or remove a label

In metric_relabel_configs, action: drop matches combined source-label values and discards the matching sample. Using __name__ as the source lets a rule reject selected metric families. It does not mean remove the named source label while retaining the sample. A broad expression against a common value can therefore discard much more data than intended.

action: labeldrop instead applies its regular expression to label names and removes matching labels from retained samples. labelkeep retains only names that match. Neither operation sums values or calculates a rollup. The distinction is between removing observations and changing their identity, so choose the operation based on the intended data contract rather than the similar action names.

Relabeling expressions are anchored at both ends. A replacement or drop rule needs the correct source labels and matching pattern at its particular stage. Discovery metadata usually belongs to target relabeling and is no longer available as __meta_ input during sample processing. Copy required context into ordinary target labels earlier instead of expecting an internal discovery label to accompany every exported metric.

Preserve uniqueness after removing dimensions

A time series is identified by its metric name and full label set. Removing a label that distinguishes two observations can collapse their identities, causing duplicate-sample conflicts or lost distinctions. Prometheus does not infer whether their values should be summed, averaged or selected. Prefer bounded instrumentation and explicit query aggregation when the real goal is a service total.

The example removes a build label only where each remaining route identity is already unique. That is a controlled precondition, not permission to remove any expensive label from any exporter. Check representative exposition for collisions before deploying a rule. Also inspect dashboards and alerts that reference the removed label; a successful scrape does not guarantee those consumers still select the intended data.

When it breaks

A filtering rule placed under target relabeling can remove the entire endpoint instead of one costly metric. Conversely, a metric filter will not save the exporter from generating the response or prevent its bytes from crossing the network. It reduces retained samples at ingestion. If collection itself is expensive, address instrumentation or exporter behavior as well.

Automatically generated scrape series such as up are not processed by metric relabeling. A rule attempting to drop up there will not hide the generated metric. A healthy up can coexist with an intentionally discarded business metric, so data absence must be interpreted with the configured filters in mind. Changing filters affects future ingestion and does not rewrite already stored historical labels.

Check it quickly

Save this as metric-relabel.py and run python3 metric-relabel.py. It requires Docker and Prometheus 3.5.1; Docker may fetch the image if needed. The same static text endpoint is scraped under raw and filtered jobs. Its .txt path provides a text content type; startup waits for actual samples rather than assuming an immediately populated target list.

import http.client
import io
import json
import subprocess
import time
import uuid
from urllib.parse import quote

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


def query(expression):
 response = get_json("/api/v1/query?query=" + quote(expression, safe=""))
 assert response["status"] == "success"
 return response["data"]["result"]


def value(expression):
 result = query(expression)
 assert len(result) == 1, (expression, result)
 return float(result[0]["value"][1])

port = 9090
# Requires Docker and the Prometheus 3.5.1 image.
script = r"""set -eu
mkdir -p /tmp/www /tmp/targets
cat > /tmp/prom.yml <<'PROM_CONFIG'
global:
 scrape_interval: 1s
 scrape_timeout: 500ms
scrape_configs:
 - job_name: raw
 metrics_path: /metrics.txt
 static_configs:
 - targets: ['127.0.0.1:8080']
 - job_name: filtered
 metrics_path: /metrics.txt
 static_configs:
 - targets: ['127.0.0.1:8080']
 metric_relabel_configs:
 - source_labels: [__name__]
 regex: 'debug_.+|up'
 action: drop
 - regex: build_sha
 action: labeldrop

PROM_CONFIG
cat > /tmp/www/metrics.txt <<'METRIC_DATA'
# TYPE demo_requests_total counter
demo_requests_total{route="/orders",build_sha="release-a"} 42
# TYPE debug_alloc_bytes gauge
debug_alloc_bytes{kind="scratch"} 9

METRIC_DATA
/bin/httpd -f -p 127.0.0.1:8080 -h /tmp/www > /tmp/httpd.log 2>&1 &
exec /bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data --web.listen-address=127.0.0.1:9090
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: len(query('demo_requests_total')) == 2)
 raw = query('demo_requests_total{job="raw"}')[0]
 filtered = query('demo_requests_total{job="filtered"}')[0]
 assert raw["metric"]["build_sha"] == "release-a"
 assert "build_sha" not in filtered["metric"]
 assert float(raw["value"][1]) == float(filtered["value"][1]) == 42
 assert len(query('debug_alloc_bytes{job="raw"}')) == 1
 assert query('debug_alloc_bytes{job="filtered"}') == []
 assert value('up{job="filtered"}') == 1
 print(json.dumps({"raw_labels": raw["metric"], "filtered_labels": filtered["metric"],
 "debug_series_dropped": True, "generated_up_preserved": True}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The raw request series retains build_sha="release-a"; the filtered one loses that label while retaining its route and value of 42. The debug metric exists in the raw job and is absent in the filtered job. Although the drop pattern also names up, the filtered job’s generated up remains one. Assertions check all these differences through the actual query API.

The two jobs have distinct job labels, and this fixture has no competing request series after label removal. It verifies stage semantics, not a general collision-free migration. All traffic stays on container loopback without host mounts or published ports, and cleanup removes only the test container. Validate real exporter identities and affected consumers before applying an equivalent production filter.

Sources

Prometheus: configuration, metric relabeling, drop, labeldrop and generated-series exclusions, checked 2026-09-13.
Prometheus: data model, metric names and complete label sets define series identity, checked 2026-09-13.
Prometheus: HTTP API, querying the retained metric labels and values, checked 2026-09-13.

Alertmanager grouping batches alerts while silences suppress notifications

Tue, 27 Jul 2027 00:00:00 +0000

One incident can produce many alerts, but a group is not proof that those alerts have one cause. Alertmanager groups alerts according to configured labels and routing context so their notifications can be handled together. The original alert identities remain available inside the group. Silences add another operation: they suppress matching notifications for a time window without resolving the underlying conditions.

A group key determines which details stay together

With group_by: [alertname, service], two QueueHigh alerts for the checkout service can share a group even when their instance labels differ. A QueueHigh alert for search belongs to another group. A DiskFull alert for checkout also belongs to another group because its alert name differs. Grouping therefore depends on the selected label values, not on all alerts having an identical fingerprint.

Adding instance to the group key creates more groups and may undo the desired batching across replicas. Omitting service can combine unrelated owners or workloads. Choose the key around the investigation and ownership model, then inspect the member list to retain individual context. The special grouping value ... groups by all labels and effectively disables aggregation into broader notification groups.

Routing context matters too. Alerts routed to different receivers are not necessarily one notification merely because their selected grouping labels match. Test the combination of route and group configuration. A receiver responsible for a whole service may benefit from batching several instances, while another integration can require a different grouping contract. Neither setting automatically determines whether the alerts share a root cause.

Waiting, changes and repetition use different timers

group_wait delays the first notification for a new group so nearby arrivals can join it. An alert that resolves before that initial wait ends may never generate a notification. group_interval governs checks for subsequent group updates, including newly firing or resolved members. It is not simply a promise to resend every still-open group at that frequency.

repeat_interval controls reminders when the group’s notification content has not otherwise changed. Its eligibility is checked on the group interval, so choose a compatible multiple instead of assuming an independent precise alarm clock. Delivery processing and failures can add further delay. Set these values from the response objective, and keep the rule’s pending duration separate from notification scheduling.

When it breaks

Grouping too narrowly turns a replica outage into many notifications; grouping too broadly can hide which service needs attention. A dashboard showing one group does not mean only one alert remains. Expand the members and inspect the group key before treating a lower notification count as evidence that the incident has improved.

A silence with overly broad matchers can suppress alerts beyond the planned maintenance scope. Use a bounded time window and inspect which active alerts match it. Silence expiration restores notification eligibility; it does not fix the application or guarantee an immediate message. Inhibition is a separate configured relationship where an active source alert can suppress matching target notifications, rather than a time-bounded manual silence.

Check it quickly

Save this as grouping.py and run python3 grouping.py with Docker available. It uses Alertmanager 0.28.1, fetching the image if needed. The actual API receives four synthetic alerts. The receiver has no delivery integrations and the initial notification wait is deliberately long, so the experiment observes grouping and suppression state without sending messages.

import http.client
import io
import json
import subprocess
import time
import uuid

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


from datetime import datetime, timedelta, timezone

port = 9093
script = r"""set -eu
cat > /tmp/alertmanager.yml <<'ALERT_CONFIG'
route:
 receiver: preview
 group_by: [alertname, service]
 group_wait: 1h
 group_interval: 5m
 repeat_interval: 4h
receivers:
 - name: preview
ALERT_CONFIG
exec /bin/alertmanager --config.file=/tmp/alertmanager.yml --storage.path=/tmp/data \
 --web.listen-address=127.0.0.1:9093 --cluster.listen-address=
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: request("GET", "/-/ready")[0] == 200)
 labels = [
 {"alertname": "QueueHigh", "service": "checkout", "instance": "api-a"},
 {"alertname": "QueueHigh", "service": "checkout", "instance": "api-b"},
 {"alertname": "QueueHigh", "service": "search", "instance": "api-c"},
 {"alertname": "DiskFull", "service": "checkout", "instance": "api-a"},
 ]
 assert request("POST", "/api/v2/alerts", json.dumps([{"labels": x} for x in labels]))[0] == 200
 groups = eventually(lambda: (groups if len(groups := get_json("/api/v2/alerts/groups")) == 3 else False))
 sizes = {g["labels"]["service"] + "/" + g["labels"]["alertname"]: len(g["alerts"]) for g in groups}
 assert sizes == {"checkout/QueueHigh": 2, "checkout/DiskFull": 1, "search/QueueHigh": 1}
 now = datetime.now(timezone.utc)
 silence = {"matchers": [{"name": "service", "value": "checkout", "isRegex": False, "isEqual": True}],
 "startsAt": (now - timedelta(seconds=1)).isoformat(),
 "endsAt": (now + timedelta(minutes=10)).isoformat(),
 "createdBy": "local-fixture", "comment": "Disposable grouping test"}
 status, body = request("POST", "/api/v2/silences", json.dumps(silence))
 assert status == 200
 silence_id = json.loads(body)["silenceID"]
 def silenced_state():
 alerts = get_json("/api/v2/alerts")
 count = sum(silence_id in x["status"]["silencedBy"] for x in alerts)
 return alerts if count == 3 else None
 alerts = eventually(silenced_state)
 assert len(alerts) == 4
 assert all((silence_id in x["status"]["silencedBy"]) == (x["labels"]["service"] == "checkout") for x in alerts)
 assert request("DELETE", "/api/v2/silence/" + silence_id)[0] == 200
 def unsilenced_state():
 alerts = get_json("/api/v2/alerts")
 return len(alerts) == 4 and all(silence_id not in x["status"]["silencedBy"] for x in alerts)
 eventually(unsilenced_state)
 print(json.dumps({"group_sizes": sizes, "silenced_alerts": 3,
 "alerts_remain_visible": 4, "silence_expired": True}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The output must report group sizes of two for checkout/QueueHigh, one for checkout/DiskFull, and one for search/QueueHigh. A temporary silence matching checkout suppresses three alerts, while all four remain visible through the API. The search alert stays outside that silence. The program then expires the silence and verifies that its identifier no longer appears in the alerts’ suppression lists.

All requests stay on container loopback, with no host ports or mounts; cleanup removes the unique container. These assertions test membership and silence lifecycle. They do not measure notification timer accuracy, validate inhibition, or establish downstream delivery. Those behaviors require tests using the actual routing, scheduling and receiver configuration.

Sources

Alertmanager: configuration, group keys, notification intervals and inhibition rules, checked 2026-09-13.
Prometheus: Alertmanager, grouping and silence semantics, checked 2026-09-13.
Alertmanager 0.28.1: README, API v2 and local Alertmanager operation, checked 2026-09-13.

External labels identify outgoing Prometheus data

Wed, 09 Jun 2027 00:00:00 +0000

When several Prometheus servers send data to a common system, downstream readers need to know which collector or environment supplied it. Global external_labels can add that context during communication with external systems. They do not rewrite every locally stored series, and they do not override labels already present on the outgoing series. Understanding both boundaries prevents misleading configuration checks.

Local storage and outgoing identity differ

A local query of up generally shows the labels created by scraping and relabeling. An external label such as prom_region need not appear there. Its absence from that query does not prove the configuration failed. Compare the appropriate outgoing interface with the local result; federation is a direct way to observe this difference without configuring a remote storage backend.

The configuration contract also covers communication such as remote storage and alerting, but each downstream path has its own relabeling and processing. Inspect the actual consumer when validating that path. A locally evaluated recording rule does not acquire every external label merely because it stores a new series. External labels are added at the relevant communication boundary, not as a universal local query transformation.

Existing labels take precedence

If a target series already has cluster="target-cluster", configuring an external cluster="collector-cluster" does not replace that value. External labels supplement missing label names. This is defined behavior, not a promise that newer versions always reject collisions or log an error. A local metric can therefore export a different cluster value from the one an operator expected the global setting to enforce.

Reserve distinct collector label names when that distinction matters, and check exporter and relabeling conventions for collisions. For federation, the destination’s honor_labels setting determines how it handles source labels that conflict with its own scrape labels. That destination behavior is a separate issue from how the source adds external labels to its federation response.

Stable labels do not provide deduplication

Changing a global label value changes the identity of subsequently exported series. Previously stored downstream history remains under its old labels; it is not rewritten. Update selectors and dashboards with that historical boundary in mind. Adding many labels is not automatically useful: a constant label adds context, while different values across collectors create additional distinctions consumers must handle.

Two collectors scraping the same targets can export duplicate observations under different replica labels. A naive sum across replicas can double apparent traffic. External labels make that distinction expressible but do not configure the downstream system’s deduplication. Nor are labels a tenant authorization boundary. Verify the storage system’s actual isolation and query behavior when those requirements apply.

When it breaks

A team may look for an external label in a local query, find none, and repeatedly change a working configuration. The opposite mistake is seeing the desired label in a federation response and assuming remote write or alert routing has also been tested. Check the specific path and final labels, including any downstream relabeling, instead of using one observation as evidence for all integrations.

Check it quickly

Save this as external-labels.py and run python3 external-labels.py. It requires Docker and the Prometheus 3.5.1 image; Docker may download an uncached image. One disposable Prometheus scrapes itself, then the program compares its local query response with its text federation output. All HTTP requests stay on container loopback, with no host ports or external network.

import json
import re
import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
# All HTTP traffic stays on loopback inside one disposable container.
script = r"""set -eu
cat > /tmp/prom.yml <<'CONFIG'
global:
 scrape_interval: 1s
 scrape_timeout: 500ms
 external_labels:
 cluster: collector-cluster
 prom_region: lab-region
 replica: one
scrape_configs:
 - job_name: self
 static_configs:
 - targets: ['127.0.0.1:9090']
 labels:
 cluster: target-cluster
CONFIG
/bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data \
 --web.listen-address=127.0.0.1:9090 > /tmp/prom.log 2>&1 &
for attempt in $(seq 1 30); do
 if wget -qO /tmp/ready.json 'http://127.0.0.1:9090/api/v1/query?query=up%20%3D%3D%201'; then
 if grep -q '"status":"success"' /tmp/ready.json && ! grep -q '"result":\[\]' /tmp/ready.json; then
 wget -qO /tmp/local.json 'http://127.0.0.1:9090/api/v1/query?query=up'
 wget -qO /tmp/federated.txt --header='Accept: text/plain; version=0.0.4' \
 'http://127.0.0.1:9090/federate?match%5B%5D=up'
 cat /tmp/local.json
 printf '\n'
 cat /tmp/federated.txt
 exit 0
 fi
 fi
 sleep 1
done
cat /tmp/prom.log >&2
exit 1
"""
name = "xbcatm-prom-live-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if result.returncode:
 raise RuntimeError(result.stderr or result.stdout)
local_json, _, exported = result.stdout.partition("\n")
payload = json.loads(local_json)
assert payload["status"] == "success"
local = next(a["metric"] for a in payload["data"]["result"]
 if a["metric"]["job"] == "self")
line = next(line for line in exported.splitlines() if line.startswith("up{"))
# The fixture's controlled label values contain no quotes or escape sequences.
outbound = dict(re.findall(r'(\w+)="([^"]*)"', line))
assert outbound["prom_region"] == "lab-region" and outbound["replica"] == "one"
assert outbound["cluster"] == local["cluster"] == "target-cluster"
assert "prom_region" not in local and "replica" not in local
print(json.dumps({"federated_labels": outbound, "local_labels": local}, indent=2))

The local labels contain cluster="target-cluster" but no prom_region or replica. Federation adds prom_region="lab-region" and replica="one", while the existing cluster value remains unchanged. Assertions verify all three facts. The parser is deliberately limited to the fixture’s controlled label values, which contain no quotes or escape sequences; it is not a general exposition parser.

The uniquely named container is removed on exit. This establishes source-side federation behavior and the local/outgoing distinction. It does not validate a federation destination, remote write receiver, high-availability deduplication, or notification delivery. Use the corresponding downstream interface and configuration to complete those separate integration checks.

Sources

Prometheus: configuration, external labels supplement labels during outbound communication, checked 2026-09-13.
Prometheus: federation, federation selectors and preservation of upstream labels, checked 2026-09-13.
Prometheus: HTTP API, local query results used for comparison, checked 2026-09-13.

Pushgateway retains batch metrics until their lifecycle is handled

Thu, 13 May 2027 00:00:00 +0000

A short-lived job can finish before Prometheus next scrapes it. Pushgateway offers an intermediary that retains pushed metrics for later scraping, but it is not the default answer for every short process. Prometheus recommends it primarily for suitable service-level batch outcomes. The retained data has a lifecycle independent of the job, so successful scraping does not prove the job recently ran or succeeded.

Identify a stable batch outcome

A service-level job, such as a nightly reconciliation, can publish its last result under a stable grouping key. Putting a new run identifier, timestamp, or machine identity in that key creates additional groups instead of updating one result. Pushgateway has no automatic per-group TTL that removes obsolete runs. Owners must replace or delete the relevant data deliberately.

Machine-specific batch results can often use a node exporter’s textfile collector instead. A gateway also introduces a shared availability dependency. Its up value describes whether Prometheus can scrape the gateway, not whether each original pusher is healthy. Choose the collection model around the monitored event and ownership of cleanup, rather than treating pushing as a universal way around discovery or network policy.

Replacement is defined by the HTTP method

The URL path defines a grouping key, including the job and any additional label pairs. HTTP PUT replaces the metrics belonging to that exact group. POST replaces metric families with the names included in the request while preserving other families in the group. POST does not mean increment the previous value or append an event to a durable log.

DELETE removes the exact group identified by its URL. Deleting a job-only grouping key does not delete every group that also has that job plus additional labels. The gateway’s documented method ordering allows a later successful push to follow an earlier queued deletion. This matters when verifying cleanup instead of assuming that an accepted deletion has already affected every matching-looking series.

Scrape time is not last success time

The gateway exports push_time_seconds for the last successful push to a group. Reading the metrics endpoint does not advance that value. Prometheus assigns scrape timestamps to the samples it collects, so recently scraped samples can still describe a very old job result. Publish a separate last-success timestamp as a gauge value when the job’s successful completion time is the actual question.

Do not attach an explicit sample timestamp to pushed exposition: Pushgateway rejects those timestamps. A Unix timestamp stored as a gauge’s value is different and is supported. A successful push also establishes neither business success nor immediate durable persistence. A job can report failure successfully, and a gateway without persistence can lose held data when it restarts.

When it breaks

A failed or retired job can leave its old success metric available indefinitely. An alert checking only that the series exists will miss the missing execution. Compare the last-success value with the expected schedule, and keep job removal synchronized with group cleanup. A changing grouping label can make old and new runs appear side by side forever.

Using POST when a complete snapshot was intended can retain a metric the new run no longer publishes. Using PUT with only one metric can erase other results needed by dashboards. Review which component owns the entire group and which method it uses. There is no --replace option that substitutes for these HTTP semantics.

Check it quickly

Save this as pushgateway.py and run python3 pushgateway.py. Docker and the Pushgateway 1.11.1 image are required; an uncached image may be downloaded. The disposable server accepts requests only through container loopback. No host port is published, no host directory is mounted, and cleanup removes only the uniquely named test container.

import http.client
import io
import json
import subprocess
import time
import uuid

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


port = 9091
script = r"""exec /bin/pushgateway --web.listen-address=127.0.0.1:9091
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/pushgateway:v1.11.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: request("GET", "/metrics")[0] == 200)
 group = "/metrics/job/nightly_sync/team/data"
 first = "# TYPE items_processed gauge\nitems_processed 7\n# TYPE last_success_unixtime gauge\nlast_success_unixtime 1234\n"
 assert request("PUT", group, first, "text/plain; version=0.0.4")[0] == 200
 def snapshot():
 status, body = request("GET", "/metrics")
 assert status == 200
 return {line.split("{", 1)[0]: float(line.rsplit(" ", 1)[1])
 for line in body.splitlines()
 if not line.startswith("#") and 'job="nightly_sync"' in line}
 def first_snapshot():
 data = snapshot()
 return data if data.get("items_processed") == 7 else None
 initial = eventually(first_snapshot)
 assert initial["last_success_unixtime"] == 1234
 time.sleep(0.2)
 assert snapshot()["push_time_seconds"] == initial["push_time_seconds"]
 partial = "# TYPE items_processed gauge\nitems_processed 9\n"
 assert request("POST", group, partial, "text/plain; version=0.0.4")[0] == 200
 after_post = snapshot()
 assert after_post["items_processed"] == 9 and after_post["last_success_unixtime"] == 1234
 replacement = "# TYPE items_processed gauge\nitems_processed 11\n"
 assert request("PUT", group, replacement, "text/plain; version=0.0.4")[0] == 200
 after_put = snapshot()
 assert after_put["items_processed"] == 11 and "last_success_unixtime" not in after_put
 assert request("DELETE", "/metrics/job/nightly_sync")[0] == 202
 # A successful later POST orders the preceding queued deletion before this check.
 assert request("POST", group, "", "text/plain; version=0.0.4")[0] == 200
 assert snapshot()["items_processed"] == 11
 assert request("DELETE", group)[0] == 202
 eventually(lambda: not snapshot())
 print(json.dumps({"initial_items": 7, "post_items": 9, "put_items": 11,
 "post_preserved_other_metric": True, "put_removed_other_metric": True,
 "scrapes_did_not_refresh_push_time": True,
 "parent_group_delete_did_not_match": True, "group_deleted": True}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

Assertions show seven processed items after the initial PUT, nine after POST while the separate last-success gauge remains, and eleven after a replacement PUT that removes that other gauge. Repeated reads leave push time unchanged. Deleting the job-only group leaves the team-qualified group intact; deleting the exact group removes its metrics. The fixture tests gateway API behavior, not job scheduling, Prometheus integration or durable recovery.

Sources

Prometheus: when to use the Pushgateway, limited service-level batch use cases and retained series, checked 2026-09-13.
Pushgateway 1.11.1: README, PUT, POST, DELETE, grouping keys and push timestamps, checked 2026-09-13.

Target relabeling selects and labels targets before scraping

Tue, 06 Apr 2027 00:00:00 +0000

Service discovery produces candidate targets and metadata. Target relabeling transforms those candidates before Prometheus scrapes them. It can keep or drop targets, change how they are addressed, and create labels carried into collected series. This is a different stage from rewriting individual scraped samples. Understanding the stage prevents a configuration intended to remove one metric from accidentally removing an entire target.

Inspect discovery input before defining the rule

Discovered targets include internal labels such as __address__ and discovery-specific __meta_ fields. For file-based discovery, __meta_filepath identifies the file that supplied a target. Those metadata labels are available during target relabeling, but internal labels are removed from the final target label set after that stage. Copy the useful value into an ordinary label if downstream queries need it.

The exact metadata names depend on the discovery mechanism. A rule copied from Kubernetes discovery will not automatically work against file discovery. Missing source labels contribute empty values when relabeling combines its inputs, so a keep rule matching an unavailable field can discard every candidate. Inspect the actual discovered labels through the target API or discovery view before diagnosing a network outage.

Order determines the resulting identity

Relabeling steps run in their configured order. A keep action retains a target only when the combined source-label value matches its regular expression; a drop action removes matching targets. The regular expressions are anchored at both ends. A replace action can copy captured text into a destination label, and that new value is available to later rules in the same sequence.

Copying a value does not inherently delete the original label. If removal is needed, it is a separate operation and must preserve the identity required by the scrape and consumers. Changing __address__ changes the endpoint Prometheus will contact, while changing an ordinary label changes context. Neither action should be described merely as cosmetic cleanup of dashboard names.

Target labels can affect every series collected from the target. Choose stable fields such as a service or meaningful deployment scope rather than copying every discovery annotation. A frequently changing build identifier creates a new series identity across many metrics. A small discovery configuration change can therefore affect a much larger stored dataset and the queries that reference it.

When it breaks

A target excluded by relabeling is not a configured target with up=0. It is not scraped, so a failed-scrape query may never report it. When an expected service disappears, compare the discovered candidates, dropped targets and active targets. An incorrect keep rule can look like an empty healthy fleet if the alert assumes all expected members still have up series.

Copying the wrong metadata can also produce convincing but incorrect ownership labels. Query results may look tidy while representing the wrong service scope. Review rule order, capture groups, and final labels together. Broad label removal can merge previously distinct identities; target relabeling does not calculate an aggregate to compensate for the lost distinction.

Check it quickly

Save this as target-relabel.py and run python3 target-relabel.py. Docker and Prometheus 3.5.1 are required; the image may be downloaded if missing. One isolated container hosts a static text metrics endpoint and a file-discovery document containing an API target plus a discard target. The file endpoint uses /metrics.txt so its HTTP response has a text content type accepted by Prometheus.

import http.client
import io
import json
import subprocess
import time
import uuid
from urllib.parse import quote

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


def query(expression):
 response = get_json("/api/v1/query?query=" + quote(expression, safe=""))
 assert response["status"] == "success"
 return response["data"]["result"]


def value(expression):
 result = query(expression)
 assert len(result) == 1, (expression, result)
 return float(result[0]["value"][1])

port = 9090
# Requires Docker and the Prometheus 3.5.1 image.
script = r"""set -eu
mkdir -p /tmp/www /tmp/targets
cat > /tmp/prom.yml <<'PROM_CONFIG'
global:
 scrape_interval: 1s
 scrape_timeout: 500ms
scrape_configs:
 - job_name: discovered
 metrics_path: /metrics.txt
 file_sd_configs:
 - files: ['/tmp/targets/*.json']
 relabel_configs:
 - source_labels: [role]
 regex: api
 action: keep
 - source_labels: [__meta_filepath]
 regex: '.*/([^/]+)[.]json'
 target_label: discovery_file
 replacement: '$1'

PROM_CONFIG
cat > /tmp/www/metrics.txt <<'METRIC_DATA'
# TYPE demo_requests_total counter
demo_requests_total{route="/orders",build_sha="release-a"} 42
# TYPE debug_alloc_bytes gauge
debug_alloc_bytes{kind="scratch"} 9

METRIC_DATA
cat > /tmp/targets/lab.json <<'TARGETS'
[{"targets": ["127.0.0.1:8080"], "labels": {"role": "api"}}, {"targets": ["127.0.0.1:1"], "labels": {"role": "discard"}}]
TARGETS
/bin/httpd -f -p 127.0.0.1:8080 -h /tmp/www > /tmp/httpd.log 2>&1 &
exec /bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data --web.listen-address=127.0.0.1:9090
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: value('up{job="discovered"}') == 1)
 targets = get_json('/api/v1/targets?state=any')["data"]
 assert len(targets["activeTargets"]) == 1 and len(targets["droppedTargets"]) == 1
 active = targets["activeTargets"][0]
 dropped = targets["droppedTargets"][0]
 assert active["discoveredLabels"]["__meta_filepath"] == "/tmp/targets/lab.json"
 assert dropped["discoveredLabels"]["role"] == "discard"
 labels = query('up{job="discovered"}')[0]["metric"]
 assert labels["discovery_file"] == "lab" and labels["role"] == "api"
 assert not any(label.startswith("__meta_") for label in labels)
 print(json.dumps({"active": 1, "dropped": 1, "stored_labels": labels,
 "source_file": active["discoveredLabels"]["__meta_filepath"]}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The program waits for actual collection, then checks one active target and one dropped target. The active target’s discovered metadata contains /tmp/targets/lab.json. Its stored up series has role="api" and discovery_file="lab", with no surviving __meta_ label. The discard target is visible in the dropped-target evidence and was not treated as a failing scrape.

All requests stay on container loopback, with no external network, host mounts or published ports. Cleanup removes the uniquely named container. These assertions test real file discovery, filtering, metadata copying and final labels. They do not validate another discovery provider’s metadata or prove that a production inventory contains every intended service.

Sources

Prometheus: configuration, ordered target relabeling and file-discovery metadata, checked 2026-09-13.
Prometheus: jobs and instances, target identity and generated scrape metrics, checked 2026-09-13.
Prometheus: HTTP API, active and dropped target inspection, checked 2026-09-13.

Prometheus staleness and alert waiting are different mechanisms

Wed, 03 Feb 2027 00:00:00 +0000

A missing metric is not a numeric zero. Prometheus distinguishes a series with a recent value, a series marked stale, and a selector with no usable sample. Alerting then applies its own evaluation and waiting rules. Treating all these mechanisms as one universal grace period produces alerts that miss disappearing targets or report an outage without establishing what actually disappeared.

A marker is not a five-minute timer

An instant selector normally uses the newest eligible sample within its lookback period, which defaults to five minutes. A stale marker tells the engine to stop returning that series in instant selections after the marker. It does not require the previous value to remain visible for another five minutes. Historical samples are still stored and can remain relevant to range queries.

Collection circumstances matter. A series no longer exported, a target removed from discovery, and an exporter assigning its own timestamps need not follow identical timing. Explicit timestamps and staleness configuration can alter behavior. Determine which path your exporter uses before predicting disappearance time from the scrape interval alone. A configurable lookback is a query selection rule, not a universal service outage detector.

Choose the absence question deliberately

absent(selector) returns a value of one when the selector has no elements. It returns an empty vector when matching data exists, even if that data’s numeric value is zero. absent_over_time(selector[3m]) asks whether the selected range contains any samples. A real sample from before a stale marker can keep that range-based absence expression empty until the sample leaves the window.

The alert rule’s for duration is another layer: the expression must keep returning the relevant alert label set at successive evaluations before it fires. It does not configure staleness or extend the query’s lookback. Combining a three-minute absence window with a waiting duration delays notification further, with evaluation cadence and collection timing also contributing. Choose that delay from the detection objective.

When it breaks

absent(up{job="api"}) is not a failed-scrape alert. If the configured targets are present with up=0, the selector still has elements, so absence is false. A comparison against zero addresses that case. If a target disappears from discovery altogether, checking only zero can miss it once its series stops appearing. Monitor expected membership separately when that distinction matters.

A broad selector also cannot enumerate missing members. If one replica survives, absent over the whole job remains empty even when several expected replicas are gone. Match a deliberately known identity or compare observed series with an authoritative inventory. A label rename can look like an outage under the old matcher while the process continues working under a new identity.

Check it quickly

Save the following as stale.py and run python3 stale.py with Docker available. The program uses the public Prometheus 3.5.1 image and real promtool queries. Docker may download an uncached image. Synthetic samples include an explicit stale marker, missing positions, and ordinary zeros; this is a query-semantics test, not a measurement of live discovery timing.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
tests:
 - interval: 1m
 input_series:
 - series: "up{job=\"api\",instance=\"gone\"}"
 values: "1 0 0 0 0"
 - series: "service_events_total{job=\"api\",instance=\"gone\"}"
 values: "10 stale _ _ _"
 - series: "service_events_total{job=\"api\",instance=\"zero\"}"
 values: "0 0 0 0 0"
 promql_expr_test:
 - expr: "service_events_total{job=\"api\",instance=\"gone\"}"
 eval_time: 1m
 exp_samples: []
 - expr: "up{job=\"api\",instance=\"gone\"}"
 eval_time: 1m
 exp_samples:
 - labels: "{__name__=\"up\",job=\"api\",instance=\"gone\"}"
 value: 0
 - expr: "absent(service_events_total{job=\"api\",instance=\"gone\"})"
 eval_time: 1m
 exp_samples:
 - labels: "{job=\"api\",instance=\"gone\"}"
 value: 1
 - expr: "absent_over_time(service_events_total{job=\"api\",instance=\"gone\"}[3m])"
 eval_time: 2m
 exp_samples: []
 - expr: "absent_over_time(service_events_total{job=\"api\",instance=\"gone\"}[3m])"
 eval_time: 4m
 exp_samples:
 - labels: "{job=\"api\",instance=\"gone\"}"
 value: 1
 - expr: "absent(service_events_total{job=\"api\",instance=\"zero\"})"
 eval_time: 4m
 exp_samples: []
 - expr: "absent(up{job=\"api\"})"
 eval_time: 4m
 exp_samples: []
"""
rules = r""""""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
 script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

The test prints SUCCESS. At minute one, the marked business series is absent from an instant selection, while its separately supplied up series exists with value zero. Instant absent returns one. At minute two, the three-minute range still contains the original business sample, so absent_over_time returns nothing; at minute four it returns one.

The continuously exported zero-valued business series is never absent, and the job-wide up selector is not absent either. These assertions distinguish missing values from failing values instead of converting both to zero. The container has no network or host mounts and is removed on exit. For a production rule, supplement these cases with expected target membership and an independently tested pending-to-firing timeline.

Sources

Prometheus: querying basics, instant selectors, lookback and staleness, checked 2026-09-13.
Prometheus: query functions, absent and absent_over_time behavior, checked 2026-09-13.
Prometheus: unit testing rules, missing and stale sample notation, checked 2026-09-13.

Prometheus histograms retain a bucketed distribution

Wed, 06 Jan 2027 00:00:00 +0000

A latency mean can conceal a slow subset of requests. A histogram retains more distribution information by counting observations into buckets, but it still discards detail within each bucket. Prometheus can estimate quantiles from those counts without storing every observation. The estimate depends on bucket boundaries and aggregation; it is not a recovery of the complete original latency distribution.

A classic histogram exports cumulative _bucket counters, a _count, and a _sum. The le label identifies each inclusive bucket upper bound. An observation of 1.5 seconds increments the two-second bucket and every larger bucket, so bucket values must not be summed as though they were disjoint counts. The positive-infinity bucket includes all observations and corresponds to the count.

For nonnegative latency observations, applying rate to the sum and count gives total seconds accumulated per second and observations per second. Their ratio is mean latency in seconds. Bucket rates describe the distribution over the selected time range. Apply the counter rate before aggregating instances so each series’ observed reset is handled independently.

Aggregate buckets before estimating a quantile

For one fleet percentile, sum the bucket rates while retaining le, then call histogram_quantile. Keep any grouping dimensions required by the question, such as service. Compatible bucket boundaries and matching event definitions are necessary for meaningful aggregation. Removing le destroys the bucket identity the classic quantile function needs.

Do not average instance percentiles. A percentile is a position in a distribution, not an additive count, and even equal request volumes do not make percentile averaging valid. The combined bucket curve can have a very different shape from either instance. If the question is the fraction meeting a two-second objective, a bucket exactly at two seconds divided by the total count answers that threshold question directly.

A summary may export client-computed quantiles. Those values cannot be merged into a fleet quantile by summing or averaging them. Its count and sum can still contribute to a combined mean when event definitions match. Select the metric type from the aggregation you need, not solely from a panel’s preferred display.

When it breaks

A wide bucket near the desired percentile can make the estimate look more precise than the measurements justify. Classic quantile calculation interpolates within the selected bucket; it does not know where the individual observations occurred inside it. Keep boundaries useful around service objectives, and review them before a rollout that mixes different schemas across replicas. Refining future buckets cannot recover detail already discarded from historical observations.

A missing bucket or exporter is also different from zero observations. An apparently better percentile can result from losing the slow instance’s metrics. Check scrape completeness, contributing series, and sample counts before interpreting improvement. Sparse or absent observations can yield no useful quantile; replacing such results with zero latency creates a misleading success signal.

Check it quickly

Save the program as histogram-query.py and run python3 histogram-query.py. It requires Docker and the Prometheus 3.5.1 image, which may be fetched initially. Real promtool evaluates classic histogram queries against minute-spaced synthetic counters. Each instance contributes one hundred observations per minute, but their bucket shapes and accumulated latency differ substantially.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
tests:
 - interval: 1m
 input_series:
 - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"1\"}"
 values: "0+95x5"
 - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"2\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"5\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_bucket{instance=\"fast\",le=\"+Inf\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_count{instance=\"fast\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_sum{instance=\"fast\"}"
 values: "0+17x5"
 - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"1\"}"
 values: "0+0x5"
 - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"2\"}"
 values: "0+0x5"
 - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"5\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_bucket{instance=\"slow\",le=\"+Inf\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_count{instance=\"slow\"}"
 values: "0+100x5"
 - series: "api_duration_seconds_sum{instance=\"slow\"}"
 values: "0+400x5"
 promql_expr_test:
 - expr: "abs(histogram_quantile(0.99, sum by (le) (rate(api_duration_seconds_bucket[2m]))) - 4.94) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "abs(avg(histogram_quantile(0.99, rate(api_duration_seconds_bucket[2m]))) - 3.385) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "sum(rate(api_duration_seconds_bucket{le=\"2\"}[2m])) / sum(rate(api_duration_seconds_count[2m]))"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 0.5
 - expr: "abs(sum(rate(api_duration_seconds_sum[2m])) / sum(rate(api_duration_seconds_count[2m])) - 2.085) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
"""
rules = r""""""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
 script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

The expected output is SUCCESS. The correctly combined p99 estimate is 4.94 seconds. Averaging the two instance p99 estimates produces 3.385 seconds, which is not the fleet p99. Half the observations meet the two-second threshold, and the combined mean is 2.085 seconds. Numerical assertions allow a tiny floating-point tolerance rather than requiring an identical binary representation of decimal fractions.

The fixture validates PromQL arithmetic and aggregation, not actual request instrumentation or the distribution within a bucket. It uses no external network or host mounts and removes its disposable container. Current Prometheus guidance also discusses native histograms, which encode buckets differently and can simplify queries when the instrumentation and storage path support them. This example deliberately targets classic histograms: retain le, aggregate their counter rates correctly, and describe the returned percentile as an estimate.

Sources

Prometheus: histograms and summaries, bucketed observations, aggregation and metric choices, checked 2026-09-13.
Prometheus: query functions, classic histogram_quantile interpolation and rate ordering, checked 2026-09-13.
Prometheus: unit testing rules, executing expected PromQL values, checked 2026-09-13.

Put the unit in the metric name

Fri, 30 Oct 2026 00:00:00 +0000

A metric named request_duration leaves readers guessing whether 250 means seconds, milliseconds, or something else. A Prometheus-facing name such as http_request_duration_seconds carries that contract into queries, recording rules, and alerts. The contract still needs a correct implementation. Appending _seconds to measurements collected in milliseconds changes the spelling, not the magnitude, and can create a thousandfold monitoring error.

Convert at a defined boundary

Choose one quantity and one unit for each metric family. Prometheus recommends base units such as seconds and bytes, with a unit suffix where applicable. A duration and an absolute timestamp can both use seconds, but they are different quantities: last_success_timestamp_seconds should not be combined with request_duration_seconds. Describe the measurement and boundary in its help text so that a useful name has an equally clear definition.

If a library reports milliseconds, convert once before observation or export. Preserve sufficient precision instead of truncating a short duration to zero. Configure the dashboard to display the base unit appropriately; a formatter may show 0.250 seconds as 250 milliseconds without changing stored samples. Do not add another conversion in a recording rule because the panel happens to use milliseconds on its axis.

Unit and type answer different questions

An accumulating byte count conventionally ends in _bytes_total; a current memory usage gauge ends in _bytes. Both measure bytes, but only the former represents accumulated work and is suitable for a counter rate. Counts of requests conventionally use _total without inventing a physical unit. Histogram families add their own component suffixes, such as _seconds_bucket, _seconds_sum, and _seconds_count, to the duration family name.

A ratio commonly represents a fraction between zero and one; percentages use a different numeric scale. State which one a panel expects. A counter may reset when its process restarts, and Prometheus counter-rate functions account for decreases within the observed series. An additional _created series is not required for that behavior. Apply rate calculations before aggregating independent counters so that their individual resets remain visible.

When it breaks

A correctly named metric can still contain incorrect values. Mixing seconds and milliseconds under the same identity contaminates history, so changing only a label or help string cannot repair old samples. Treat a unit migration as a contract change: update the producer, queries, alerts, and dashboards together, and keep historical comparisons explicit. A new identity may be appropriate when the old series has an incompatible meaning.

The suffix also cannot enforce counter behavior. A gauge named _total remains a bad input to a counter-rate interpretation if it moves down during ordinary operation. Conversely, dividing an already calculated per-second rate by the interval again invents a different quantity. These are dimensional and semantic errors, not display problems. Other telemetry conventions may carry units in metadata rather than names; follow the documented exporter mapping instead of renaming every upstream instrument blindly.

Check it quickly

Save the following as units.py and run Python 3. It converts three synthetic latency measurements and emits a small Prometheus text exposition with help and type declarations. Decimal arithmetic makes the millisecond-to-second conversion explicit. The comparison demonstrates why a renamed but unconverted producer cannot share the same latency threshold.

from decimal import Decimal

latency_ms = [Decimal('120'), Decimal('250'), Decimal('630')]
latency_seconds = [value / Decimal('1000') for value in latency_ms]
threshold_seconds = Decimal('0.300')
correct_good = sum(value <= threshold_seconds for value in latency_seconds)
rename_only_good = sum(value <= threshold_seconds for value in latency_ms)
transferred_bytes = 1024 * 1024
used_ratio = Decimal('25') / Decimal('100')

assert correct_good == 2 and rename_only_good == 0
assert sum(latency_seconds) == Decimal('1.000')
assert used_ratio == Decimal('0.25')
print(f"# within 0.300s: converted={correct_good}/3, rename_only={rename_only_good}/3")
print('# HELP demo_last_request_duration_seconds Duration of the most recent completed request.')
print('# TYPE demo_last_request_duration_seconds gauge')
print(f'demo_last_request_duration_seconds {latency_seconds[-1]:.3f}')
print('# HELP demo_transferred_bytes_total Bytes transferred since this process started.')
print('# TYPE demo_transferred_bytes_total counter')
print(f'demo_transferred_bytes_total {transferred_bytes}')
print('# HELP demo_capacity_used_ratio Fraction of configured capacity currently used.')
print('# TYPE demo_capacity_used_ratio gauge')
print(f'demo_capacity_used_ratio {used_ratio}')

The first comment reports two of three observations within 0.300 seconds after conversion, compared with zero of three when millisecond values are merely treated as seconds. The exported last-request gauge is 0.630 seconds, the byte counter is 1,048,576, and the capacity ratio is 0.25, corresponding to 25% in a percentage display.

The complete output passed promtool check metrics using Prometheus 3.5.1. That parser check establishes valid exposition and naming lint for this sample; it cannot infer whether real instrumentation measured the promised quantity. Verify a known input through collection, export, query, and display before trusting a renamed production metric.

Sources

Prometheus: Metric and Label Naming, base units, suffixes, and conventions at the Prometheus boundary, checked 2026-09-13.
Prometheus: Data Model, metric identity and labels, checked 2026-09-13.
Prometheus: Query Functions, counter rates and reset handling, checked 2026-09-13.

Prometheus · XB Field Notes

Annotations explain an alert without changing its identity

Separate identity from the current explanation

A group does not necessarily share its summary

When it breaks

Check it quickly

Sources

Recording rules store scheduled query results

Three clocks influence the displayed value

Precomputation changes the data you retain

When it breaks

Check it quickly

Sources

High-cardinality labels increase stored series before anyone queries them

Count observed combinations, not isolated values

Querying fewer rows does not store fewer rows

Diagnose growth with the right inventory

When it breaks

Check it quickly

Sources

Prometheus rate estimates counter growth per second

Sampling determines what can be inferred

The selected range is part of the meaning

When it breaks

Check it quickly

Sources

Alertmanager routes alerts through a label-matching tree

Follow the tree, including its stopping rule

Ownership is a label contract

When it breaks

Check it quickly

Sources

Prometheus rules track alert state before notification delivery

Vector membership activates the alert

Waiting is measured at evaluations

When it breaks

Check it quickly

Sources

Recording rules should preserve the components of a ratio

A percentage needs its population

Build the rule dependency in order

When it breaks

Check it quickly

Sources

Prometheus pulls metrics and records scrape outcomes

A job contains independently scraped instances

Read the generated metric precisely

When it breaks

Check it quickly

Turn the outcome into a useful investigation

Sources

Metric relabeling filters samples before ingestion

Decide whether to remove data or remove a label

Preserve uniqueness after removing dimensions

When it breaks

Check it quickly

Sources

Alertmanager grouping batches alerts while silences suppress notifications

A group key determines which details stay together

Waiting, changes and repetition use different timers

When it breaks

Check it quickly

Sources

External labels identify outgoing Prometheus data

Local storage and outgoing identity differ

Existing labels take precedence

Stable labels do not provide deduplication

When it breaks

Check it quickly

Sources

Pushgateway retains batch metrics until their lifecycle is handled

Identify a stable batch outcome

Replacement is defined by the HTTP method

Scrape time is not last success time

When it breaks

Check it quickly

Sources

Target relabeling selects and labels targets before scraping

Inspect discovery input before defining the rule

Order determines the resulting identity