Alerting · XB Field Notes

Annotations explain an alert without changing its identity

Tue, 22 Feb 2028 00:00:00 +0000

Alert labels identify which condition is being tracked and provide fields for routing and grouping. Annotations explain that condition: a current value, a concise summary, or a runbook link. Putting changing prose in labels can split one ongoing problem into many identities. Putting routing fields only in annotations can leave a route unable to recognize the alert’s intended owner.

Separate identity from the current explanation

A queue alert might keep alertname, service, and instance as stable labels while its summary changes from six waiting items to eight. Alertmanager can update the annotation for the same alert fingerprint rather than adding another alert. A fingerprint is derived from identity labels; it is not a hash of the rendered notification text.

This distinction also matters before Alertmanager receives the alert. Prometheus tracks its pending duration for an alert label set. A label containing a changing current value can keep creating new identities and prevent the intended waiting period from completing. Put that changing value in an annotation unless it deliberately defines a different alert condition. Review templates and rule labels together when diagnosing repeated pending alerts.

Notification templates receive several scopes of data. GroupLabels contains the labels chosen for grouping. CommonLabels contains label pairs common to all alerts in the notification. CommonAnnotations similarly contains annotations shared by the group. The Alerts list retains the labels and annotations of individual members, allowing a notification to explain their different states.

If two replicas have summaries describing different queue sizes, there is no shared summary to put in CommonAnnotations. A template that prints only that field can omit the very information an operator needs. Use common context for the heading, then iterate over individual alerts for changing summaries. A shared runbook link can remain common even when the individual values differ.

Do not assume a grouping label such as instance exists in every group. Grouping by service and alert name intentionally combines instances. A template should reflect the selected grouping contract and handle an absent common field deliberately. Representative fixtures need multiple alerts, unequal annotations, and the relevant firing or resolved states, not just one idealized alert where every field happens to be common.

When it breaks

A dynamic label creates a new fingerprint for every value change, increasing alert churn and potentially restarting pending state. A route matching team cannot use a team annotation in place of a label. An overly broad notification heading can also describe one instance while listing alerts from several services if grouping and template assumptions drift apart.

A template that parses successfully can still render an empty or misleading message for actual group data. Check the rendered text, not only syntax. Conversely, rendering successfully does not prove the integration accepts the message or that a notification was delivered. Keep credentials and sensitive request details out of annotations because they can be copied to every receiver selected by routing.

Check it quickly

Save this as annotations.py and run python3 annotations.py with Docker available. It uses Alertmanager 0.28.1, downloading the image if needed. The isolated server has a receiver with no delivery integrations. Two synthetic alerts are submitted through its actual API; one annotation is then updated without changing any labels.

import http.client
import io
import json
import subprocess
import time
import uuid

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


port = 9093
script = r"""set -eu
cat > /tmp/alertmanager.yml <<'ALERT_CONFIG'
route:
 receiver: preview
 group_by: [alertname, service]
 group_wait: 1h
 group_interval: 5m
 repeat_interval: 4h
receivers:
 - name: preview
ALERT_CONFIG
exec /bin/alertmanager --config.file=/tmp/alertmanager.yml --storage.path=/tmp/data \
 --web.listen-address=127.0.0.1:9093 --cluster.listen-address=
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: request("GET", "/-/ready")[0] == 200)
 alerts = [{"labels": {"alertname": "QueueHigh", "service": "checkout", "instance": instance},
 "annotations": {"summary": "queue=" + str(depth), "runbook": "https://example.invalid/queue"}}
 for instance, depth in [("api-a", 6), ("api-b", 9)]]
 assert request("POST", "/api/v2/alerts", json.dumps(alerts))[0] == 200
 before = eventually(lambda: (a if len(a := get_json("/api/v2/alerts")) == 2 else False))
 fingerprints = {a["labels"]["instance"]: a["fingerprint"] for a in before}
 alerts[0]["annotations"]["summary"] = "queue=8"
 assert request("POST", "/api/v2/alerts", json.dumps([alerts[0]]))[0] == 200
 def updated_annotations():
 items = get_json("/api/v2/alerts")
 changed = any(x["labels"]["instance"] == "api-a" and
 x["annotations"]["summary"] == "queue=8" for x in items)
 return items if changed else None
 updated = eventually(updated_annotations)
 assert len(updated) == 2
 assert {a["labels"]["instance"]: a["fingerprint"] for a in updated} == fingerprints
 # Supply the documented notification Data shape to the real template renderer.
 data = {"receiver": "preview", "status": "firing",
 "groupLabels": {"alertname": "QueueHigh", "service": "checkout"},
 "commonLabels": {"alertname": "QueueHigh", "service": "checkout"},
 "commonAnnotations": {"runbook": "https://example.invalid/queue"},
 "alerts": [{"status": "firing", "labels": a["labels"], "annotations": a["annotations"]}
 for a in sorted(updated, key=lambda a: a["labels"]["instance"])]}
 template = '{{define "preview"}}{{.GroupLabels.service}}|common-summary={{with .CommonAnnotations.summary}}{{.}}{{else}}none{{end}}{{range .Alerts}}\n{{.Labels.instance}}:{{.Annotations.summary}}{{end}}{{end}}'
 for path, content in [("/tmp/preview.tmpl", template), ("/tmp/data.json", json.dumps(data))]:
 subprocess.run(["docker", "exec", "-i", name, "/bin/sh", "-c", "cat > " + path],
 input=content, text=True, capture_output=True, check=True, timeout=5)
 rendered = subprocess.run(["docker", "exec", name, "/bin/amtool", "template", "render",
 "--template.glob=/tmp/preview.tmpl", "--template.data=/tmp/data.json",
 '--template.text={{ template "preview" . }}'],
 text=True, capture_output=True, check=True, timeout=5).stdout.strip()
 assert rendered == "checkout|common-summary=none\napi-a:queue=8\napi-b:queue=9", rendered
 print(json.dumps({"alert_count": 2, "fingerprints_unchanged": True, "rendered": rendered}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The result confirms that two alerts remain and their fingerprints are unchanged after the update. The program then supplies the documented notification data structure to the real amtool template renderer. Its common annotations intentionally include the shared runbook but exclude the unequal summaries. This explicit template input tests rendering, not the server’s notification assembly or delivery pipeline.

The rendered heading says common-summary=none, followed by api-a:queue=8 and api-b:queue=9. Assertions verify the updated value and the separate per-alert details. All HTTP traffic remains on container loopback, with no host mounts or published ports, and the container is removed on exit. A production notification still needs validation using its actual route, template data, and receiver integration.

Sources

Alertmanager: notification template reference, group labels, common annotations and individual alert data, checked 2026-09-13.
Prometheus: alerting rules, labels identify alerts while annotations carry explanatory content, checked 2026-09-13.
Alertmanager 0.28.1: README, local template rendering with amtool, checked 2026-09-13.

Alertmanager routes alerts through a label-matching tree

Wed, 08 Dec 2027 00:00:00 +0000

Alertmanager selects notification receivers by walking a routing tree against an alert’s labels. The tree expresses ownership and escalation policy, while receivers describe notification integrations. A configuration can parse correctly and still send the wrong class of alert to the wrong destination. Verify the label contract and tree behavior before treating a successful reload as evidence that routing works.

Follow the tree, including its stopping rule

Every alert enters the root route, which must match all alerts. Child routes inherit settings they do not override. If no child matches, the current node handles the alert. Otherwise, traversal normally stops after the first matching child. Setting continue: true on a matching child allows subsequent siblings to be considered. It does not mean that every receiver in the configuration receives the alert.

One route node names one receiver. A receiver can contain multiple notification integrations, and multiple matching routes can also select different receivers. These are different ways to deliver to several destinations. A route cannot simply replace its receiver field with a list and expect the same behavior. Review the chosen approach for duplicate notifications and intentionally different grouping or escalation settings.

Ownership is a label contract

A route matching team="payments" needs that exact label value on the alert entering Alertmanager. An annotation named team does not satisfy the matcher. Aggregation in a Prometheus expression can remove a needed dimension before the alert reaches routing. Inspect the final alert label set, including alert relabeling, rather than assuming the raw metric’s labels survived unchanged.

Keep the root receiver useful for alerts that miss more specific children. An unmatched alert still needs an observable destination and an owner who can correct its classification. Test absent labels as well as recognized teams. A missing team on a warning can take the root route, while a critical alert without that team can still match a separate severity route.

Route order is part of behavior. A broad early route can prevent a narrower later route from being reached unless continuation is enabled. Moving a block of YAML can therefore change recipients without changing any matcher text. Review ordering changes like policy changes, and keep representative label fixtures beside the configuration so the intended selection remains explicit.

When it breaks

A typo or a label renamed by an exporter can send alerts to the root receiver. A new catch-all child can capture alerts intended for a later team. Excessive continuation can create multiple notifications for one event. Diagnose these failures with the actual labels and the entire matching path, not by reading one route in isolation.

Correct receiver selection still does not establish delivery. Grouping delays, silence or inhibition, template errors, transport failures, and destination behavior can intervene afterward. Inspect routing first, then follow the selected receiver through the notification pipeline. Increasing the rule’s waiting duration will not fix an incorrect team matcher, and reducing grouping delay will not repair missing credentials at a receiver.

Check it quickly

Save this as routing.py and run python3 routing.py. It requires Docker and the public Alertmanager 0.28.1 image, which may be downloaded if absent. The real amtool validates the configuration and checks seven label cases. Receivers deliberately have names without delivery integrations, so the experiment evaluates routing without contacting any person or external service.

import subprocess
import uuid

# Requires Docker and Alertmanager 0.28.1. No receiver has a delivery integration.
script = r"""set -eu
cat > /tmp/routes.yml <<'ROUTES'
route:
 receiver: default
 group_by: [alertname, service]
 routes:
 - receiver: payments
 matchers: ['team="payments"']
 continue: true
 - receiver: pager
 matchers: ['severity="critical"']
receivers:
 - name: default
 - name: payments
 - name: pager

ROUTES
/bin/amtool check-config /tmp/routes.yml
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=payments,pager team=payments severity=critical
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=payments team=payments severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=pager team=data severity=critical
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default team=data severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default team=payments-preview severity=warning
sed 's/continue: true/continue: false/' /tmp/routes.yml > /tmp/stop.yml
/bin/amtool config routes test --config.file=/tmp/stop.yml --verify.receivers=payments team=payments severity=critical
"""
name = "xbcatm-route-test-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-s"],
 input=script, text=True, capture_output=True, timeout=30,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 raise RuntimeError(result.stderr or result.stdout)
print("Verified seven route cases, including continuation and unmatched labels.")

The first case resolves to payments,pager: the payments route matches and continues, then the critical route also matches. A payments warning selects only payments; a critical data-team alert selects pager. Unmatched and missing team values select the root receiver. The payments-preview value does not match the exact payments value.

The final check changes continuation to false and confirms that the payments critical alert now selects only payments. Each expected receiver set is enforced by --verify.receivers, so a mismatch fails execution. The container has no external network or host mounts and is removed on exit. Notification delivery requires a separate controlled test of the real receiver integration.

Sources

Alertmanager: configuration, route inheritance, sibling continuation and receiver definitions, checked 2026-09-13.
Alertmanager 0.28.1: README, amtool configuration and routing checks, checked 2026-09-13.
Prometheus: Alertmanager, separation of routing, grouping and suppression, checked 2026-09-13.

Prometheus rules track alert state before notification delivery

Wed, 01 Sep 2027 00:00:00 +0000

An alerting rule evaluates a PromQL expression and tracks the returned label sets over time. Prometheus decides whether each resulting alert is pending or firing. Alertmanager receives alert information and handles notification routing, grouping, silencing, and inhibition. A reliable investigation follows those stages separately: a firing alert is evidence of rule state, not proof that a person received a message.

Vector membership activates the alert

Each element returned by the expression represents an active alert candidate. A comparison such as queue_depth > 5 filters out values that do not meet the threshold. Adding the bool modifier changes comparison behavior: false comparisons remain as elements with value zero. An alert rule tests the presence of elements, so using that modifier can activate an alert even when the number is zero.

Labels identify alert instances. Keep stable routing dimensions, such as service, queue, and severity, in labels. Put changing explanatory values in annotations. If a label includes the current queue size, every change can create a different identity and restart the pending period. An annotation can include that size without redefining which alert is being tracked.

Waiting is measured at evaluations

With for: 2m, a newly active label set is pending until it has remained active for two minutes of evaluated time. A rule evaluated every minute first seeing the condition at minute one can fire at minute three. The engine observes evaluations, not continuous physical reality between samples. The timing therefore includes the wait until the next evaluation as well as the configured duration.

Omitting for permits an active expression to fire on its first evaluation. That may suit an unequivocal failure, but it also makes a single bad sample sufficient; choose deliberately rather than relying on a forgotten default.

If the expression stops returning that label set before firing, pending state clears. With the default behavior used here, a firing alert also clears when its condition disappears. An explicitly configured keep_firing_for can extend firing after the condition clears; that option is not present in this example. Do not describe all alerts as latched incidents that require manual acknowledgement to resolve.

When it breaks

A threshold over a missing series usually returns nothing. That can clear an alert even though the application has stopped exporting data. Decide whether absence needs its own rule rather than blindly treating it as recovery. Test label changes and missing samples alongside threshold transitions. A syntax check alone cannot reveal these semantic failures.

A firing alert can produce no visible page because a silence, an inhibition rule, routing, grouping delay, or delivery failure intervened. These mechanisms belong to the notification path and do not change Prometheus’s threshold evaluation. Inspect the active alert labels and then the matching Alertmanager route and delivery evidence. Increasing for does not repair a wrong receiver or a missing route label.

Check it quickly

Save this as alert-state.py and run python3 alert-state.py with Docker available. It uses Prometheus 3.5.1; Docker may fetch the image if it is not cached. Real promtool evaluates the rule against queue values 0 6 6 6 0 6 0 0, one sample per minute. No Alertmanager or external notification destination is involved.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
rule_files:
 - /tmp/rules.yml
tests:
 - interval: 1m
 input_series:
 - series: "queue_depth{queue=\"orders\"}"
 values: "0 6 6 6 0 6 0 0"
 promql_expr_test:
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\",alertstate=\"pending\"}) or vector(0)"
 eval_time: 1m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\",alertstate=\"firing\"}) or vector(0)"
 eval_time: 2m
 exp_samples:
 - labels: "{}"
 value: 0
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\",alertstate=\"firing\"}) or vector(0)"
 eval_time: 3m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\"}) or vector(0)"
 eval_time: 4m
 exp_samples:
 - labels: "{}"
 value: 0
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\",alertstate=\"pending\"}) or vector(0)"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
 - expr: "sum(ALERTS{alertname=\"QueueBacklog\"}) or vector(0)"
 eval_time: 6m
 exp_samples:
 - labels: "{}"
 value: 0
"""
rules = r"""groups:
 - name: queue-alerts
 interval: 1m
 rules:
 - alert: QueueBacklog
 expr: queue_depth{queue="orders"} > 5
 for: 2m
 labels:
 severity: page
 annotations:
 summary: Orders queue remains above five
"""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
 script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

The expected output is SUCCESS. At minute one the alert is pending, at minute two it is not firing, and at minute three it fires. At minute four the condition recovers and no active ALERTS element remains. A second spike starts pending at minute five but clears at minute six, so it never reaches firing.

The assertions inspect the synthetic ALERTS series; their or vector(0) only makes absence countable inside the test. It is not part of the production alert expression. The disposable container has no network access or host mounts. This verifies state transitions for the supplied timeline. Receiver matching, grouping delays, and actual message delivery require separate tests with the intended Alertmanager configuration.

Sources

Prometheus: alerting rules, active label sets, for and synthetic ALERTS series, checked 2026-09-13.
Prometheus: unit testing rules, evaluation timelines and assertion fixtures, checked 2026-09-13.
Prometheus: Alertmanager, notification grouping, routing, silences and inhibition, checked 2026-09-13.

Alertmanager grouping batches alerts while silences suppress notifications

Tue, 27 Jul 2027 00:00:00 +0000

One incident can produce many alerts, but a group is not proof that those alerts have one cause. Alertmanager groups alerts according to configured labels and routing context so their notifications can be handled together. The original alert identities remain available inside the group. Silences add another operation: they suppress matching notifications for a time window without resolving the underlying conditions.

A group key determines which details stay together

With group_by: [alertname, service], two QueueHigh alerts for the checkout service can share a group even when their instance labels differ. A QueueHigh alert for search belongs to another group. A DiskFull alert for checkout also belongs to another group because its alert name differs. Grouping therefore depends on the selected label values, not on all alerts having an identical fingerprint.

Adding instance to the group key creates more groups and may undo the desired batching across replicas. Omitting service can combine unrelated owners or workloads. Choose the key around the investigation and ownership model, then inspect the member list to retain individual context. The special grouping value ... groups by all labels and effectively disables aggregation into broader notification groups.

Routing context matters too. Alerts routed to different receivers are not necessarily one notification merely because their selected grouping labels match. Test the combination of route and group configuration. A receiver responsible for a whole service may benefit from batching several instances, while another integration can require a different grouping contract. Neither setting automatically determines whether the alerts share a root cause.

Waiting, changes and repetition use different timers

group_wait delays the first notification for a new group so nearby arrivals can join it. An alert that resolves before that initial wait ends may never generate a notification. group_interval governs checks for subsequent group updates, including newly firing or resolved members. It is not simply a promise to resend every still-open group at that frequency.

repeat_interval controls reminders when the group’s notification content has not otherwise changed. Its eligibility is checked on the group interval, so choose a compatible multiple instead of assuming an independent precise alarm clock. Delivery processing and failures can add further delay. Set these values from the response objective, and keep the rule’s pending duration separate from notification scheduling.

When it breaks

Grouping too narrowly turns a replica outage into many notifications; grouping too broadly can hide which service needs attention. A dashboard showing one group does not mean only one alert remains. Expand the members and inspect the group key before treating a lower notification count as evidence that the incident has improved.

A silence with overly broad matchers can suppress alerts beyond the planned maintenance scope. Use a bounded time window and inspect which active alerts match it. Silence expiration restores notification eligibility; it does not fix the application or guarantee an immediate message. Inhibition is a separate configured relationship where an active source alert can suppress matching target notifications, rather than a time-bounded manual silence.

Check it quickly

Save this as grouping.py and run python3 grouping.py with Docker available. It uses Alertmanager 0.28.1, fetching the image if needed. The actual API receives four synthetic alerts. The receiver has no delivery integrations and the initial notification wait is deliberately long, so the experiment observes grouping and suppression state without sending messages.

import http.client
import io
import json
import subprocess
import time
import uuid

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


from datetime import datetime, timedelta, timezone

port = 9093
script = r"""set -eu
cat > /tmp/alertmanager.yml <<'ALERT_CONFIG'
route:
 receiver: preview
 group_by: [alertname, service]
 group_wait: 1h
 group_interval: 5m
 repeat_interval: 4h
receivers:
 - name: preview
ALERT_CONFIG
exec /bin/alertmanager --config.file=/tmp/alertmanager.yml --storage.path=/tmp/data \
 --web.listen-address=127.0.0.1:9093 --cluster.listen-address=
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: request("GET", "/-/ready")[0] == 200)
 labels = [
 {"alertname": "QueueHigh", "service": "checkout", "instance": "api-a"},
 {"alertname": "QueueHigh", "service": "checkout", "instance": "api-b"},
 {"alertname": "QueueHigh", "service": "search", "instance": "api-c"},
 {"alertname": "DiskFull", "service": "checkout", "instance": "api-a"},
 ]
 assert request("POST", "/api/v2/alerts", json.dumps([{"labels": x} for x in labels]))[0] == 200
 groups = eventually(lambda: (groups if len(groups := get_json("/api/v2/alerts/groups")) == 3 else False))
 sizes = {g["labels"]["service"] + "/" + g["labels"]["alertname"]: len(g["alerts"]) for g in groups}
 assert sizes == {"checkout/QueueHigh": 2, "checkout/DiskFull": 1, "search/QueueHigh": 1}
 now = datetime.now(timezone.utc)
 silence = {"matchers": [{"name": "service", "value": "checkout", "isRegex": False, "isEqual": True}],
 "startsAt": (now - timedelta(seconds=1)).isoformat(),
 "endsAt": (now + timedelta(minutes=10)).isoformat(),
 "createdBy": "local-fixture", "comment": "Disposable grouping test"}
 status, body = request("POST", "/api/v2/silences", json.dumps(silence))
 assert status == 200
 silence_id = json.loads(body)["silenceID"]
 def silenced_state():
 alerts = get_json("/api/v2/alerts")
 count = sum(silence_id in x["status"]["silencedBy"] for x in alerts)
 return alerts if count == 3 else None
 alerts = eventually(silenced_state)
 assert len(alerts) == 4
 assert all((silence_id in x["status"]["silencedBy"]) == (x["labels"]["service"] == "checkout") for x in alerts)
 assert request("DELETE", "/api/v2/silence/" + silence_id)[0] == 200
 def unsilenced_state():
 alerts = get_json("/api/v2/alerts")
 return len(alerts) == 4 and all(silence_id not in x["status"]["silencedBy"] for x in alerts)
 eventually(unsilenced_state)
 print(json.dumps({"group_sizes": sizes, "silenced_alerts": 3,
 "alerts_remain_visible": 4, "silence_expired": True}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The output must report group sizes of two for checkout/QueueHigh, one for checkout/DiskFull, and one for search/QueueHigh. A temporary silence matching checkout suppresses three alerts, while all four remain visible through the API. The search alert stays outside that silence. The program then expires the silence and verifies that its identifier no longer appears in the alerts’ suppression lists.

All requests stay on container loopback, with no host ports or mounts; cleanup removes the unique container. These assertions test membership and silence lifecycle. They do not measure notification timer accuracy, validate inhibition, or establish downstream delivery. Those behaviors require tests using the actual routing, scheduling and receiver configuration.

Sources

Alertmanager: configuration, group keys, notification intervals and inhibition rules, checked 2026-09-13.
Prometheus: Alertmanager, grouping and silence semantics, checked 2026-09-13.
Alertmanager 0.28.1: README, API v2 and local Alertmanager operation, checked 2026-09-13.

Prometheus staleness and alert waiting are different mechanisms

Wed, 03 Feb 2027 00:00:00 +0000

A missing metric is not a numeric zero. Prometheus distinguishes a series with a recent value, a series marked stale, and a selector with no usable sample. Alerting then applies its own evaluation and waiting rules. Treating all these mechanisms as one universal grace period produces alerts that miss disappearing targets or report an outage without establishing what actually disappeared.

A marker is not a five-minute timer

An instant selector normally uses the newest eligible sample within its lookback period, which defaults to five minutes. A stale marker tells the engine to stop returning that series in instant selections after the marker. It does not require the previous value to remain visible for another five minutes. Historical samples are still stored and can remain relevant to range queries.

Collection circumstances matter. A series no longer exported, a target removed from discovery, and an exporter assigning its own timestamps need not follow identical timing. Explicit timestamps and staleness configuration can alter behavior. Determine which path your exporter uses before predicting disappearance time from the scrape interval alone. A configurable lookback is a query selection rule, not a universal service outage detector.

Choose the absence question deliberately

absent(selector) returns a value of one when the selector has no elements. It returns an empty vector when matching data exists, even if that data’s numeric value is zero. absent_over_time(selector[3m]) asks whether the selected range contains any samples. A real sample from before a stale marker can keep that range-based absence expression empty until the sample leaves the window.

The alert rule’s for duration is another layer: the expression must keep returning the relevant alert label set at successive evaluations before it fires. It does not configure staleness or extend the query’s lookback. Combining a three-minute absence window with a waiting duration delays notification further, with evaluation cadence and collection timing also contributing. Choose that delay from the detection objective.

When it breaks

absent(up{job="api"}) is not a failed-scrape alert. If the configured targets are present with up=0, the selector still has elements, so absence is false. A comparison against zero addresses that case. If a target disappears from discovery altogether, checking only zero can miss it once its series stops appearing. Monitor expected membership separately when that distinction matters.

A broad selector also cannot enumerate missing members. If one replica survives, absent over the whole job remains empty even when several expected replicas are gone. Match a deliberately known identity or compare observed series with an authoritative inventory. A label rename can look like an outage under the old matcher while the process continues working under a new identity.

Check it quickly

Save the following as stale.py and run python3 stale.py with Docker available. The program uses the public Prometheus 3.5.1 image and real promtool queries. Docker may download an uncached image. Synthetic samples include an explicit stale marker, missing positions, and ordinary zeros; this is a query-semantics test, not a measurement of live discovery timing.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
tests:
 - interval: 1m
 input_series:
 - series: "up{job=\"api\",instance=\"gone\"}"
 values: "1 0 0 0 0"
 - series: "service_events_total{job=\"api\",instance=\"gone\"}"
 values: "10 stale _ _ _"
 - series: "service_events_total{job=\"api\",instance=\"zero\"}"
 values: "0 0 0 0 0"
 promql_expr_test:
 - expr: "service_events_total{job=\"api\",instance=\"gone\"}"
 eval_time: 1m
 exp_samples: []
 - expr: "up{job=\"api\",instance=\"gone\"}"
 eval_time: 1m
 exp_samples:
 - labels: "{__name__=\"up\",job=\"api\",instance=\"gone\"}"
 value: 0
 - expr: "absent(service_events_total{job=\"api\",instance=\"gone\"})"
 eval_time: 1m
 exp_samples:
 - labels: "{job=\"api\",instance=\"gone\"}"
 value: 1
 - expr: "absent_over_time(service_events_total{job=\"api\",instance=\"gone\"}[3m])"
 eval_time: 2m
 exp_samples: []
 - expr: "absent_over_time(service_events_total{job=\"api\",instance=\"gone\"}[3m])"
 eval_time: 4m
 exp_samples:
 - labels: "{job=\"api\",instance=\"gone\"}"
 value: 1
 - expr: "absent(service_events_total{job=\"api\",instance=\"zero\"})"
 eval_time: 4m
 exp_samples: []
 - expr: "absent(up{job=\"api\"})"
 eval_time: 4m
 exp_samples: []
"""
rules = r""""""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
 script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

The test prints SUCCESS. At minute one, the marked business series is absent from an instant selection, while its separately supplied up series exists with value zero. Instant absent returns one. At minute two, the three-minute range still contains the original business sample, so absent_over_time returns nothing; at minute four it returns one.

The continuously exported zero-valued business series is never absent, and the job-wide up selector is not absent either. These assertions distinguish missing values from failing values instead of converting both to zero. The container has no network or host mounts and is removed on exit. For a production rule, supplement these cases with expected target membership and an independently tested pending-to-firing timeline.

Sources

Prometheus: querying basics, instant selectors, lookback and staleness, checked 2026-09-13.
Prometheus: query functions, absent and absent_over_time behavior, checked 2026-09-13.
Prometheus: unit testing rules, missing and stale sample notation, checked 2026-09-13.

Alerting · XB Field Notes

Annotations explain an alert without changing its identity

Separate identity from the current explanation

A group does not necessarily share its summary

When it breaks

Check it quickly

Sources

Alertmanager routes alerts through a label-matching tree

Follow the tree, including its stopping rule

Ownership is a label contract

When it breaks

Check it quickly

Sources

Prometheus rules track alert state before notification delivery

Vector membership activates the alert

Waiting is measured at evaluations

When it breaks

Check it quickly

Sources

Alertmanager grouping batches alerts while silences suppress notifications

A group key determines which details stay together

Waiting, changes and repetition use different timers

When it breaks

Check it quickly

Sources

Prometheus staleness and alert waiting are different mechanisms

A marker is not a five-minute timer

Choose the absence question deliberately

When it breaks

Check it quickly

Sources