Alert labels identify which condition is being tracked and provide fields for routing and grouping. Annotations explain that condition: a current value, a concise summary, or a runbook link. Putting changing prose in labels can split one ongoing problem into many identities. Putting routing fields only in annotations can leave a route unable to recognize the alert’s intended owner.

Separate identity from the current explanation

A queue alert might keep alertname, service, and instance as stable labels while its summary changes from six waiting items to eight. Alertmanager can update the annotation for the same alert fingerprint rather than adding another alert. A fingerprint is derived from identity labels; it is not a hash of the rendered notification text.

This distinction also matters before Alertmanager receives the alert. Prometheus tracks its pending duration for an alert label set. A label containing a changing current value can keep creating new identities and prevent the intended waiting period from completing. Put that changing value in an annotation unless it deliberately defines a different alert condition. Review templates and rule labels together when diagnosing repeated pending alerts.

A group does not necessarily share its summary

Notification templates receive several scopes of data. GroupLabels contains the labels chosen for grouping. CommonLabels contains label pairs common to all alerts in the notification. CommonAnnotations similarly contains annotations shared by the group. The Alerts list retains the labels and annotations of individual members, allowing a notification to explain their different states.

If two replicas have summaries describing different queue sizes, there is no shared summary to put in CommonAnnotations. A template that prints only that field can omit the very information an operator needs. Use common context for the heading, then iterate over individual alerts for changing summaries. A shared runbook link can remain common even when the individual values differ.

Do not assume a grouping label such as instance exists in every group. Grouping by service and alert name intentionally combines instances. A template should reflect the selected grouping contract and handle an absent common field deliberately. Representative fixtures need multiple alerts, unequal annotations, and the relevant firing or resolved states, not just one idealized alert where every field happens to be common.

When it breaks

A dynamic label creates a new fingerprint for every value change, increasing alert churn and potentially restarting pending state. A route matching team cannot use a team annotation in place of a label. An overly broad notification heading can also describe one instance while listing alerts from several services if grouping and template assumptions drift apart.

A template that parses successfully can still render an empty or misleading message for actual group data. Check the rendered text, not only syntax. Conversely, rendering successfully does not prove the integration accepts the message or that a notification was delivered. Keep credentials and sensitive request details out of annotations because they can be copied to every receiver selected by routing.

Check it quickly

Save this as annotations.py and run python3 annotations.py with Docker available. It uses Alertmanager 0.28.1, downloading the image if needed. The isolated server has a receiver with no delivery integrations. Two synthetic alerts are submitted through its actual API; one annotation is then updated without changing any labels.

import http.client
import io
import json
import subprocess
import time
import uuid

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
    def __init__(self, data):
        self.data = data

    def makefile(self, mode):
        return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
    body = body.encode()
    headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
               f"Connection: close\r\nContent-Type: {content_type}\r\n"
               f"Content-Length: {len(body)}\r\n\r\n").encode()
    result = subprocess.run(
        ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
        input=headers + body, capture_output=True, timeout=5,
    )
    if result.returncode:
        raise RuntimeError(result.stderr.decode())
    response = http.client.HTTPResponse(ResponseSocket(result.stdout))
    response.begin()
    return response.status, response.read().decode()


def get_json(path):
    status, body = request("GET", path)
    assert status == 200, (status, body)
    return json.loads(body)


def eventually(check):
    for attempt in range(150):
        try:
            value = check()
            if value:
                return value
        except (RuntimeError, AssertionError, http.client.HTTPException):
            pass
        time.sleep(0.1)
    raise AssertionError("Expected local server state did not appear")


port = 9093
script = r"""set -eu
cat > /tmp/alertmanager.yml <<'ALERT_CONFIG'
route:
  receiver: preview
  group_by: [alertname, service]
  group_wait: 1h
  group_interval: 5m
  repeat_interval: 4h
receivers:
  - name: preview
ALERT_CONFIG
exec /bin/alertmanager --config.file=/tmp/alertmanager.yml --storage.path=/tmp/data \
  --web.listen-address=127.0.0.1:9093 --cluster.listen-address=
"""
try:
    subprocess.run(
        ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
         "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-c", script],
        check=True, capture_output=True, text=True, timeout=20,
    )
    eventually(lambda: request("GET", "/-/ready")[0] == 200)
    alerts = [{"labels": {"alertname": "QueueHigh", "service": "checkout", "instance": instance},
               "annotations": {"summary": "queue=" + str(depth), "runbook": "https://example.invalid/queue"}}
              for instance, depth in [("api-a", 6), ("api-b", 9)]]
    assert request("POST", "/api/v2/alerts", json.dumps(alerts))[0] == 200
    before = eventually(lambda: (a if len(a := get_json("/api/v2/alerts")) == 2 else False))
    fingerprints = {a["labels"]["instance"]: a["fingerprint"] for a in before}
    alerts[0]["annotations"]["summary"] = "queue=8"
    assert request("POST", "/api/v2/alerts", json.dumps([alerts[0]]))[0] == 200
    def updated_annotations():
        items = get_json("/api/v2/alerts")
        changed = any(x["labels"]["instance"] == "api-a" and
                      x["annotations"]["summary"] == "queue=8" for x in items)
        return items if changed else None
    updated = eventually(updated_annotations)
    assert len(updated) == 2
    assert {a["labels"]["instance"]: a["fingerprint"] for a in updated} == fingerprints
    # Supply the documented notification Data shape to the real template renderer.
    data = {"receiver": "preview", "status": "firing",
            "groupLabels": {"alertname": "QueueHigh", "service": "checkout"},
            "commonLabels": {"alertname": "QueueHigh", "service": "checkout"},
            "commonAnnotations": {"runbook": "https://example.invalid/queue"},
            "alerts": [{"status": "firing", "labels": a["labels"], "annotations": a["annotations"]}
                       for a in sorted(updated, key=lambda a: a["labels"]["instance"])]}
    template = '{{define "preview"}}{{.GroupLabels.service}}|common-summary={{with .CommonAnnotations.summary}}{{.}}{{else}}none{{end}}{{range .Alerts}}\n{{.Labels.instance}}:{{.Annotations.summary}}{{end}}{{end}}'
    for path, content in [("/tmp/preview.tmpl", template), ("/tmp/data.json", json.dumps(data))]:
        subprocess.run(["docker", "exec", "-i", name, "/bin/sh", "-c", "cat > " + path],
                       input=content, text=True, capture_output=True, check=True, timeout=5)
    rendered = subprocess.run(["docker", "exec", name, "/bin/amtool", "template", "render",
                               "--template.glob=/tmp/preview.tmpl", "--template.data=/tmp/data.json",
                               '--template.text={{ template "preview" . }}'],
                              text=True, capture_output=True, check=True, timeout=5).stdout.strip()
    assert rendered == "checkout|common-summary=none\napi-a:queue=8\napi-b:queue=9", rendered
    print(json.dumps({"alert_count": 2, "fingerprints_unchanged": True, "rendered": rendered}, indent=2))
finally:
    subprocess.run(["docker", "rm", "-f", name],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The result confirms that two alerts remain and their fingerprints are unchanged after the update. The program then supplies the documented notification data structure to the real amtool template renderer. Its common annotations intentionally include the shared runbook but exclude the unequal summaries. This explicit template input tests rendering, not the server’s notification assembly or delivery pipeline.

The rendered heading says common-summary=none, followed by api-a:queue=8 and api-b:queue=9. Assertions verify the updated value and the separate per-alert details. All HTTP traffic remains on container loopback, with no host mounts or published ports, and the container is removed on exit. A production notification still needs validation using its actual route, template data, and receiver integration.

Sources