One incident can produce many alerts, but a group is not proof that those alerts have one cause. Alertmanager groups alerts according to configured labels and routing context so their notifications can be handled together. The original alert identities remain available inside the group. Silences add another operation: they suppress matching notifications for a time window without resolving the underlying conditions.

A group key determines which details stay together

With group_by: [alertname, service], two QueueHigh alerts for the checkout service can share a group even when their instance labels differ. A QueueHigh alert for search belongs to another group. A DiskFull alert for checkout also belongs to another group because its alert name differs. Grouping therefore depends on the selected label values, not on all alerts having an identical fingerprint.

Adding instance to the group key creates more groups and may undo the desired batching across replicas. Omitting service can combine unrelated owners or workloads. Choose the key around the investigation and ownership model, then inspect the member list to retain individual context. The special grouping value ... groups by all labels and effectively disables aggregation into broader notification groups.

Routing context matters too. Alerts routed to different receivers are not necessarily one notification merely because their selected grouping labels match. Test the combination of route and group configuration. A receiver responsible for a whole service may benefit from batching several instances, while another integration can require a different grouping contract. Neither setting automatically determines whether the alerts share a root cause.

Waiting, changes and repetition use different timers

group_wait delays the first notification for a new group so nearby arrivals can join it. An alert that resolves before that initial wait ends may never generate a notification. group_interval governs checks for subsequent group updates, including newly firing or resolved members. It is not simply a promise to resend every still-open group at that frequency.

repeat_interval controls reminders when the group’s notification content has not otherwise changed. Its eligibility is checked on the group interval, so choose a compatible multiple instead of assuming an independent precise alarm clock. Delivery processing and failures can add further delay. Set these values from the response objective, and keep the rule’s pending duration separate from notification scheduling.

When it breaks

Grouping too narrowly turns a replica outage into many notifications; grouping too broadly can hide which service needs attention. A dashboard showing one group does not mean only one alert remains. Expand the members and inspect the group key before treating a lower notification count as evidence that the incident has improved.

A silence with overly broad matchers can suppress alerts beyond the planned maintenance scope. Use a bounded time window and inspect which active alerts match it. Silence expiration restores notification eligibility; it does not fix the application or guarantee an immediate message. Inhibition is a separate configured relationship where an active source alert can suppress matching target notifications, rather than a time-bounded manual silence.

Check it quickly

Save this as grouping.py and run python3 grouping.py with Docker available. It uses Alertmanager 0.28.1, fetching the image if needed. The actual API receives four synthetic alerts. The receiver has no delivery integrations and the initial notification wait is deliberately long, so the experiment observes grouping and suppression state without sending messages.

import http.client
import io
import json
import subprocess
import time
import uuid

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
    def __init__(self, data):
        self.data = data

    def makefile(self, mode):
        return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
    body = body.encode()
    headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
               f"Connection: close\r\nContent-Type: {content_type}\r\n"
               f"Content-Length: {len(body)}\r\n\r\n").encode()
    result = subprocess.run(
        ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
        input=headers + body, capture_output=True, timeout=5,
    )
    if result.returncode:
        raise RuntimeError(result.stderr.decode())
    response = http.client.HTTPResponse(ResponseSocket(result.stdout))
    response.begin()
    return response.status, response.read().decode()


def get_json(path):
    status, body = request("GET", path)
    assert status == 200, (status, body)
    return json.loads(body)


def eventually(check):
    for attempt in range(150):
        try:
            value = check()
            if value:
                return value
        except (RuntimeError, AssertionError, http.client.HTTPException):
            pass
        time.sleep(0.1)
    raise AssertionError("Expected local server state did not appear")


from datetime import datetime, timedelta, timezone

port = 9093
script = r"""set -eu
cat > /tmp/alertmanager.yml <<'ALERT_CONFIG'
route:
  receiver: preview
  group_by: [alertname, service]
  group_wait: 1h
  group_interval: 5m
  repeat_interval: 4h
receivers:
  - name: preview
ALERT_CONFIG
exec /bin/alertmanager --config.file=/tmp/alertmanager.yml --storage.path=/tmp/data \
  --web.listen-address=127.0.0.1:9093 --cluster.listen-address=
"""
try:
    subprocess.run(
        ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
         "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-c", script],
        check=True, capture_output=True, text=True, timeout=20,
    )
    eventually(lambda: request("GET", "/-/ready")[0] == 200)
    labels = [
        {"alertname": "QueueHigh", "service": "checkout", "instance": "api-a"},
        {"alertname": "QueueHigh", "service": "checkout", "instance": "api-b"},
        {"alertname": "QueueHigh", "service": "search", "instance": "api-c"},
        {"alertname": "DiskFull", "service": "checkout", "instance": "api-a"},
    ]
    assert request("POST", "/api/v2/alerts", json.dumps([{"labels": x} for x in labels]))[0] == 200
    groups = eventually(lambda: (groups if len(groups := get_json("/api/v2/alerts/groups")) == 3 else False))
    sizes = {g["labels"]["service"] + "/" + g["labels"]["alertname"]: len(g["alerts"]) for g in groups}
    assert sizes == {"checkout/QueueHigh": 2, "checkout/DiskFull": 1, "search/QueueHigh": 1}
    now = datetime.now(timezone.utc)
    silence = {"matchers": [{"name": "service", "value": "checkout", "isRegex": False, "isEqual": True}],
               "startsAt": (now - timedelta(seconds=1)).isoformat(),
               "endsAt": (now + timedelta(minutes=10)).isoformat(),
               "createdBy": "local-fixture", "comment": "Disposable grouping test"}
    status, body = request("POST", "/api/v2/silences", json.dumps(silence))
    assert status == 200
    silence_id = json.loads(body)["silenceID"]
    def silenced_state():
        alerts = get_json("/api/v2/alerts")
        count = sum(silence_id in x["status"]["silencedBy"] for x in alerts)
        return alerts if count == 3 else None
    alerts = eventually(silenced_state)
    assert len(alerts) == 4
    assert all((silence_id in x["status"]["silencedBy"]) == (x["labels"]["service"] == "checkout") for x in alerts)
    assert request("DELETE", "/api/v2/silence/" + silence_id)[0] == 200
    def unsilenced_state():
        alerts = get_json("/api/v2/alerts")
        return len(alerts) == 4 and all(silence_id not in x["status"]["silencedBy"] for x in alerts)
    eventually(unsilenced_state)
    print(json.dumps({"group_sizes": sizes, "silenced_alerts": 3,
                      "alerts_remain_visible": 4, "silence_expired": True}, indent=2))
finally:
    subprocess.run(["docker", "rm", "-f", name],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The output must report group sizes of two for checkout/QueueHigh, one for checkout/DiskFull, and one for search/QueueHigh. A temporary silence matching checkout suppresses three alerts, while all four remain visible through the API. The search alert stays outside that silence. The program then expires the silence and verifies that its identifier no longer appears in the alerts’ suppression lists.

All requests stay on container loopback, with no host ports or mounts; cleanup removes the unique container. These assertions test membership and silence lifecycle. They do not measure notification timer accuracy, validate inhibition, or establish downstream delivery. Those behaviors require tests using the actual routing, scheduling and receiver configuration.

Sources