Alertmanager selects notification receivers by walking a routing tree against an alert’s labels. The tree expresses ownership and escalation policy, while receivers describe notification integrations. A configuration can parse correctly and still send the wrong class of alert to the wrong destination. Verify the label contract and tree behavior before treating a successful reload as evidence that routing works.

Follow the tree, including its stopping rule

Every alert enters the root route, which must match all alerts. Child routes inherit settings they do not override. If no child matches, the current node handles the alert. Otherwise, traversal normally stops after the first matching child. Setting continue: true on a matching child allows subsequent siblings to be considered. It does not mean that every receiver in the configuration receives the alert.

One route node names one receiver. A receiver can contain multiple notification integrations, and multiple matching routes can also select different receivers. These are different ways to deliver to several destinations. A route cannot simply replace its receiver field with a list and expect the same behavior. Review the chosen approach for duplicate notifications and intentionally different grouping or escalation settings.

Ownership is a label contract

A route matching team="payments" needs that exact label value on the alert entering Alertmanager. An annotation named team does not satisfy the matcher. Aggregation in a Prometheus expression can remove a needed dimension before the alert reaches routing. Inspect the final alert label set, including alert relabeling, rather than assuming the raw metric’s labels survived unchanged.

Keep the root receiver useful for alerts that miss more specific children. An unmatched alert still needs an observable destination and an owner who can correct its classification. Test absent labels as well as recognized teams. A missing team on a warning can take the root route, while a critical alert without that team can still match a separate severity route.

Route order is part of behavior. A broad early route can prevent a narrower later route from being reached unless continuation is enabled. Moving a block of YAML can therefore change recipients without changing any matcher text. Review ordering changes like policy changes, and keep representative label fixtures beside the configuration so the intended selection remains explicit.

When it breaks

A typo or a label renamed by an exporter can send alerts to the root receiver. A new catch-all child can capture alerts intended for a later team. Excessive continuation can create multiple notifications for one event. Diagnose these failures with the actual labels and the entire matching path, not by reading one route in isolation.

Correct receiver selection still does not establish delivery. Grouping delays, silence or inhibition, template errors, transport failures, and destination behavior can intervene afterward. Inspect routing first, then follow the selected receiver through the notification pipeline. Increasing the rule’s waiting duration will not fix an incorrect team matcher, and reducing grouping delay will not repair missing credentials at a receiver.

Check it quickly

Save this as routing.py and run python3 routing.py. It requires Docker and the public Alertmanager 0.28.1 image, which may be downloaded if absent. The real amtool validates the configuration and checks seven label cases. Receivers deliberately have names without delivery integrations, so the experiment evaluates routing without contacting any person or external service.

import subprocess
import uuid

# Requires Docker and Alertmanager 0.28.1. No receiver has a delivery integration.
script = r"""set -eu
cat > /tmp/routes.yml <<'ROUTES'
route:
  receiver: default
  group_by: [alertname, service]
  routes:
    - receiver: payments
      matchers: ['team="payments"']
      continue: true
    - receiver: pager
      matchers: ['severity="critical"']
receivers:
  - name: default
  - name: payments
  - name: pager

ROUTES
/bin/amtool check-config /tmp/routes.yml
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=payments,pager team=payments severity=critical
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=payments team=payments severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=pager team=data severity=critical
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default team=data severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default team=payments-preview severity=warning
sed 's/continue: true/continue: false/' /tmp/routes.yml > /tmp/stop.yml
/bin/amtool config routes test --config.file=/tmp/stop.yml --verify.receivers=payments team=payments severity=critical
"""
name = "xbcatm-route-test-" + uuid.uuid4().hex[:12]
try:
    result = subprocess.run(
        ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
         "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-s"],
        input=script, text=True, capture_output=True, timeout=30,
    )
finally:
    subprocess.run(["docker", "rm", "-f", name],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
    raise RuntimeError(result.stderr or result.stdout)
print("Verified seven route cases, including continuation and unmatched labels.")

The first case resolves to payments,pager: the payments route matches and continues, then the critical route also matches. A payments warning selects only payments; a critical data-team alert selects pager. Unmatched and missing team values select the root receiver. The payments-preview value does not match the exact payments value.

The final check changes continuation to false and confirms that the payments critical alert now selects only payments. Each expected receiver set is enforced by --verify.receivers, so a mismatch fails execution. The container has no external network or host mounts and is removed on exit. Notification delivery requires a separate controlled test of the real receiver integration.

Sources