Sre · XB Field Notes

Alertmanager routes alerts through a label-matching tree

Wed, 08 Dec 2027 00:00:00 +0000

Alertmanager selects notification receivers by walking a routing tree against an alert’s labels. The tree expresses ownership and escalation policy, while receivers describe notification integrations. A configuration can parse correctly and still send the wrong class of alert to the wrong destination. Verify the label contract and tree behavior before treating a successful reload as evidence that routing works.

Follow the tree, including its stopping rule

Every alert enters the root route, which must match all alerts. Child routes inherit settings they do not override. If no child matches, the current node handles the alert. Otherwise, traversal normally stops after the first matching child. Setting continue: true on a matching child allows subsequent siblings to be considered. It does not mean that every receiver in the configuration receives the alert.

One route node names one receiver. A receiver can contain multiple notification integrations, and multiple matching routes can also select different receivers. These are different ways to deliver to several destinations. A route cannot simply replace its receiver field with a list and expect the same behavior. Review the chosen approach for duplicate notifications and intentionally different grouping or escalation settings.

Ownership is a label contract

A route matching team="payments" needs that exact label value on the alert entering Alertmanager. An annotation named team does not satisfy the matcher. Aggregation in a Prometheus expression can remove a needed dimension before the alert reaches routing. Inspect the final alert label set, including alert relabeling, rather than assuming the raw metric’s labels survived unchanged.

Keep the root receiver useful for alerts that miss more specific children. An unmatched alert still needs an observable destination and an owner who can correct its classification. Test absent labels as well as recognized teams. A missing team on a warning can take the root route, while a critical alert without that team can still match a separate severity route.

Route order is part of behavior. A broad early route can prevent a narrower later route from being reached unless continuation is enabled. Moving a block of YAML can therefore change recipients without changing any matcher text. Review ordering changes like policy changes, and keep representative label fixtures beside the configuration so the intended selection remains explicit.

When it breaks

A typo or a label renamed by an exporter can send alerts to the root receiver. A new catch-all child can capture alerts intended for a later team. Excessive continuation can create multiple notifications for one event. Diagnose these failures with the actual labels and the entire matching path, not by reading one route in isolation.

Correct receiver selection still does not establish delivery. Grouping delays, silence or inhibition, template errors, transport failures, and destination behavior can intervene afterward. Inspect routing first, then follow the selected receiver through the notification pipeline. Increasing the rule’s waiting duration will not fix an incorrect team matcher, and reducing grouping delay will not repair missing credentials at a receiver.

Check it quickly

Save this as routing.py and run python3 routing.py. It requires Docker and the public Alertmanager 0.28.1 image, which may be downloaded if absent. The real amtool validates the configuration and checks seven label cases. Receivers deliberately have names without delivery integrations, so the experiment evaluates routing without contacting any person or external service.

import subprocess
import uuid

# Requires Docker and Alertmanager 0.28.1. No receiver has a delivery integration.
script = r"""set -eu
cat > /tmp/routes.yml <<'ROUTES'
route:
 receiver: default
 group_by: [alertname, service]
 routes:
 - receiver: payments
 matchers: ['team="payments"']
 continue: true
 - receiver: pager
 matchers: ['severity="critical"']
receivers:
 - name: default
 - name: payments
 - name: pager

ROUTES
/bin/amtool check-config /tmp/routes.yml
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=payments,pager team=payments severity=critical
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=payments team=payments severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=pager team=data severity=critical
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default team=data severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default severity=warning
/bin/amtool config routes test --config.file=/tmp/routes.yml --verify.receivers=default team=payments-preview severity=warning
sed 's/continue: true/continue: false/' /tmp/routes.yml > /tmp/stop.yml
/bin/amtool config routes test --config.file=/tmp/stop.yml --verify.receivers=payments team=payments severity=critical
"""
name = "xbcatm-route-test-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-s"],
 input=script, text=True, capture_output=True, timeout=30,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 raise RuntimeError(result.stderr or result.stdout)
print("Verified seven route cases, including continuation and unmatched labels.")

The first case resolves to payments,pager: the payments route matches and continues, then the critical route also matches. A payments warning selects only payments; a critical data-team alert selects pager. Unmatched and missing team values select the root receiver. The payments-preview value does not match the exact payments value.

The final check changes continuation to false and confirms that the payments critical alert now selects only payments. Each expected receiver set is enforced by --verify.receivers, so a mismatch fails execution. The container has no external network or host mounts and is removed on exit. Notification delivery requires a separate controlled test of the real receiver integration.

Sources

Alertmanager: configuration, route inheritance, sibling continuation and receiver definitions, checked 2026-09-13.
Alertmanager 0.28.1: README, amtool configuration and routing checks, checked 2026-09-13.
Prometheus: Alertmanager, separation of routing, grouping and suppression, checked 2026-09-13.

The horizontal autoscaler scales on demand

Thu, 04 Nov 2027 00:00:00 +0000

The autoscaler derives desired replicas from current usage and a target, so the metric you choose matters more than the replica count you set.

Metrics drive the count

The HPA does not watch load directly; it polls a configured metric on a schedule, reads a current value against the target you set, and derives a desired replica count from the ratio. If the observed value runs twice the target it scales up, if half it scales down, and the result is recomputed on each reconciliation. That is why the metric choice, its aggregation (per pod or cluster-level), and the window matter, they define what the load even is. The desired count is then clamped by maxReplicas and minReplicas, and reconciled on the default --horizontal-pod-autoscaler-sync-period. To make scaling predictable you reason about the input metric, not about a single constant.

Scale up eagerly, down gently

Scale-down is deliberately asymmetric. The controller does not remove a pod the instant utilization dips, because a transient spike would otherwise ping-pong the replica count and thrash the cluster. It waits out a stabilization window set by --horizontal-pod-autoscaler-downscale-stabilization, keeps a floor via minReplicas, and only then shrinks. Scaling up is fast, because queued latency hurts right now; scaling down is slow, because idle capacity is cheaper than a flapping fleet. A dependable setup states a floor generous enough for the spikes you actually see, a target it can digest, and treats the stabilization window as the amount of load noise you are willing to ride out, so the group does not bounce every time the graph blips.

The metric is the weak link

The whole controller inherits the quality of your metric. If the metric lags real demand, the controller is always chasing a stale number; if it watches a proxy that correlates poorly with the actual bottleneck, the controller scales the wrong thing. So the hard part of an HPA is rarely the YAML, it is the metric and the target. Putting CPU on a latency-bound service, or memory on an I-O-bound worker, often points scaling in the wrong direction. Begin from what actually saturates, requests in flight, queue depth, or connections, choose an average over a window you can act on, and prove that a rise in that metric really predicts worse latency. Then the HPA becomes a tool that scales your workload, not a timer that wastes a guess.

When it breaks

Stale or missing metrics make the controller flap. When the metrics pipeline lags, HPA scales against last minute’s load: it adds replicas after the spike already passed, then holds them through the downscale stabilization window, so dashboards show oscillation rather than tracking. If metrics go missing entirely, the controller refuses to scale and surfaces ScalingActive=False; treating “no data” as “no load” and deleting capacity is the classic outage version of this bug.

Bounds freeze the loop in the other direction. A maxReplicas set from cost anxiety caps the fleet during a real surge and reports ScalingLimited=True, while a generous minReplicas hides an overestimated target because the floor never lets the controller prove demand fell. Finally, do not let two controllers steer the same knob: HPA fighting VPA (one changes count, the other changes requests, which moves utilization) or an aggressive rolling update (new pods report zero usage while old pods still serve) produces sawtooth replicas that no target tuning will fix.

Check it quickly

Save the following as hpa_loop.py in a scratch directory and run python3 hpa_loop.py. It models one HPA control loop with pure standard library: desired replicas are the ceiling of current average utilization over target, clamped to min and max, with missing metrics holding the last decision and a downscale stabilization window blocking fast shrink. The script asserts scale-up on a hot sample, cooldown-blocked scale-down on the next cool sample, eventual shrink after the window, and clamping at both bounds.

import math

TARGET = 50.0
MIN_REPLICAS = 2
MAX_REPLICAS = 10
STABILIZATION_S = 300


def desired(current_replicas, current_util):
 raw = math.ceil(current_replicas * current_util / TARGET)
 return max(MIN_REPLICAS, min(MAX_REPLICAS, raw))


def step(state, now, current_util):
 replicas, last_scale = state
 if current_util is None:
 print(f"t={now}s metric=missing hold={replicas}")
 return state
 want = desired(replicas, current_util)
 if want < replicas and now - last_scale < STABILIZATION_S:
 print(f"t={now}s metric={current_util} want={want} hold={replicas} (cooldown)")
 return state
 print(f"t={now}s metric={current_util} {replicas}->{want}")
 if want != replicas:
 return (want, now)
 return state


def main():
 state = (3, -10**9)
 state = step(state, 0, 100)
 assert state[0] == 6, state
 state = step(state, 60, 15)
 assert state[0] == 6, state
 state = step(state, 400, 15)
 assert state[0] == 2, state
 state = step(state, 460, None)
 assert state[0] == 2, state
 assert desired(2, 500) == 10
 assert desired(2, 5) == 2
 print("PASS: scale-up 3->6, cooldown holds 6, then 6->2; clamps at 10 and 2")


if __name__ == "__main__":
 main()

The run prints each decision and ends with PASS. The hot sample grows 3 replicas to 6, the immediate cool sample stays at 6 while the stabilization window is fresh, the same cool value shrinks to 2 after the window expires, and out-of-range calculations stick at max 10 and min 2. The fixture checks control-loop math, not metrics-server latency or startup time. Validate cluster tuning against kubectl describe hpa conditions.

Sources

Kubernetes horizontal pod autoscaling, how HPA derives replica counts, checked 2026-05-06.
HorizontalPodAutoscaler walkthrough, worked example with multiple metrics and status conditions, checked 2026-05-06.

Error budgets are a real budget

Thu, 25 Mar 2027 00:00:00 +0000

An error budget makes a reliability agreement usable during planning. Product and operations choose an objective, measure failures against it, and agree what changes when the allowance is exhausted. The number supports that agreement; it does not predict whether tomorrow’s deployment will cause an incident. Shipping a release creates risk, while observed bad events consume the measured budget.

Give the account a measurement rule

For a request-based objective, allowed bad events equal eligible requests multiplied by one minus the target. A 99.9% objective over three million requests permits three thousand bad requests. Specify which requests count, which outcomes are bad, the measurement location, and the window before attaching a release rule. A traffic-weighted allowance is not automatically a fixed number of downtime minutes.

A rolling thirty-day account drops old observations as new ones arrive. Its denominator changes when traffic changes, so the allowance can change even when the target stays fixed. A calendar-month account resets at a calendar boundary instead. Those approaches answer different questions; switching between them during an incident makes an apparently precise balance difficult to audit. Keep the definition versioned with the policy and show the window beside the balance.

Decide the response before the argument

An example policy might hold nonessential feature releases when the budget is overspent, prioritize reliability work, and allow urgent security or incident fixes through review. Name who owns the decision, what evidence permits an exception, and when ordinary release review can resume. Google’s published example is a policy for its example service, not a universal requirement to copy every threshold or escalation rule.

Remaining budget alone does not establish release readiness. An open incident, missing monitoring, a failed regression, or unsafe rollback can still prevent a change. Conversely, a necessary repair should not be blocked merely because the incident it repairs exhausted the budget. Record these decisions explicitly. The useful result is a shared response to customer impact, not an automated mechanism for blaming the team whose dashboard turned red.

When it breaks

A team can appear to recover simply because yesterday’s large incident leaves the rolling window. The arithmetic is correct, but it says nothing about whether the defect was repaired. Changing exclusions, silently replacing unavailable samples with successes, or averaging differently sized service percentages can also create a misleading surplus. Display measurement coverage and investigate definition changes before interpreting an improved balance.

Short-window burn rate supplies another view: the observed bad-event fraction divided by the allowed fraction. A high burn can warn before the longer account is exhausted. Estimating time to exhaustion from that rate assumes comparable traffic and continued behavior; it is not a promised deadline. Pair actionable alerts with the long-window account and account for sparse traffic, rather than paging on every isolated failure.

Check it quickly

Save the following as budget-policy.py and run it with Python 3. It uses synthetic daily request counts to verify rolling-window accounting and a deliberately small example policy. It does not change a deployment system or estimate the risk of a real release.

from fractions import Fraction

target = Fraction(999, 1000)
# Every synthetic daily bin has 100,000 eligible requests.
days = [(100000, 1300)] + [(100000, 70)] * 29

def account(window):
 eligible = sum(total for total, _ in window)
 bad = sum(errors for _, errors in window)
 allowed = eligible * (1 - target)
 return bad, allowed, allowed - bad

def feature_policy(remaining, incident_open):
 return "hold" if remaining < 0 or incident_open else "eligible for review"

before = account(days)
after = account(days[1:] + [(100000, 50)])
assert before == (3330, 3000, -330)
assert after == (2080, 3000, 920)
assert feature_policy(before[2], False) == "hold"
assert feature_policy(after[2], True) == "hold"
assert feature_policy(after[2], False) == "eligible for review"
for label, (bad, allowed, remaining) in [("before", before), ("after", after)]:
 print(f"{label}: bad={bad}, allowed={float(allowed):.0f}, remaining={float(remaining):.0f}, spent={float(bad / allowed):.2%}")
print("after rotation, incident still open:", feature_policy(after[2], True))
print("after rotation, incident closed:", feature_policy(after[2], False))

The first account has 3,330 bad requests against an allowance of 3,000: 111% spent and a balance of minus 330. Removing the oldest day and adding the new day leaves 2,080 bad requests, 69.33% spent, and 920 remaining. Both windows contain the same three million eligible requests in this fixture; real traffic need not stay constant.

The policy still prints hold while an incident is open, despite the positive new balance. Only the final case becomes eligible for review, which intentionally stops short of granting deployment approval. This separates a verified calculation from the operational judgment that uses it. Review the definition, coverage, incident evidence, and release checks together before acting on the account.

Sources

Google SRE Workbook: Example Error Budget Policy, release restrictions, exceptions, and ownership, checked 2026-09-13.
Google SRE: Service Level Objectives, measurement boundaries and reliability targets, checked 2026-09-13.
Google SRE Workbook: Alerting on SLOs, burn rate and alerting windows, checked 2026-09-13.

An error budget names the failures you allow

Thu, 17 Sep 2026 00:00:00 +0000

An objective such as “99.9% reliable” is incomplete until someone can classify an observation. Reliable for which users, at which boundary, over what window, and according to which outcome? An error budget gives the permitted failures a quantity only after those questions have answers. Its first engineering task is defining the population and the good-event rule, not choosing a dashboard color.

Count the experience you promise

A service level indicator is a measured quantity; an objective specifies its target and window. For an event-based indicator, divide good eligible events by all eligible events. “Good” might mean a completed business operation within a latency threshold. An HTTP success code alone cannot prove that a checkout was recorded correctly, and an internal server timer may omit the network delay experienced by a customer.

Write the denominator explicitly. It might include all valid customer attempts to one API, while excluding documented synthetic probes. Decide how retries, client cancellations, authentication failures, timeouts, and asynchronous completion are represented. Exclusions should follow the promised service boundary, not be invented after an incident to improve the score. Retain enough evidence to distinguish a deliberately excluded event from one the collector failed to observe.

One bad event can fail several tests

Suppose a good request must return the expected response within 300 milliseconds. A slow server error fails both requirements but remains one bad request. Count the union of failure conditions, or equivalently the complement of the good predicate. Adding separate slow-request and error counters can double-count overlapping failures unless the categories are explicitly disjoint.

For a target S and N eligible requests, the allowance is N multiplied by one minus S. Remaining budget subtracts measured bad requests from that allowance. A request-based budget tracks events, whereas a time-based budget tracks time in a defined state. A 99.9% time objective over thirty days corresponds to 43.2 minutes of bad time; that conversion does not describe arbitrary request traffic. Partial failures and changing load break that equivalence.

When it breaks

Server-only records can miss requests rejected before reaching the application. Measurements of completed requests can also omit requests still hung in a queue. A beautiful ratio over this incomplete population may coexist with a customer outage. Measure at a suitable boundary, include timeout outcomes, and expose coverage or independent checks rather than treating unavailable telemetry as successful service.

Zero eligible events do not produce a meaningful success fraction. Treat the window as having no observations and handle monitoring absence separately. Very small populations also make high targets sensitive to individual failures: the allowance is a mathematical quantity and need not be a whole event. Do not round it upward to manufacture tolerance. Keep the same population and weighting when comparing instances; a simple average of their percentages changes the meaning.

Check it quickly

Save this as slo-population.py and run Python 3. The synthetic API contract accepts only status 200 within 0.300 seconds. The fixture contains completed customer events and separately identified probes; it tests the classification math, not the completeness of a real collector or the business correctness of an HTTP response.

from fractions import Fraction

# Synthetic events: (kind, status, duration_seconds).
events = ([('user', 200, 0.10)] * 9970
 + [('user', 200, 0.50)] * 10
 + [('user', 503, 0.02)] * 10
 + [('user', 503, 0.50)] * 10
 + [('probe', 503, 0.50)] * 100)
eligible = [(status, duration) for kind, status, duration in events if kind == 'user']
bad_status = sum(status != 200 for status, _ in eligible)
too_slow = sum(duration > 0.30 for _, duration in eligible)
bad = sum(status != 200 or duration > 0.30 for status, duration in eligible)
good = len(eligible) - bad
target = Fraction(999, 1000)
allowed = len(eligible) * (1 - target)
remaining = allowed - bad

assert len(eligible) == 10000 and bad_status == too_slow == 20
assert bad == 30 and bad_status + too_slow == 40
assert allowed == 10 and remaining == -20
assert Fraction(good, len(eligible)) == Fraction(997, 1000)
print(f"eligible={len(eligible)}; good={good}; bad_union={bad}")
print(f"status_failures={bad_status}; slow={too_slow}; wrong_sum={bad_status + too_slow}")
print(f"SLI={good / len(eligible):.3%}; allowed={float(allowed):.0f}; remaining={float(remaining):.0f}")

There are 10,000 eligible customer events and 100 excluded probes. Twenty eligible events have an unexpected status and twenty are slow, but ten belong to both sets. The correct union contains thirty bad events, leaving 9,970 good ones. Adding the two failure counts would incorrectly report forty.

The printed SLI is 99.700%, below the 99.9% target. The request allowance is ten and the balance is minus twenty. Those are event counts, not outage seconds. Inspecting individual categories remains useful for diagnosis, but the objective must use its single declared predicate. Before connecting this calculation to alerts or a release policy, verify that the production counter uses the same scope, threshold boundary, window, and missing-data treatment.

Sources

Google SRE: Service Level Objectives, SLI populations, measurement, and targets, checked 2026-09-13.
Google SRE: Monitoring Distributed Systems, explicit and implicit request failures, checked 2026-09-13.
Google SRE Workbook: Alerting on SLOs, event-based objectives and budget consumption, checked 2026-09-13.

Sre · XB Field Notes

Alertmanager routes alerts through a label-matching tree

Follow the tree, including its stopping rule

Ownership is a label contract

When it breaks

Check it quickly

Sources

The horizontal autoscaler scales on demand

Metrics drive the count

Scale up eagerly, down gently

The metric is the weak link

When it breaks

Check it quickly

Further reading

Sources

Error budgets are a real budget

Give the account a measurement rule

Decide the response before the argument

When it breaks

Check it quickly

Further reading

Sources

An error budget names the failures you allow

Count the experience you promise

One bad event can fail several tests

When it breaks

Check it quickly

Sources