An alert should give its reader a useful first investigation, not merely a red indicator. Name the observed condition, affected scope, persistence window, responsible team, and next evidence to inspect. These details are part of the rule’s behavior: test them with the trigger instead of treating notification text as decoration added after the incident.

Separate observations from explanations

A failed scrape means Prometheus could not collect that target’s metrics. It does not, by itself, prove that the public API is unavailable. A service can answer users while its metrics handler fails, and a successful scrape can coexist with broken business requests. Combine symptom-oriented checks with internal evidence, and state precisely which observation triggered this alert.

Recent releases are useful investigation leads, not proof of causality. A certificate expiry, dependency failure, resource shortage, or network fault can occur without a deployment. Link the relevant change history alongside status, logs, and user-facing measurements. Do not label the latest commit as the cause before establishing the connection.

Make identity stable and context useful

Prometheus evaluates each resulting label set as an alert instance. Put stable routing attributes, such as service and team, in labels. Put a current value, explanation, and runbook link in annotations. A changing timestamp or measurement in a label creates changing identities and can prevent the pending duration from completing. An annotation can explain a changing situation without redefining which alert it is.

The runbook should begin with accessible, read-only checks: inspect the target’s reported error, compare the scrape endpoint with the public request path, then correlate relevant logs and recent changes. It should identify a healthy result, the next branch if that result is absent, and the point for escalation. Keep credentials and sensitive query parameters out of notification links. Confirm that the on-call reader can actually access the destination.

When it breaks

An alert can fire correctly but route nowhere, link to a removed runbook, or lose its instance label during aggregation. A noisy threshold interrupts people for conditions requiring no decision. Choose a page only when timely human action is justified; otherwise use the appropriate dashboard, ticket, or automated response. Grouping and inhibition in Alertmanager are separate from deciding whether a Prometheus rule is firing.

The optional for duration requires the expression to remain active across evaluations before firing. It is not a moving-average window and does not include notification delivery delays. If the expression stops producing the label set, this rule deactivates. Missing data is especially dangerous: a stale or absent series is not an explicit zero. Silence from the rule is therefore not sufficient proof of recovery; monitor collection and expected target inventory separately.

Check it quickly

Use promtool from a Prometheus release; this example was executed with version 3.14.0. In an empty local directory save the next two blocks as alerts.yml and alerts.test.yml. The rule deliberately models scrape availability, not an API availability SLO. Its severity and team labels are test data, and running these tests sends no notification.

groups:
  - name: api-scrape
    rules:
      - alert: APIScrapeUnavailable
        expr: up{job="api"} == 0
        for: 2m
        labels:
          severity: page
          team: platform
        annotations:
          summary: 'Cannot scrape {{ $labels.job }} on {{ $labels.instance }}'
          description: 'up == 0 for 2m. Inspect the target error and /metrics handler; scrape failure alone does not prove API downtime.'
          runbook_url: https://www.xbcatm.com/posts/a-purpose-built-health-check-is-worth-keeping/

The tests use synthetic one-minute samples. A zero begins pending at minute zero and becomes firing at minute two. Minute three is healthy; a later failure must wait through its own pending period. Expected annotations verify the expanded target name, troubleshooting text, and runbook URL rather than merely counting alerts.

rule_files:
  - alerts.yml
evaluation_interval: 1m
tests:
  - name: sustained failure, recovery, and recurrence
    interval: 1m
    input_series:
      - series: 'up{job="api",instance="api.example:9090"}'
        values: '0 0 0 1 0 0 0'
    alert_rule_test:
      - eval_time: 0m
        alertname: APIScrapeUnavailable
        exp_alerts: []
      - eval_time: 1m
        alertname: APIScrapeUnavailable
        exp_alerts: []
      - eval_time: 2m
        alertname: APIScrapeUnavailable
        exp_alerts: &firing
          - exp_labels:
              job: api
              instance: api.example:9090
              severity: page
              team: platform
            exp_annotations:
              summary: 'Cannot scrape api on api.example:9090'
              description: 'up == 0 for 2m. Inspect the target error and /metrics handler; scrape failure alone does not prove API downtime.'
              runbook_url: https://www.xbcatm.com/posts/a-purpose-built-health-check-is-worth-keeping/
      - eval_time: 3m
        alertname: APIScrapeUnavailable
        exp_alerts: []
      - eval_time: 5m
        alertname: APIScrapeUnavailable
        exp_alerts: []
      - eval_time: 6m
        alertname: APIScrapeUnavailable
        exp_alerts: *firing
    promql_expr_test:
      - expr: 'ALERTS{alertname="APIScrapeUnavailable",alertstate="pending"}'
        eval_time: 1m
        exp_samples:
          - labels: 'ALERTS{alertname="APIScrapeUnavailable",alertstate="pending",instance="api.example:9090",job="api",severity="page",team="platform"}'
            value: 1
  - name: transient failure does not fire
    interval: 1m
    input_series:
      - series: 'up{job="api",instance="api.example:9090"}'
        values: '1 0 1 1'
    alert_rule_test:
      - eval_time: 2m
        alertname: APIScrapeUnavailable
        exp_alerts: []
      - eval_time: 3m
        alertname: APIScrapeUnavailable
        exp_alerts: []
  - name: stale series is not an explicit zero
    interval: 1m
    input_series:
      - series: 'up{job="api",instance="api.example:9090"}'
        values: '0 stale _ _'
    alert_rule_test:
      - eval_time: 2m
        alertname: APIScrapeUnavailable
        exp_alerts: []
    promql_expr_test:
      - expr: 'ALERTS{alertname="APIScrapeUnavailable"}'
        eval_time: 2m
        exp_samples: []

Run these commands from that directory:

promtool --version
promtool check rules alerts.yml
promtool test rules alerts.test.yml

The rule check and unit tests must report SUCCESS. The cases cover sustained failure, pending state, recovery, recurrence, a transient failure, and a stale target that disappears before firing. The stale case intentionally proves a detection gap instead of pretending missing samples are healthy. All expected labels and annotations must match.

For a negative control, change the rule’s runbook URL while leaving the expectations unchanged; the test should fail. Restore it and repeat successfully. A changed for duration must likewise fail the early-time expectations. These checks validate rule evaluation and explanatory context, not live scraping, authorization on linked pages, Alertmanager grouping, or actual pager delivery. Exercise those separately before enabling the rule in production.

Further reading

Sources