A missing metric is not a numeric zero. Prometheus distinguishes a series with a recent value, a series marked stale, and a selector with no usable sample. Alerting then applies its own evaluation and waiting rules. Treating all these mechanisms as one universal grace period produces alerts that miss disappearing targets or report an outage without establishing what actually disappeared.

A marker is not a five-minute timer

An instant selector normally uses the newest eligible sample within its lookback period, which defaults to five minutes. A stale marker tells the engine to stop returning that series in instant selections after the marker. It does not require the previous value to remain visible for another five minutes. Historical samples are still stored and can remain relevant to range queries.

Collection circumstances matter. A series no longer exported, a target removed from discovery, and an exporter assigning its own timestamps need not follow identical timing. Explicit timestamps and staleness configuration can alter behavior. Determine which path your exporter uses before predicting disappearance time from the scrape interval alone. A configurable lookback is a query selection rule, not a universal service outage detector.

Choose the absence question deliberately

absent(selector) returns a value of one when the selector has no elements. It returns an empty vector when matching data exists, even if that data’s numeric value is zero. absent_over_time(selector[3m]) asks whether the selected range contains any samples. A real sample from before a stale marker can keep that range-based absence expression empty until the sample leaves the window.

The alert rule’s for duration is another layer: the expression must keep returning the relevant alert label set at successive evaluations before it fires. It does not configure staleness or extend the query’s lookback. Combining a three-minute absence window with a waiting duration delays notification further, with evaluation cadence and collection timing also contributing. Choose that delay from the detection objective.

When it breaks

absent(up{job="api"}) is not a failed-scrape alert. If the configured targets are present with up=0, the selector still has elements, so absence is false. A comparison against zero addresses that case. If a target disappears from discovery altogether, checking only zero can miss it once its series stops appearing. Monitor expected membership separately when that distinction matters.

A broad selector also cannot enumerate missing members. If one replica survives, absent over the whole job remains empty even when several expected replicas are gone. Match a deliberately known identity or compare observed series with an authoritative inventory. A label rename can look like an outage under the old matcher while the process continues working under a new identity.

Check it quickly

Save the following as stale.py and run python3 stale.py with Docker available. The program uses the public Prometheus 3.5.1 image and real promtool queries. Docker may download an uncached image. Synthetic samples include an explicit stale marker, missing positions, and ordinary zeros; this is a query-semantics test, not a measurement of live discovery timing.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: "up{job=\"api\",instance=\"gone\"}"
        values: "1 0 0 0 0"
      - series: "service_events_total{job=\"api\",instance=\"gone\"}"
        values: "10 stale _ _ _"
      - series: "service_events_total{job=\"api\",instance=\"zero\"}"
        values: "0 0 0 0 0"
    promql_expr_test:
      - expr: "service_events_total{job=\"api\",instance=\"gone\"}"
        eval_time: 1m
        exp_samples: []
      - expr: "up{job=\"api\",instance=\"gone\"}"
        eval_time: 1m
        exp_samples:
          - labels: "{__name__=\"up\",job=\"api\",instance=\"gone\"}"
            value: 0
      - expr: "absent(service_events_total{job=\"api\",instance=\"gone\"})"
        eval_time: 1m
        exp_samples:
          - labels: "{job=\"api\",instance=\"gone\"}"
            value: 1
      - expr: "absent_over_time(service_events_total{job=\"api\",instance=\"gone\"}[3m])"
        eval_time: 2m
        exp_samples: []
      - expr: "absent_over_time(service_events_total{job=\"api\",instance=\"gone\"}[3m])"
        eval_time: 4m
        exp_samples:
          - labels: "{job=\"api\",instance=\"gone\"}"
            value: 1
      - expr: "absent(service_events_total{job=\"api\",instance=\"zero\"})"
        eval_time: 4m
        exp_samples: []
      - expr: "absent(up{job=\"api\"})"
        eval_time: 4m
        exp_samples: []
"""
rules = r""""""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
    script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
    result = subprocess.run(
        ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
         "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
        input=script, text=True, capture_output=True, timeout=45,
    )
finally:
    subprocess.run(["docker", "rm", "-f", name],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
    print(result.stderr, end="")
    raise SystemExit(result.returncode)

The test prints SUCCESS. At minute one, the marked business series is absent from an instant selection, while its separately supplied up series exists with value zero. Instant absent returns one. At minute two, the three-minute range still contains the original business sample, so absent_over_time returns nothing; at minute four it returns one.

The continuously exported zero-valued business series is never absent, and the job-wide up selector is not absent either. These assertions distinguish missing values from failing values instead of converting both to zero. The container has no network or host mounts and is removed on exit. For a production rule, supplement these cases with expected target membership and an independently tested pending-to-firing timeline.

Sources