A recording rule runs a PromQL expression at scheduled evaluation times and stores the result as a new metric. Repeated dashboard queries can then read that metric instead of recomputing the expression. Calling this a cache can obscure an important difference: the recorded series has its own sample history, cadence, and labels. It is not a transparent cache of every possible query request.

Three clocks influence the displayed value

Scraping determines when raw observations enter storage. Rule evaluation determines when an expression produces recorded samples. A dashboard chooses when and at what step to query either series. Refreshing a panel more frequently does not make a recording rule evaluate sooner. Reading the recorded metric between evaluations can therefore return an older value than evaluating the raw expression at that moment.

A rule group has an interval, defaulting to the global evaluation interval when not specified. Rules within a group run in order with the same evaluation timestamp. That permits a later rule to consume an earlier rule’s output. Separate groups do not provide that same sequencing guarantee merely because their interval strings match. Design dependencies explicitly rather than relying on coincidental execution order.

Precomputation changes the data you retain

Aggregation determines the dimensions of the output. If a rule sums away instance, dashboards cannot later recover an instance breakdown from that recorded series alone. Keep the raw metric when that investigation matters, and choose the recorded dimensions for a known consumer. Each remaining label combination creates another stored series, so recording every conceivable aggregation can increase ingestion and storage substantially.

Changing an expression affects subsequent evaluations. A normal rule reload does not recalculate all earlier samples under the new definition or fill history from before the rule existed. A semantic change under an unchanged metric name can place incompatible definitions on one graph. Document the definition and window, and use a deliberate migration when consumers require historical comparability.

When it breaks

A dashboard may appear frozen even though raw samples keep arriving. Compare the last recorded sample with the rule group’s last evaluation, duration, and errors. If a group is still evaluating when its next iteration is due, subsequent evaluations are skipped until the current one finishes or times out. Recording a costly query moves work into a scheduler; it does not make that work free.

A query can also be quick because it answers a narrower question after aggregation. Check that the removed dimensions were intentional before celebrating the speedup. A trivial expression rarely needs a recording rule solely for performance. Measure the repeated query cost and expected reuse, then account for the extra series, rule execution, and freshness delay in the decision.

Check it quickly

Save the program as recording-cadence.py and run python3 recording-cadence.py. It requires Docker and the public Prometheus 3.5.1 image, which Docker can fetch if missing. The real promtool engine receives queue values one minute apart. Both the unit-test evaluation step and the rule group’s interval are two minutes, deliberately separating observations from precomputation.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 2m
rule_files:
  - /tmp/rules.yml
tests:
  - interval: 1m
    input_series:
      - series: "queue_depth{queue=\"orders\"}"
        values: "1 4 9 16"
    promql_expr_test:
      - expr: "sum(queue_depth)"
        eval_time: 1m
        exp_samples:
          - labels: "{}"
            value: 4
      - expr: "queue:depth:sum"
        eval_time: 1m
        exp_samples:
          - labels: "{__name__=\"queue:depth:sum\"}"
            value: 1
      - expr: "queue:depth:sum"
        eval_time: 2m
        exp_samples:
          - labels: "{__name__=\"queue:depth:sum\"}"
            value: 9
"""
rules = r"""groups:
  - name: slow-recording
    interval: 2m
    rules:
      - record: queue:depth:sum
        expr: sum(queue_depth)
"""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
    script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
    result = subprocess.run(
        ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
         "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
        input=script, text=True, capture_output=True, timeout=45,
    )
finally:
    subprocess.run(["docker", "rm", "-f", name],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
    print(result.stderr, end="")
    raise SystemExit(result.returncode)

The expected output is SUCCESS. At minute one, the raw sum is four while the recorded value is still one from minute zero. At minute two, the recorded series becomes nine. The expression is intentionally cheap so the example isolates cadence; it is not a recommendation to precompute every queue gauge, and it does not benchmark a live rule scheduler.

The container has no external network, no host mounts, and a unique cleanup name. These assertions establish that recorded and on-demand values can legitimately differ between evaluation times. For an actual deployment, verify rule loading, execution duration, output cardinality, and consumer queries. A successful file reload alone does not establish that a new metric has accumulated the history needed by a long-range dashboard.

Sources