Recording · XB Field Notes

Recording rules store scheduled query results

Wed, 26 Jan 2028 00:00:00 +0000

A recording rule runs a PromQL expression at scheduled evaluation times and stores the result as a new metric. Repeated dashboard queries can then read that metric instead of recomputing the expression. Calling this a cache can obscure an important difference: the recorded series has its own sample history, cadence, and labels. It is not a transparent cache of every possible query request.

Three clocks influence the displayed value

Scraping determines when raw observations enter storage. Rule evaluation determines when an expression produces recorded samples. A dashboard chooses when and at what step to query either series. Refreshing a panel more frequently does not make a recording rule evaluate sooner. Reading the recorded metric between evaluations can therefore return an older value than evaluating the raw expression at that moment.

A rule group has an interval, defaulting to the global evaluation interval when not specified. Rules within a group run in order with the same evaluation timestamp. That permits a later rule to consume an earlier rule’s output. Separate groups do not provide that same sequencing guarantee merely because their interval strings match. Design dependencies explicitly rather than relying on coincidental execution order.

Precomputation changes the data you retain

Aggregation determines the dimensions of the output. If a rule sums away instance, dashboards cannot later recover an instance breakdown from that recorded series alone. Keep the raw metric when that investigation matters, and choose the recorded dimensions for a known consumer. Each remaining label combination creates another stored series, so recording every conceivable aggregation can increase ingestion and storage substantially.

Changing an expression affects subsequent evaluations. A normal rule reload does not recalculate all earlier samples under the new definition or fill history from before the rule existed. A semantic change under an unchanged metric name can place incompatible definitions on one graph. Document the definition and window, and use a deliberate migration when consumers require historical comparability.

When it breaks

A dashboard may appear frozen even though raw samples keep arriving. Compare the last recorded sample with the rule group’s last evaluation, duration, and errors. If a group is still evaluating when its next iteration is due, subsequent evaluations are skipped until the current one finishes or times out. Recording a costly query moves work into a scheduler; it does not make that work free.

A query can also be quick because it answers a narrower question after aggregation. Check that the removed dimensions were intentional before celebrating the speedup. A trivial expression rarely needs a recording rule solely for performance. Measure the repeated query cost and expected reuse, then account for the extra series, rule execution, and freshness delay in the decision.

Check it quickly

Save the program as recording-cadence.py and run python3 recording-cadence.py. It requires Docker and the public Prometheus 3.5.1 image, which Docker can fetch if missing. The real promtool engine receives queue values one minute apart. Both the unit-test evaluation step and the rule group’s interval are two minutes, deliberately separating observations from precomputation.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 2m
rule_files:
 - /tmp/rules.yml
tests:
 - interval: 1m
 input_series:
 - series: "queue_depth{queue=\"orders\"}"
 values: "1 4 9 16"
 promql_expr_test:
 - expr: "sum(queue_depth)"
 eval_time: 1m
 exp_samples:
 - labels: "{}"
 value: 4
 - expr: "queue:depth:sum"
 eval_time: 1m
 exp_samples:
 - labels: "{__name__=\"queue:depth:sum\"}"
 value: 1
 - expr: "queue:depth:sum"
 eval_time: 2m
 exp_samples:
 - labels: "{__name__=\"queue:depth:sum\"}"
 value: 9
"""
rules = r"""groups:
 - name: slow-recording
 interval: 2m
 rules:
 - record: queue:depth:sum
 expr: sum(queue_depth)
"""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
 script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

The expected output is SUCCESS. At minute one, the raw sum is four while the recorded value is still one from minute zero. At minute two, the recorded series becomes nine. The expression is intentionally cheap so the example isolates cadence; it is not a recommendation to precompute every queue gauge, and it does not benchmark a live rule scheduler.

The container has no external network, no host mounts, and a unique cleanup name. These assertions establish that recorded and on-demand values can legitimately differ between evaluation times. For an actual deployment, verify rule loading, execution duration, output cardinality, and consumer queries. A successful file reload alone does not establish that a new metric has accumulated the history needed by a long-range dashboard.

Sources

Prometheus: recording rules, stored evaluations, group intervals and missed iterations, checked 2026-09-13.
Prometheus: configuration, separate scrape and evaluation settings, checked 2026-09-13.
Prometheus: unit testing rules, test evaluation cadence and input intervals, checked 2026-09-13.

Recording rules should preserve the components of a ratio

Tue, 31 Aug 2027 00:00:00 +0000

Repeated service-level queries are good candidates for recording rules when their cost and reuse justify precomputation. The design question is what to store. Saving only an instance’s error percentage can discard the traffic volume needed for later aggregation. Saving error rates and request rates preserves the numerator and denominator, allowing consumers to calculate a correctly weighted service fraction.

A percentage needs its population

Suppose a busy instance handles ten requests per second with 0.1 errors per second. A quiet instance handles one request per second with 0.5 errors per second. Their individual error fractions are one percent and fifty percent. Averaging those fractions gives 25.5 percent, which weights instances equally even though their request volumes differ by a factor of ten.

The service sees eleven requests and 0.6 errors per second. Its error fraction is 0.6/11, approximately 5.45 percent. This is the ratio of aggregate errors to aggregate requests. Both measurements must describe the same eligible traffic population and time window. A faster dashboard is still wrong if its numerator excludes failures that its denominator includes, or the two sides aggregate different services.

Build the rule dependency in order

Start with rate on each counter series so observed resets are handled before aggregation. Then sum by the dimensions consumers actually need. The example first records request rates by service and instance, then service request rates, service error rates, and finally the service error fraction. The intermediate instance rate is included to demonstrate an explicit dependency consumed by a later rule.

All four rules share a group and run in declaration order with the same evaluation time. Matching service labels on numerator and denominator make their division pair the intended series. A denominator retaining an extra instance label would change vector matching and could remove results or require a different join. Test the actual output labels as well as the numeric answer.

Keep names descriptive about aggregation level and range. A two-minute rate and a thirty-minute rate are different quantities even when their current values happen to match. Record the eligible event definition beside the rule and identify which dashboards or alerts consume it. Changing the formula under a stable name changes future samples, not the meaning of already stored history.

When it breaks

An average of percentages can make a small failing replica dominate a fleet report or conceal a busy failing replica. Increasing the evaluation frequency does not repair that arithmetic. Recover the underlying counts or rates and divide after aggregation. Once only percentages remain, the original traffic weights generally cannot be reconstructed from those percentages alone.

Zero traffic is another distinct case. A zero denominator does not define a zero error fraction, and a missing numerator may mean absent instrumentation rather than no errors. Decide those semantics explicitly with the metric contract. Avoid automatically filling every empty or undefined result with a healthy zero. Validate that errors are exported consistently, including during healthy operation.

Check it quickly

Save this as recording-weighted.py and run python3 recording-weighted.py. Docker and the Prometheus 3.5.1 image are required; the image may be downloaded on first use. Real promtool evaluates four recording rules over synthetic minute-spaced counter samples. The fixture deliberately gives its two instances very different traffic weights and error fractions.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
rule_files:
 - /tmp/rules.yml
tests:
 - interval: 1m
 input_series:
 - series: "http_requests_total{service=\"checkout\",instance=\"busy\"}"
 values: "0+600x5"
 - series: "http_errors_total{service=\"checkout\",instance=\"busy\"}"
 values: "0+6x5"
 - series: "http_requests_total{service=\"checkout\",instance=\"quiet\"}"
 values: "0+60x5"
 - series: "http_errors_total{service=\"checkout\",instance=\"quiet\"}"
 values: "0+30x5"
 promql_expr_test:
 - expr: "service:http_requests:rate2m"
 eval_time: 5m
 exp_samples:
 - labels: "{__name__=\"service:http_requests:rate2m\",service=\"checkout\"}"
 value: 11
 - expr: "abs(service:http_errors:rate2m - 0.6) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{service=\"checkout\"}"
 value: 1
 - expr: "abs(service:http_errors:ratio_rate2m - 0.6/11) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{service=\"checkout\"}"
 value: 1
 - expr: "abs(avg(rate(http_errors_total[2m]) / rate(http_requests_total[2m])) - 0.255) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
"""
rules = r"""groups:
 - name: service-rates
 interval: 1m
 rules:
 - record: instance:http_requests:rate2m
 expr: sum by (service, instance) (rate(http_requests_total[2m]))
 - record: service:http_requests:rate2m
 expr: sum by (service) (instance:http_requests:rate2m)
 - record: service:http_errors:rate2m
 expr: sum by (service) (rate(http_errors_total[2m]))
 - record: service:http_errors:ratio_rate2m
 expr: service:http_errors:rate2m / service:http_requests:rate2m
"""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
 script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

The expected output is SUCCESS. The recorded request rate is eleven, the error rate is 0.6, and the resulting fraction is 0.6/11. A separate assertion demonstrates that the incorrect average of instance fractions is 0.255. Numerical comparisons use a small tolerance where floating-point division is involved, and label assertions retain the checkout service identity.

The disposable container has no external network or host mounts. This is an arithmetic and dependency test, not a performance benchmark or proof of production instrumentation completeness. Before deploying the rules, add empty-traffic, counter-reset, and missing-series cases reflecting the actual service. Measure query cost and rule execution time to establish whether precomputation delivers the intended operational benefit.

Sources

Prometheus: recording rules, ordered rule dependencies and stored output, checked 2026-09-13.
Prometheus: query functions, rate before aggregation for counter resets, checked 2026-09-13.
Prometheus: unit testing rules, testing recorded series and PromQL results, checked 2026-09-13.