Recording rules should preserve the components of a ratio

Tue, 31 Aug 2027 00:00:00 +0000

Repeated service-level queries are good candidates for recording rules when their cost and reuse justify precomputation. The design question is what to store. Saving only an instance’s error percentage can discard the traffic volume needed for later aggregation. Saving error rates and request rates preserves the numerator and denominator, allowing consumers to calculate a correctly weighted service fraction.

A percentage needs its population

Suppose a busy instance handles ten requests per second with 0.1 errors per second. A quiet instance handles one request per second with 0.5 errors per second. Their individual error fractions are one percent and fifty percent. Averaging those fractions gives 25.5 percent, which weights instances equally even though their request volumes differ by a factor of ten.

The service sees eleven requests and 0.6 errors per second. Its error fraction is 0.6/11, approximately 5.45 percent. This is the ratio of aggregate errors to aggregate requests. Both measurements must describe the same eligible traffic population and time window. A faster dashboard is still wrong if its numerator excludes failures that its denominator includes, or the two sides aggregate different services.

Build the rule dependency in order

Start with rate on each counter series so observed resets are handled before aggregation. Then sum by the dimensions consumers actually need. The example first records request rates by service and instance, then service request rates, service error rates, and finally the service error fraction. The intermediate instance rate is included to demonstrate an explicit dependency consumed by a later rule.

All four rules share a group and run in declaration order with the same evaluation time. Matching service labels on numerator and denominator make their division pair the intended series. A denominator retaining an extra instance label would change vector matching and could remove results or require a different join. Test the actual output labels as well as the numeric answer.

Keep names descriptive about aggregation level and range. A two-minute rate and a thirty-minute rate are different quantities even when their current values happen to match. Record the eligible event definition beside the rule and identify which dashboards or alerts consume it. Changing the formula under a stable name changes future samples, not the meaning of already stored history.

When it breaks

An average of percentages can make a small failing replica dominate a fleet report or conceal a busy failing replica. Increasing the evaluation frequency does not repair that arithmetic. Recover the underlying counts or rates and divide after aggregation. Once only percentages remain, the original traffic weights generally cannot be reconstructed from those percentages alone.

Zero traffic is another distinct case. A zero denominator does not define a zero error fraction, and a missing numerator may mean absent instrumentation rather than no errors. Decide those semantics explicitly with the metric contract. Avoid automatically filling every empty or undefined result with a healthy zero. Validate that errors are exported consistently, including during healthy operation.

Check it quickly

Save this as recording-weighted.py and run python3 recording-weighted.py. Docker and the Prometheus 3.5.1 image are required; the image may be downloaded on first use. Real promtool evaluates four recording rules over synthetic minute-spaced counter samples. The fixture deliberately gives its two instances very different traffic weights and error fractions.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
rule_files:
 - /tmp/rules.yml
tests:
 - interval: 1m
 input_series:
 - series: "http_requests_total{service=\"checkout\",instance=\"busy\"}"
 values: "0+600x5"
 - series: "http_errors_total{service=\"checkout\",instance=\"busy\"}"
 values: "0+6x5"
 - series: "http_requests_total{service=\"checkout\",instance=\"quiet\"}"
 values: "0+60x5"
 - series: "http_errors_total{service=\"checkout\",instance=\"quiet\"}"
 values: "0+30x5"
 promql_expr_test:
 - expr: "service:http_requests:rate2m"
 eval_time: 5m
 exp_samples:
 - labels: "{__name__=\"service:http_requests:rate2m\",service=\"checkout\"}"
 value: 11
 - expr: "abs(service:http_errors:rate2m - 0.6) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{service=\"checkout\"}"
 value: 1
 - expr: "abs(service:http_errors:ratio_rate2m - 0.6/11) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{service=\"checkout\"}"
 value: 1
 - expr: "abs(avg(rate(http_errors_total[2m]) / rate(http_requests_total[2m])) - 0.255) < bool 1e-12"
 eval_time: 5m
 exp_samples:
 - labels: "{}"
 value: 1
"""
rules = r"""groups:
 - name: service-rates
 interval: 1m
 rules:
 - record: instance:http_requests:rate2m
 expr: sum by (service, instance) (rate(http_requests_total[2m]))
 - record: service:http_requests:rate2m
 expr: sum by (service) (instance:http_requests:rate2m)
 - record: service:http_errors:rate2m
 expr: sum by (service) (rate(http_errors_total[2m]))
 - record: service:http_errors:ratio_rate2m
 expr: service:http_errors:rate2m / service:http_requests:rate2m
"""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
 script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
 print(result.stderr, end="")
 raise SystemExit(result.returncode)

The expected output is SUCCESS. The recorded request rate is eleven, the error rate is 0.6, and the resulting fraction is 0.6/11. A separate assertion demonstrates that the incorrect average of instance fractions is 0.255. Numerical comparisons use a small tolerance where floating-point division is involved, and label assertions retain the checkout service identity.

The disposable container has no external network or host mounts. This is an arithmetic and dependency test, not a performance benchmark or proof of production instrumentation completeness. Before deploying the rules, add empty-traffic, counter-reset, and missing-series cases reflecting the actual service. Measure query cost and rule execution time to establish whether precomputation delivers the intended operational benefit.

Sources