A request counter answers how many events an instrumented process has accumulated. A rate answers how quickly those events are arriving. Prometheus rate estimates average growth per second over a selected range, adjusts observed counter resets, and extrapolates toward the range boundaries. The result is useful precisely because it has a time unit, but it is not an exact reconstruction of every event between scrapes.

Sampling determines what can be inferred

An ordinary counter increases until it resets, commonly after a restart. A decrease between observed samples is treated as a reset by the counter functions. A restart and recovery entirely between two scrapes may be invisible if the new value already exceeds the previous sample. More frequent scraping improves observation resolution; no query can recover an unobserved history with certainty.

Apply rate to individual counter series before aggregating them. Summing first can let one instance’s growth hide another instance’s reset. Labels define series identity, so a replacement instance with different labels starts a different series rather than continuing the old one. A gauge such as queue depth can fall normally and should not be interpreted as a resetting counter.

The selected range is part of the meaning

A range selector is open at its left boundary and closed at its right boundary. At minute five, [5m] excludes a sample exactly at minute zero. The float-counter calculation needs at least two usable samples. A range containing one sample produces no rate value; it does not establish zero traffic. Leave enough room for the configured cadence and realistic missed scrapes.

Longer windows generally smooth short bursts and delay their disappearance from the average. Shorter windows respond sooner but offer fewer observations and can become empty. Choose the window from the event duration and detection objective, then label it explicitly. irate uses the last two samples, whereas rate uses the selected range. Both can be useful on the same metric if panels explain their different questions.

The query evaluation interval controls when a rule asks the question, not which samples were scraped. Refreshing a dashboard every second cannot create new observations from a target scraped every minute. Inspect these two cadences separately when assessing alert responsiveness.

When it breaks

A newly created series may vanish from a rate panel until a second sample arrives. Filling every missing result with zero can conceal collection failure. Check the raw series, timestamps, and scrape status before treating a blank panel as idle traffic. A very smooth long-window line can also hide a brief overload that matters to users.

Reset adjustment is not immunity to gaps. Missing samples can hide bursts or multiple restarts, and boundary extrapolation introduces estimation. increase applies the corresponding range estimate in event-count units; it may return a fractional count even for integer counters. Use application accounting records when exact totals are required, rather than promising that a dashboard reconstructs them.

Check it quickly

Save this program as rate.py and run python3 rate.py. Docker and the Prometheus 3.5.1 image are required; Docker can download the image if needed. The program invokes the real promtool engine against synthetic samples spaced one minute apart. The isolated container has no network access or host mounts and is removed on exit.

import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
tests = r"""evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: "demo_requests_total{instance=\"steady\"}"
        values: "0+60x5"
      - series: "demo_requests_total{instance=\"restart\"}"
        values: "100 160 220 20 80 140"
      - series: "demo_requests_total{instance=\"gaps\"}"
        values: "0 60 _ 180 _ 300"
      - series: "demo_requests_total{instance=\"one\"}"
        values: "_ _ _ _ _ 7"
    promql_expr_test:
      - expr: "rate(demo_requests_total{instance=\"steady\"}[5m])"
        eval_time: 5m
        exp_samples:
          - labels: "{instance=\"steady\"}"
            value: 1
      - expr: "abs(rate(demo_requests_total{instance=\"restart\"}[5m]) - 5/6) < bool 1e-12"
        eval_time: 5m
        exp_samples:
          - labels: "{instance=\"restart\"}"
            value: 1
      - expr: "increase(demo_requests_total{instance=\"restart\"}[5m])"
        eval_time: 5m
        exp_samples:
          - labels: "{instance=\"restart\"}"
            value: 250
      - expr: "irate(demo_requests_total{instance=\"restart\"}[5m])"
        eval_time: 5m
        exp_samples:
          - labels: "{instance=\"restart\"}"
            value: 1
      - expr: "rate(demo_requests_total{instance=\"gaps\"}[5m])"
        eval_time: 5m
        exp_samples:
          - labels: "{instance=\"gaps\"}"
            value: 1
      - expr: "rate(demo_requests_total{instance=\"one\"}[5m])"
        eval_time: 5m
        exp_samples: []
"""
rules = r""""""
script = "set -eu\ncat > /tmp/tests.yml <<'TEST_DATA'\n" + tests + "\nTEST_DATA\n"
if rules:
    script += "cat > /tmp/rules.yml <<'RULE_DATA'\n" + rules + "\nRULE_DATA\n"
script += "exec /bin/promtool test rules /tmp/tests.yml\n"
name = "xbcatm-prom-unit-" + uuid.uuid4().hex[:12]
try:
    result = subprocess.run(
        ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
         "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
        input=script, text=True, capture_output=True, timeout=45,
    )
finally:
    subprocess.run(["docker", "rm", "-f", name],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(result.stdout, end="")
if result.returncode:
    print(result.stderr, end="")
    raise SystemExit(result.returncode)

Successful assertions print SUCCESS. The steady series grows at one request per second. For samples 100 160 220 20 80 140, the five-minute rate is 5/6, increase is 250, and irate is one. These differ because the selected range excludes the first sample, corrects the observed reset, and extrapolates its average across the requested duration.

The deliberately linear series with gaps still yields one per second; that controlled result does not prove arbitrary gaps harmless. The one-sample series yields an empty vector. The test checks values and absence separately, using numerical tolerance for the repeating fraction. Before deploying a rule, add representative restart, sparse-traffic, and missing-data cases from the service whose counter you will monitor.

Sources