High-cardinality labels increase stored series before anyone queries them

Mon, 10 Jan 2028 00:00:00 +0000

A Prometheus series is identified by a metric name and its complete label set. Adding customer or request identity to a counter can create many series even when every dashboard immediately sums those identities away. Query aggregation changes the returned result; it does not remove the raw series that were ingested. Storage and ingestion costs arise before a user opens the dashboard.

Count observed combinations, not isolated values

One new label value is not automatically one new series across the entire database. It creates a new identity wherever the metric name and other labels form a distinct combination. Multiplying possible value counts gives a Cartesian upper bound for a given metric and target scope; it equals the observed count only when every combination is present. Dependencies between dimensions can make the bound much larger than reality. Actual series count also depends on target labels and metric families.

A route label based on a small set of route templates can be bounded. A full URL containing identifiers and query strings often is not. Customer identifiers, request IDs, arbitrary error messages and timestamps also need deliberate scrutiny. A histogram with multiple bucket series amplifies the impact of each label combination further. Budget from the actual exposition and expected workload rather than one dimension in isolation.

Querying fewer rows does not store fewer rows

An expression summing requests by route can return three results after reading twelve customer-specific series. Those twelve input series still exist. If route totals are the required metric contract, instrumentation can accumulate a route-level counter directly and export three series. That reduces the identities emitted at the source instead of relying on every consumer to hide the extra dimension.

Removing a label during ingestion is not equivalent to this accumulation. Distinct samples can acquire the same identity, and relabeling will not decide how to sum them. Prefer a deliberate instrumentation change or an explicitly defined recording query, understanding that recording adds derived series while retaining raw ones under the storage policy. Preserve detailed correlation in logs or traces when needed, with their own access, retention and cost limits.

Diagnose growth with the right inventory

The TSDB status API exposes head statistics, including series counts by metric name. Those values help identify large families but are not a complete cost model. Sample interval, label length, churn, chunk handling, retention and storage behavior also matter. Do not translate a small fixture’s series reduction into an identical percentage reduction in production memory or disk.

Instant query counts and head-series statistics can differ when some head series no longer have recent selectable samples. Historical blocks introduce another scope again. Record the measurement time and population when comparing reports. A release that changes label values may create churn while older identities remain in storage, so a graph returning fewer current results does not prove that all earlier cost has vanished.

When it breaks

A newly added unbounded label can increase ingestion and memory pressure until scrapes or queries suffer. Start by identifying the metric family and instrumentation change, then stop unnecessary new identities at their source or through a carefully reviewed sample filter. Immediately deleting historical data is not a universal recovery requirement, and changing retention does not remove every resource cost instantly.

A subtler failure is a successful aggregation that makes a dashboard look small while the raw metric continues growing. Review both exported identities and query output. Give new dimensions an expected value range, useful consumer and owner. For counters, preserve reset handling when creating production rate queries; a static sum used to compare fixture totals is not a replacement for rate-before-aggregation semantics.

Check it quickly

Save this as cardinality.py and run python3 cardinality.py. It requires Docker and Prometheus 3.5.1, downloading the image if needed. A local text endpoint exports three routes with four synthetic customer identities each, plus independently accumulated route totals. The real server scrapes those samples and exposes both query and TSDB statistics.

import http.client
import io
import json
import subprocess
import time
import uuid
from urllib.parse import quote

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


def query(expression):
 response = get_json("/api/v1/query?query=" + quote(expression, safe=""))
 assert response["status"] == "success"
 return response["data"]["result"]


def value(expression):
 result = query(expression)
 assert len(result) == 1, (expression, result)
 return float(result[0]["value"][1])

port = 9090
# Requires Docker and the Prometheus 3.5.1 image.
script = r"""set -eu
mkdir -p /tmp/www /tmp/targets
cat > /tmp/prom.yml <<'PROM_CONFIG'
global:
 scrape_interval: 1s
 scrape_timeout: 500ms
scrape_configs:
 - job_name: cardinality
 metrics_path: /metrics.txt
 static_configs:
 - targets: ['127.0.0.1:8080']

PROM_CONFIG
cat > /tmp/www/metrics.txt <<'METRIC_DATA'
# TYPE raw_requests_total counter
# TYPE bounded_requests_total counter
raw_requests_total{route="/orders",customer_id="synthetic-1"} 1
raw_requests_total{route="/orders",customer_id="synthetic-2"} 2
raw_requests_total{route="/orders",customer_id="synthetic-3"} 3
raw_requests_total{route="/orders",customer_id="synthetic-4"} 4
bounded_requests_total{route="/orders"} 10
raw_requests_total{route="/cart",customer_id="synthetic-1"} 1
raw_requests_total{route="/cart",customer_id="synthetic-2"} 2
raw_requests_total{route="/cart",customer_id="synthetic-3"} 3
raw_requests_total{route="/cart",customer_id="synthetic-4"} 4
bounded_requests_total{route="/cart"} 10
raw_requests_total{route="/search",customer_id="synthetic-1"} 1
raw_requests_total{route="/search",customer_id="synthetic-2"} 2
raw_requests_total{route="/search",customer_id="synthetic-3"} 3
raw_requests_total{route="/search",customer_id="synthetic-4"} 4
bounded_requests_total{route="/search"} 10

METRIC_DATA
/bin/httpd -f -p 127.0.0.1:8080 -h /tmp/www > /tmp/httpd.log 2>&1 &
exec /bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data --web.listen-address=127.0.0.1:9090
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: len(query('raw_requests_total')) == 12)
 assert value('count(raw_requests_total)') == 12
 assert value('count(bounded_requests_total)') == 3
 assert value('sum(raw_requests_total)') == value('sum(bounded_requests_total)') == 30
 raw_by_route = {x["metric"]["route"]: float(x["value"][1])
 for x in query('sum by (route) (raw_requests_total)')}
 bounded_by_route = {x["metric"]["route"]: float(x["value"][1])
 for x in query('bounded_requests_total')}
 assert raw_by_route == bounded_by_route
 statistics = get_json('/api/v1/status/tsdb')["data"]
 counts = {x["name"]: x["value"] for x in statistics["seriesCountByMetricName"]}
 assert counts["raw_requests_total"] == 12 and counts["bounded_requests_total"] == 3
 print(json.dumps({"raw_series": 12, "bounded_series": 3, "snapshot_total": 30,
 "per_route": raw_by_route, "tsdb_counts_confirmed": True}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The assertions confirm twelve raw series, three bounded series, and a total of thirty in both representations. Each route totals ten. After query aggregation, the TSDB still reports twelve raw series, demonstrating that a smaller result has not erased its inputs. The isolated container uses no external network, host mounts or published ports and is removed on exit. This validates identities and arithmetic, not a production capacity benchmark.

Sources

Prometheus: data model, series identity is the metric name plus its label set, checked 2026-09-13.
Prometheus: metric and label naming, bounded dimensions and high-cardinality label guidance, checked 2026-09-13.
Prometheus: storage, head data, blocks and retention costs, checked 2026-09-13.
Prometheus: HTTP API, head-series statistics by metric name, checked 2026-09-13.