Prometheus has several relabeling stages, and their similar syntax can hide different effects. Target relabeling transforms discovered targets before collection. Metric relabeling transforms scraped samples just before ingestion. A drop action at the first stage can prevent an endpoint from being scraped; the same action at the second stage discards matching samples from an endpoint that was already contacted.

Decide whether to remove data or remove a label

In metric_relabel_configs, action: drop matches combined source-label values and discards the matching sample. Using __name__ as the source lets a rule reject selected metric families. It does not mean remove the named source label while retaining the sample. A broad expression against a common value can therefore discard much more data than intended.

action: labeldrop instead applies its regular expression to label names and removes matching labels from retained samples. labelkeep retains only names that match. Neither operation sums values or calculates a rollup. The distinction is between removing observations and changing their identity, so choose the operation based on the intended data contract rather than the similar action names.

Relabeling expressions are anchored at both ends. A replacement or drop rule needs the correct source labels and matching pattern at its particular stage. Discovery metadata usually belongs to target relabeling and is no longer available as __meta_ input during sample processing. Copy required context into ordinary target labels earlier instead of expecting an internal discovery label to accompany every exported metric.

Preserve uniqueness after removing dimensions

A time series is identified by its metric name and full label set. Removing a label that distinguishes two observations can collapse their identities, causing duplicate-sample conflicts or lost distinctions. Prometheus does not infer whether their values should be summed, averaged or selected. Prefer bounded instrumentation and explicit query aggregation when the real goal is a service total.

The example removes a build label only where each remaining route identity is already unique. That is a controlled precondition, not permission to remove any expensive label from any exporter. Check representative exposition for collisions before deploying a rule. Also inspect dashboards and alerts that reference the removed label; a successful scrape does not guarantee those consumers still select the intended data.

When it breaks

A filtering rule placed under target relabeling can remove the entire endpoint instead of one costly metric. Conversely, a metric filter will not save the exporter from generating the response or prevent its bytes from crossing the network. It reduces retained samples at ingestion. If collection itself is expensive, address instrumentation or exporter behavior as well.

Automatically generated scrape series such as up are not processed by metric relabeling. A rule attempting to drop up there will not hide the generated metric. A healthy up can coexist with an intentionally discarded business metric, so data absence must be interpreted with the configured filters in mind. Changing filters affects future ingestion and does not rewrite already stored historical labels.

Check it quickly

Save this as metric-relabel.py and run python3 metric-relabel.py. It requires Docker and Prometheus 3.5.1; Docker may fetch the image if needed. The same static text endpoint is scraped under raw and filtered jobs. Its .txt path provides a text content type; startup waits for actual samples rather than assuming an immediately populated target list.

import http.client
import io
import json
import subprocess
import time
import uuid
from urllib.parse import quote

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
    def __init__(self, data):
        self.data = data

    def makefile(self, mode):
        return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
    body = body.encode()
    headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
               f"Connection: close\r\nContent-Type: {content_type}\r\n"
               f"Content-Length: {len(body)}\r\n\r\n").encode()
    result = subprocess.run(
        ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
        input=headers + body, capture_output=True, timeout=5,
    )
    if result.returncode:
        raise RuntimeError(result.stderr.decode())
    response = http.client.HTTPResponse(ResponseSocket(result.stdout))
    response.begin()
    return response.status, response.read().decode()


def get_json(path):
    status, body = request("GET", path)
    assert status == 200, (status, body)
    return json.loads(body)


def eventually(check):
    for attempt in range(150):
        try:
            value = check()
            if value:
                return value
        except (RuntimeError, AssertionError, http.client.HTTPException):
            pass
        time.sleep(0.1)
    raise AssertionError("Expected local server state did not appear")


def query(expression):
    response = get_json("/api/v1/query?query=" + quote(expression, safe=""))
    assert response["status"] == "success"
    return response["data"]["result"]


def value(expression):
    result = query(expression)
    assert len(result) == 1, (expression, result)
    return float(result[0]["value"][1])

port = 9090
# Requires Docker and the Prometheus 3.5.1 image.
script = r"""set -eu
mkdir -p /tmp/www /tmp/targets
cat > /tmp/prom.yml <<'PROM_CONFIG'
global:
  scrape_interval: 1s
  scrape_timeout: 500ms
scrape_configs:
  - job_name: raw
    metrics_path: /metrics.txt
    static_configs:
      - targets: ['127.0.0.1:8080']
  - job_name: filtered
    metrics_path: /metrics.txt
    static_configs:
      - targets: ['127.0.0.1:8080']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'debug_.+|up'
        action: drop
      - regex: build_sha
        action: labeldrop

PROM_CONFIG
cat > /tmp/www/metrics.txt <<'METRIC_DATA'
# TYPE demo_requests_total counter
demo_requests_total{route="/orders",build_sha="release-a"} 42
# TYPE debug_alloc_bytes gauge
debug_alloc_bytes{kind="scratch"} 9

METRIC_DATA
/bin/httpd -f -p 127.0.0.1:8080 -h /tmp/www > /tmp/httpd.log 2>&1 &
exec /bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data --web.listen-address=127.0.0.1:9090
"""
try:
    subprocess.run(
        ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
         "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-c", script],
        check=True, capture_output=True, text=True, timeout=20,
    )
    eventually(lambda: len(query('demo_requests_total')) == 2)
    raw = query('demo_requests_total{job="raw"}')[0]
    filtered = query('demo_requests_total{job="filtered"}')[0]
    assert raw["metric"]["build_sha"] == "release-a"
    assert "build_sha" not in filtered["metric"]
    assert float(raw["value"][1]) == float(filtered["value"][1]) == 42
    assert len(query('debug_alloc_bytes{job="raw"}')) == 1
    assert query('debug_alloc_bytes{job="filtered"}') == []
    assert value('up{job="filtered"}') == 1
    print(json.dumps({"raw_labels": raw["metric"], "filtered_labels": filtered["metric"],
                      "debug_series_dropped": True, "generated_up_preserved": True}, indent=2))
finally:
    subprocess.run(["docker", "rm", "-f", name],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The raw request series retains build_sha="release-a"; the filtered one loses that label while retaining its route and value of 42. The debug metric exists in the raw job and is absent in the filtered job. Although the drop pattern also names up, the filtered job’s generated up remains one. Assertions check all these differences through the actual query API.

The two jobs have distinct job labels, and this fixture has no competing request series after label removal. It verifies stage semantics, not a general collision-free migration. All traffic stays on container loopback without host mounts or published ports, and cleanup removes only the test container. Validate real exporter identities and affected consumers before applying an equivalent production filter.

Sources