Service discovery produces candidate targets and metadata. Target relabeling transforms those candidates before Prometheus scrapes them. It can keep or drop targets, change how they are addressed, and create labels carried into collected series. This is a different stage from rewriting individual scraped samples. Understanding the stage prevents a configuration intended to remove one metric from accidentally removing an entire target.

Inspect discovery input before defining the rule

Discovered targets include internal labels such as __address__ and discovery-specific __meta_ fields. For file-based discovery, __meta_filepath identifies the file that supplied a target. Those metadata labels are available during target relabeling, but internal labels are removed from the final target label set after that stage. Copy the useful value into an ordinary label if downstream queries need it.

The exact metadata names depend on the discovery mechanism. A rule copied from Kubernetes discovery will not automatically work against file discovery. Missing source labels contribute empty values when relabeling combines its inputs, so a keep rule matching an unavailable field can discard every candidate. Inspect the actual discovered labels through the target API or discovery view before diagnosing a network outage.

Order determines the resulting identity

Relabeling steps run in their configured order. A keep action retains a target only when the combined source-label value matches its regular expression; a drop action removes matching targets. The regular expressions are anchored at both ends. A replace action can copy captured text into a destination label, and that new value is available to later rules in the same sequence.

Copying a value does not inherently delete the original label. If removal is needed, it is a separate operation and must preserve the identity required by the scrape and consumers. Changing __address__ changes the endpoint Prometheus will contact, while changing an ordinary label changes context. Neither action should be described merely as cosmetic cleanup of dashboard names.

Target labels can affect every series collected from the target. Choose stable fields such as a service or meaningful deployment scope rather than copying every discovery annotation. A frequently changing build identifier creates a new series identity across many metrics. A small discovery configuration change can therefore affect a much larger stored dataset and the queries that reference it.

When it breaks

A target excluded by relabeling is not a configured target with up=0. It is not scraped, so a failed-scrape query may never report it. When an expected service disappears, compare the discovered candidates, dropped targets and active targets. An incorrect keep rule can look like an empty healthy fleet if the alert assumes all expected members still have up series.

Copying the wrong metadata can also produce convincing but incorrect ownership labels. Query results may look tidy while representing the wrong service scope. Review rule order, capture groups, and final labels together. Broad label removal can merge previously distinct identities; target relabeling does not calculate an aggregate to compensate for the lost distinction.

Check it quickly

Save this as target-relabel.py and run python3 target-relabel.py. Docker and Prometheus 3.5.1 are required; the image may be downloaded if missing. One isolated container hosts a static text metrics endpoint and a file-discovery document containing an API target plus a discard target. The file endpoint uses /metrics.txt so its HTTP response has a text content type accepted by Prometheus.

import http.client
import io
import json
import subprocess
import time
import uuid
from urllib.parse import quote

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
    def __init__(self, data):
        self.data = data

    def makefile(self, mode):
        return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
    body = body.encode()
    headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
               f"Connection: close\r\nContent-Type: {content_type}\r\n"
               f"Content-Length: {len(body)}\r\n\r\n").encode()
    result = subprocess.run(
        ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
        input=headers + body, capture_output=True, timeout=5,
    )
    if result.returncode:
        raise RuntimeError(result.stderr.decode())
    response = http.client.HTTPResponse(ResponseSocket(result.stdout))
    response.begin()
    return response.status, response.read().decode()


def get_json(path):
    status, body = request("GET", path)
    assert status == 200, (status, body)
    return json.loads(body)


def eventually(check):
    for attempt in range(150):
        try:
            value = check()
            if value:
                return value
        except (RuntimeError, AssertionError, http.client.HTTPException):
            pass
        time.sleep(0.1)
    raise AssertionError("Expected local server state did not appear")


def query(expression):
    response = get_json("/api/v1/query?query=" + quote(expression, safe=""))
    assert response["status"] == "success"
    return response["data"]["result"]


def value(expression):
    result = query(expression)
    assert len(result) == 1, (expression, result)
    return float(result[0]["value"][1])

port = 9090
# Requires Docker and the Prometheus 3.5.1 image.
script = r"""set -eu
mkdir -p /tmp/www /tmp/targets
cat > /tmp/prom.yml <<'PROM_CONFIG'
global:
  scrape_interval: 1s
  scrape_timeout: 500ms
scrape_configs:
  - job_name: discovered
    metrics_path: /metrics.txt
    file_sd_configs:
      - files: ['/tmp/targets/*.json']
    relabel_configs:
      - source_labels: [role]
        regex: api
        action: keep
      - source_labels: [__meta_filepath]
        regex: '.*/([^/]+)[.]json'
        target_label: discovery_file
        replacement: '$1'

PROM_CONFIG
cat > /tmp/www/metrics.txt <<'METRIC_DATA'
# TYPE demo_requests_total counter
demo_requests_total{route="/orders",build_sha="release-a"} 42
# TYPE debug_alloc_bytes gauge
debug_alloc_bytes{kind="scratch"} 9

METRIC_DATA
cat > /tmp/targets/lab.json <<'TARGETS'
[{"targets": ["127.0.0.1:8080"], "labels": {"role": "api"}}, {"targets": ["127.0.0.1:1"], "labels": {"role": "discard"}}]
TARGETS
/bin/httpd -f -p 127.0.0.1:8080 -h /tmp/www > /tmp/httpd.log 2>&1 &
exec /bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data --web.listen-address=127.0.0.1:9090
"""
try:
    subprocess.run(
        ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
         "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-c", script],
        check=True, capture_output=True, text=True, timeout=20,
    )
    eventually(lambda: value('up{job="discovered"}') == 1)
    targets = get_json('/api/v1/targets?state=any')["data"]
    assert len(targets["activeTargets"]) == 1 and len(targets["droppedTargets"]) == 1
    active = targets["activeTargets"][0]
    dropped = targets["droppedTargets"][0]
    assert active["discoveredLabels"]["__meta_filepath"] == "/tmp/targets/lab.json"
    assert dropped["discoveredLabels"]["role"] == "discard"
    labels = query('up{job="discovered"}')[0]["metric"]
    assert labels["discovery_file"] == "lab" and labels["role"] == "api"
    assert not any(label.startswith("__meta_") for label in labels)
    print(json.dumps({"active": 1, "dropped": 1, "stored_labels": labels,
                      "source_file": active["discoveredLabels"]["__meta_filepath"]}, indent=2))
finally:
    subprocess.run(["docker", "rm", "-f", name],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The program waits for actual collection, then checks one active target and one dropped target. The active target’s discovered metadata contains /tmp/targets/lab.json. Its stored up series has role="api" and discovery_file="lab", with no surviving __meta_ label. The discard target is visible in the dropped-target evidence and was not treated as a failing scrape.

All requests stay on container loopback, with no external network, host mounts or published ports. Cleanup removes the uniquely named container. These assertions test real file discovery, filtering, metadata copying and final labels. They do not validate another discovery provider’s metadata or prove that a production inventory contains every intended service.

Sources