When several Prometheus servers send data to a common system, downstream readers need to know which collector or environment supplied it. Global external_labels can add that context during communication with external systems. They do not rewrite every locally stored series, and they do not override labels already present on the outgoing series. Understanding both boundaries prevents misleading configuration checks.

Local storage and outgoing identity differ

A local query of up generally shows the labels created by scraping and relabeling. An external label such as prom_region need not appear there. Its absence from that query does not prove the configuration failed. Compare the appropriate outgoing interface with the local result; federation is a direct way to observe this difference without configuring a remote storage backend.

The configuration contract also covers communication such as remote storage and alerting, but each downstream path has its own relabeling and processing. Inspect the actual consumer when validating that path. A locally evaluated recording rule does not acquire every external label merely because it stores a new series. External labels are added at the relevant communication boundary, not as a universal local query transformation.

Existing labels take precedence

If a target series already has cluster="target-cluster", configuring an external cluster="collector-cluster" does not replace that value. External labels supplement missing label names. This is defined behavior, not a promise that newer versions always reject collisions or log an error. A local metric can therefore export a different cluster value from the one an operator expected the global setting to enforce.

Reserve distinct collector label names when that distinction matters, and check exporter and relabeling conventions for collisions. For federation, the destination’s honor_labels setting determines how it handles source labels that conflict with its own scrape labels. That destination behavior is a separate issue from how the source adds external labels to its federation response.

Stable labels do not provide deduplication

Changing a global label value changes the identity of subsequently exported series. Previously stored downstream history remains under its old labels; it is not rewritten. Update selectors and dashboards with that historical boundary in mind. Adding many labels is not automatically useful: a constant label adds context, while different values across collectors create additional distinctions consumers must handle.

Two collectors scraping the same targets can export duplicate observations under different replica labels. A naive sum across replicas can double apparent traffic. External labels make that distinction expressible but do not configure the downstream system’s deduplication. Nor are labels a tenant authorization boundary. Verify the storage system’s actual isolation and query behavior when those requirements apply.

When it breaks

A team may look for an external label in a local query, find none, and repeatedly change a working configuration. The opposite mistake is seeing the desired label in a federation response and assuming remote write or alert routing has also been tested. Check the specific path and final labels, including any downstream relabeling, instead of using one observation as evidence for all integrations.

Check it quickly

Save this as external-labels.py and run python3 external-labels.py. It requires Docker and the Prometheus 3.5.1 image; Docker may download an uncached image. One disposable Prometheus scrapes itself, then the program compares its local query response with its text federation output. All HTTP requests stay on container loopback, with no host ports or external network.

import json
import re
import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
# All HTTP traffic stays on loopback inside one disposable container.
script = r"""set -eu
cat > /tmp/prom.yml <<'CONFIG'
global:
  scrape_interval: 1s
  scrape_timeout: 500ms
  external_labels:
    cluster: collector-cluster
    prom_region: lab-region
    replica: one
scrape_configs:
  - job_name: self
    static_configs:
      - targets: ['127.0.0.1:9090']
        labels:
          cluster: target-cluster
CONFIG
/bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data \
  --web.listen-address=127.0.0.1:9090 > /tmp/prom.log 2>&1 &
for attempt in $(seq 1 30); do
  if wget -qO /tmp/ready.json 'http://127.0.0.1:9090/api/v1/query?query=up%20%3D%3D%201'; then
    if grep -q '"status":"success"' /tmp/ready.json && ! grep -q '"result":\[\]' /tmp/ready.json; then
      wget -qO /tmp/local.json 'http://127.0.0.1:9090/api/v1/query?query=up'
      wget -qO /tmp/federated.txt --header='Accept: text/plain; version=0.0.4' \
        'http://127.0.0.1:9090/federate?match%5B%5D=up'
      cat /tmp/local.json
      printf '\n'
      cat /tmp/federated.txt
      exit 0
    fi
  fi
  sleep 1
done
cat /tmp/prom.log >&2
exit 1
"""
name = "xbcatm-prom-live-" + uuid.uuid4().hex[:12]
try:
    result = subprocess.run(
        ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
         "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
        input=script, text=True, capture_output=True, timeout=45,
    )
finally:
    subprocess.run(["docker", "rm", "-f", name],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if result.returncode:
    raise RuntimeError(result.stderr or result.stdout)
local_json, _, exported = result.stdout.partition("\n")
payload = json.loads(local_json)
assert payload["status"] == "success"
local = next(a["metric"] for a in payload["data"]["result"]
             if a["metric"]["job"] == "self")
line = next(line for line in exported.splitlines() if line.startswith("up{"))
# The fixture's controlled label values contain no quotes or escape sequences.
outbound = dict(re.findall(r'(\w+)="([^"]*)"', line))
assert outbound["prom_region"] == "lab-region" and outbound["replica"] == "one"
assert outbound["cluster"] == local["cluster"] == "target-cluster"
assert "prom_region" not in local and "replica" not in local
print(json.dumps({"federated_labels": outbound, "local_labels": local}, indent=2))

The local labels contain cluster="target-cluster" but no prom_region or replica. Federation adds prom_region="lab-region" and replica="one", while the existing cluster value remains unchanged. Assertions verify all three facts. The parser is deliberately limited to the fixture’s controlled label values, which contain no quotes or escape sequences; it is not a general exposition parser.

The uniquely named container is removed on exit. This establishes source-side federation behavior and the local/outgoing distinction. It does not validate a federation destination, remote write receiver, high-availability deduplication, or notification delivery. Use the corresponding downstream interface and configuration to complete those separate integration checks.

Sources