Labels · XB Field Notes

Kubernetes labels define selection, not authorization

Thu, 09 Sep 2027 00:00:00 +0000

Labels are key-value metadata used to select Kubernetes objects. A Service selector, an operational query, and a workload controller can all rely on them, but each consumer has its own API and behavior. A label change can therefore alter group membership while leaving the underlying process healthy. Review labels as inputs to actual consumers rather than as purely descriptive stickers.

Know which selector language is in use

Label keys use a qualified-name syntax, and values have format and length constraints. A value may be empty; otherwise its permitted characters and endpoints are restricted, with a maximum of sixty-three characters. Labels are not a place for arbitrary paragraphs or unconstrained JSON. Use annotations for descriptive metadata, while still respecting annotation validation and consumer contracts.

Equality requirements can test that a key has a value, while set-based requirements support membership and existence tests. Multiple requirements in a selector are combined as conditions that must all hold. An important detail is inequality: track!=canary also matches an object with no track label. If absence is unacceptable, require the key to exist as well instead of assuming inequality implies its presence.

Selector forms differ between resources. A normal selector-based Service uses a map of equality requirements; it does not accept the complete set-expression language as arbitrary map values. Other APIs expose LabelSelector structures, and kubectl list queries accept selector expressions. Check the exact field’s schema before translating a convenient command-line query into a manifest.

When it breaks

A Service’s selector identifies matching Pods in its namespace, and the endpoint machinery tracks addresses and readiness. Matching labels alone do not prove that a server is ready or that its port is correct. With normal readiness behavior, available routing targets depend on endpoint conditions too. Inspect the relevant EndpointSlices and application checks rather than stopping at a successful label query.

The publishNotReadyAddresses setting changes how readiness is treated for Service discovery. Do not assume an endpoint’s ready flag always proves the Pod passed its application probe under that setting. The fixture below uses the ordinary default and does not exercise this override.

Relabelling a healthy Pod can remove it from a Service without terminating it. Conversely, an overly broad selector can include a workload that was never intended as a backend. Changing labels on controller-managed Pods can also interact with ownership and reconciliation. Update the intended desired state through the responsible controller, and check all important consumers of a shared label before changing its meaning.

Selectors are not network authorization. A Pod excluded from one Service can remain reachable through its own address or another Service unless other controls prevent that access. NetworkPolicy, application authorization, and routing each solve different parts of the problem. Do not describe a selector as a guest list that denies all direct traffic to unselected Pods.

Check it quickly

Use a disposable Kubernetes cluster with kubectl and python:3.11-slim-bookworm available on the nodes. Save this as label-selectors.py and run python3 label-selectors.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The script creates a unique namespace containing three standalone HTTP Pods and a Service, then removes it. The recorded server version was Kubernetes 1.34.11.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-labels-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 for name, role, track in [('a', 'reader', 'stable'), ('b', 'reader', 'canary'), ('c', 'writer', None)]:
 script = '''from http.server import BaseHTTPRequestHandler,HTTPServer
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 data=LABEL.encode();self.send_response(200);self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
 def log_message(self,*args): pass
HTTPServer(('0.0.0.0',8080),Handler).serve_forever()
'''.replace('LABEL', repr(name))
 podspec = spec(script)
 podspec['containers'][0]['readinessProbe'] = {'httpGet': {'path': '/', 'port': 8080}, 'periodSeconds': 1}
 labels = {'app': 'catalog', 'role': role}
 if track:
 labels['track'] = track
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': name, 'labels': labels}, 'spec': podspec})
 for name in ('a', 'b', 'c'):
 wait_for(lambda name=name: obj(namespace, 'pod', name), pod_ready)
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'readers'},
 'spec': {'selector': {'app': 'catalog', 'role': 'reader'}, 'ports': [{'port': 80, 'targetPort': 8080}]}})
 def selected():
 slices = obj(namespace, 'endpointslices', selector='kubernetes.io/service-name=readers')['items']
 return {e['targetRef']['name'] for item in slices for e in item.get('endpoints', [])
 if e.get('conditions', {}).get('ready') and e.get('targetRef')}
 before = wait_for(selected, lambda names: names == {'a', 'b'})
 inequality = {p['metadata']['name'] for p in obj(namespace, 'pods', selector='app=catalog,track!=canary')['items']}
 assert inequality == {'a', 'c'}
 expression = {p['metadata']['name'] for p in obj(namespace, 'pods', selector='app in (catalog),role=reader')['items']}
 assert expression == {'a', 'b'}
 writer_ip = obj(namespace, 'pod', 'c')['status']['podIP']
 probe = 'from urllib.request import urlopen; print(urlopen(' + repr('http://'+writer_ip+':8080/') + ',timeout=3).read().decode())'
 assert k('exec', '-n', namespace, 'a', '--', 'python3', '-c', probe).stdout.strip() == 'c'
 k('label', 'pod', 'b', '-n', namespace, 'role=writer', '--overwrite')
 after = wait_for(selected, lambda names: names == {'a'})
 assert pod_ready(obj(namespace, 'pod', 'b'))
 invalid = k('patch', 'pod', 'a', '-n', namespace, '--type=merge',
 '-p', json.dumps({'metadata': {'labels': {'notes.example/value': 'has spaces'}}}), expected=None)
 assert invalid.returncode != 0 and 'Invalid' in invalid.stderr
 print(json.dumps({'service_targets_before': sorted(before), 'service_targets_after': sorted(after),
 'inequality_also_matches_missing_key': sorted(inequality),
 'set_selector_matches': sorted(expression), 'excluded_pod_directly_reachable': True,
 'invalid_label_value_rejected': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

Pods a and b have role=reader; c has role=writer. All share app=catalog and become ready through HTTP probes. Pod a has a stable track, b has canary, and c lacks track. The reader Service initially has ready targets a and b. A set-based query finds those readers, while track!=canary finds a and c, confirming the missing-key behavior.

An actual request from a to c’s Pod address returns c, even though c is excluded from the reader Service. Changing b’s role to writer then leaves only a in the Service’s ready endpoint set. Pod b remains ready. A separate patch containing spaces in a label value is rejected by the API. All assertions passed.

This verifies API selection, EndpointSlice changes, and direct reachability in the isolated test network. It does not test every controller’s adoption rules, enforce a NetworkPolicy, or benchmark Service traffic. Keep selection intent, endpoint readiness, and access policy independently observable when reviewing routing changes.

Sources

Kubernetes: Labels and Selectors, valid values, equality and set-based requirements, checked 2026-09-13.
Kubernetes: Services, selector-based backend discovery, checked 2026-09-13.
Kubernetes: EndpointSlices, endpoint identity and readiness conditions, checked 2026-09-13.

External labels identify outgoing Prometheus data

Wed, 09 Jun 2027 00:00:00 +0000

When several Prometheus servers send data to a common system, downstream readers need to know which collector or environment supplied it. Global external_labels can add that context during communication with external systems. They do not rewrite every locally stored series, and they do not override labels already present on the outgoing series. Understanding both boundaries prevents misleading configuration checks.

Local storage and outgoing identity differ

A local query of up generally shows the labels created by scraping and relabeling. An external label such as prom_region need not appear there. Its absence from that query does not prove the configuration failed. Compare the appropriate outgoing interface with the local result; federation is a direct way to observe this difference without configuring a remote storage backend.

The configuration contract also covers communication such as remote storage and alerting, but each downstream path has its own relabeling and processing. Inspect the actual consumer when validating that path. A locally evaluated recording rule does not acquire every external label merely because it stores a new series. External labels are added at the relevant communication boundary, not as a universal local query transformation.

Existing labels take precedence

If a target series already has cluster="target-cluster", configuring an external cluster="collector-cluster" does not replace that value. External labels supplement missing label names. This is defined behavior, not a promise that newer versions always reject collisions or log an error. A local metric can therefore export a different cluster value from the one an operator expected the global setting to enforce.

Reserve distinct collector label names when that distinction matters, and check exporter and relabeling conventions for collisions. For federation, the destination’s honor_labels setting determines how it handles source labels that conflict with its own scrape labels. That destination behavior is a separate issue from how the source adds external labels to its federation response.

Stable labels do not provide deduplication

Changing a global label value changes the identity of subsequently exported series. Previously stored downstream history remains under its old labels; it is not rewritten. Update selectors and dashboards with that historical boundary in mind. Adding many labels is not automatically useful: a constant label adds context, while different values across collectors create additional distinctions consumers must handle.

Two collectors scraping the same targets can export duplicate observations under different replica labels. A naive sum across replicas can double apparent traffic. External labels make that distinction expressible but do not configure the downstream system’s deduplication. Nor are labels a tenant authorization boundary. Verify the storage system’s actual isolation and query behavior when those requirements apply.

When it breaks

A team may look for an external label in a local query, find none, and repeatedly change a working configuration. The opposite mistake is seeing the desired label in a federation response and assuming remote write or alert routing has also been tested. Check the specific path and final labels, including any downstream relabeling, instead of using one observation as evidence for all integrations.

Check it quickly

Save this as external-labels.py and run python3 external-labels.py. It requires Docker and the Prometheus 3.5.1 image; Docker may download an uncached image. One disposable Prometheus scrapes itself, then the program compares its local query response with its text federation output. All HTTP requests stay on container loopback, with no host ports or external network.

import json
import re
import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
# All HTTP traffic stays on loopback inside one disposable container.
script = r"""set -eu
cat > /tmp/prom.yml <<'CONFIG'
global:
 scrape_interval: 1s
 scrape_timeout: 500ms
 external_labels:
 cluster: collector-cluster
 prom_region: lab-region
 replica: one
scrape_configs:
 - job_name: self
 static_configs:
 - targets: ['127.0.0.1:9090']
 labels:
 cluster: target-cluster
CONFIG
/bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data \
 --web.listen-address=127.0.0.1:9090 > /tmp/prom.log 2>&1 &
for attempt in $(seq 1 30); do
 if wget -qO /tmp/ready.json 'http://127.0.0.1:9090/api/v1/query?query=up%20%3D%3D%201'; then
 if grep -q '"status":"success"' /tmp/ready.json && ! grep -q '"result":\[\]' /tmp/ready.json; then
 wget -qO /tmp/local.json 'http://127.0.0.1:9090/api/v1/query?query=up'
 wget -qO /tmp/federated.txt --header='Accept: text/plain; version=0.0.4' \
 'http://127.0.0.1:9090/federate?match%5B%5D=up'
 cat /tmp/local.json
 printf '\n'
 cat /tmp/federated.txt
 exit 0
 fi
 fi
 sleep 1
done
cat /tmp/prom.log >&2
exit 1
"""
name = "xbcatm-prom-live-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if result.returncode:
 raise RuntimeError(result.stderr or result.stdout)
local_json, _, exported = result.stdout.partition("\n")
payload = json.loads(local_json)
assert payload["status"] == "success"
local = next(a["metric"] for a in payload["data"]["result"]
 if a["metric"]["job"] == "self")
line = next(line for line in exported.splitlines() if line.startswith("up{"))
# The fixture's controlled label values contain no quotes or escape sequences.
outbound = dict(re.findall(r'(\w+)="([^"]*)"', line))
assert outbound["prom_region"] == "lab-region" and outbound["replica"] == "one"
assert outbound["cluster"] == local["cluster"] == "target-cluster"
assert "prom_region" not in local and "replica" not in local
print(json.dumps({"federated_labels": outbound, "local_labels": local}, indent=2))

The local labels contain cluster="target-cluster" but no prom_region or replica. Federation adds prom_region="lab-region" and replica="one", while the existing cluster value remains unchanged. Assertions verify all three facts. The parser is deliberately limited to the fixture’s controlled label values, which contain no quotes or escape sequences; it is not a general exposition parser.

The uniquely named container is removed on exit. This establishes source-side federation behavior and the local/outgoing distinction. It does not validate a federation destination, remote write receiver, high-availability deduplication, or notification delivery. Use the corresponding downstream interface and configuration to complete those separate integration checks.

Sources

Prometheus: configuration, external labels supplement labels during outbound communication, checked 2026-09-13.
Prometheus: federation, federation selectors and preservation of upstream labels, checked 2026-09-13.
Prometheus: HTTP API, local query results used for comparison, checked 2026-09-13.

Target relabeling selects and labels targets before scraping

Tue, 06 Apr 2027 00:00:00 +0000

Service discovery produces candidate targets and metadata. Target relabeling transforms those candidates before Prometheus scrapes them. It can keep or drop targets, change how they are addressed, and create labels carried into collected series. This is a different stage from rewriting individual scraped samples. Understanding the stage prevents a configuration intended to remove one metric from accidentally removing an entire target.

Inspect discovery input before defining the rule

Discovered targets include internal labels such as __address__ and discovery-specific __meta_ fields. For file-based discovery, __meta_filepath identifies the file that supplied a target. Those metadata labels are available during target relabeling, but internal labels are removed from the final target label set after that stage. Copy the useful value into an ordinary label if downstream queries need it.

The exact metadata names depend on the discovery mechanism. A rule copied from Kubernetes discovery will not automatically work against file discovery. Missing source labels contribute empty values when relabeling combines its inputs, so a keep rule matching an unavailable field can discard every candidate. Inspect the actual discovered labels through the target API or discovery view before diagnosing a network outage.

Order determines the resulting identity

Relabeling steps run in their configured order. A keep action retains a target only when the combined source-label value matches its regular expression; a drop action removes matching targets. The regular expressions are anchored at both ends. A replace action can copy captured text into a destination label, and that new value is available to later rules in the same sequence.

Copying a value does not inherently delete the original label. If removal is needed, it is a separate operation and must preserve the identity required by the scrape and consumers. Changing __address__ changes the endpoint Prometheus will contact, while changing an ordinary label changes context. Neither action should be described merely as cosmetic cleanup of dashboard names.

Target labels can affect every series collected from the target. Choose stable fields such as a service or meaningful deployment scope rather than copying every discovery annotation. A frequently changing build identifier creates a new series identity across many metrics. A small discovery configuration change can therefore affect a much larger stored dataset and the queries that reference it.

When it breaks

A target excluded by relabeling is not a configured target with up=0. It is not scraped, so a failed-scrape query may never report it. When an expected service disappears, compare the discovered candidates, dropped targets and active targets. An incorrect keep rule can look like an empty healthy fleet if the alert assumes all expected members still have up series.

Copying the wrong metadata can also produce convincing but incorrect ownership labels. Query results may look tidy while representing the wrong service scope. Review rule order, capture groups, and final labels together. Broad label removal can merge previously distinct identities; target relabeling does not calculate an aggregate to compensate for the lost distinction.

Check it quickly

Save this as target-relabel.py and run python3 target-relabel.py. Docker and Prometheus 3.5.1 are required; the image may be downloaded if missing. One isolated container hosts a static text metrics endpoint and a file-discovery document containing an API target plus a discard target. The file endpoint uses /metrics.txt so its HTTP response has a text content type accepted by Prometheus.

import http.client
import io
import json
import subprocess
import time
import uuid
from urllib.parse import quote

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


def query(expression):
 response = get_json("/api/v1/query?query=" + quote(expression, safe=""))
 assert response["status"] == "success"
 return response["data"]["result"]


def value(expression):
 result = query(expression)
 assert len(result) == 1, (expression, result)
 return float(result[0]["value"][1])

port = 9090
# Requires Docker and the Prometheus 3.5.1 image.
script = r"""set -eu
mkdir -p /tmp/www /tmp/targets
cat > /tmp/prom.yml <<'PROM_CONFIG'
global:
 scrape_interval: 1s
 scrape_timeout: 500ms
scrape_configs:
 - job_name: discovered
 metrics_path: /metrics.txt
 file_sd_configs:
 - files: ['/tmp/targets/*.json']
 relabel_configs:
 - source_labels: [role]
 regex: api
 action: keep
 - source_labels: [__meta_filepath]
 regex: '.*/([^/]+)[.]json'
 target_label: discovery_file
 replacement: '$1'

PROM_CONFIG
cat > /tmp/www/metrics.txt <<'METRIC_DATA'
# TYPE demo_requests_total counter
demo_requests_total{route="/orders",build_sha="release-a"} 42
# TYPE debug_alloc_bytes gauge
debug_alloc_bytes{kind="scratch"} 9

METRIC_DATA
cat > /tmp/targets/lab.json <<'TARGETS'
[{"targets": ["127.0.0.1:8080"], "labels": {"role": "api"}}, {"targets": ["127.0.0.1:1"], "labels": {"role": "discard"}}]
TARGETS
/bin/httpd -f -p 127.0.0.1:8080 -h /tmp/www > /tmp/httpd.log 2>&1 &
exec /bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data --web.listen-address=127.0.0.1:9090
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: value('up{job="discovered"}') == 1)
 targets = get_json('/api/v1/targets?state=any')["data"]
 assert len(targets["activeTargets"]) == 1 and len(targets["droppedTargets"]) == 1
 active = targets["activeTargets"][0]
 dropped = targets["droppedTargets"][0]
 assert active["discoveredLabels"]["__meta_filepath"] == "/tmp/targets/lab.json"
 assert dropped["discoveredLabels"]["role"] == "discard"
 labels = query('up{job="discovered"}')[0]["metric"]
 assert labels["discovery_file"] == "lab" and labels["role"] == "api"
 assert not any(label.startswith("__meta_") for label in labels)
 print(json.dumps({"active": 1, "dropped": 1, "stored_labels": labels,
 "source_file": active["discoveredLabels"]["__meta_filepath"]}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The program waits for actual collection, then checks one active target and one dropped target. The active target’s discovered metadata contains /tmp/targets/lab.json. Its stored up series has role="api" and discovery_file="lab", with no surviving __meta_ label. The discard target is visible in the dropped-target evidence and was not treated as a failing scrape.

All requests stay on container loopback, with no external network, host mounts or published ports. Cleanup removes the uniquely named container. These assertions test real file discovery, filtering, metadata copying and final labels. They do not validate another discovery provider’s metadata or prove that a production inventory contains every intended service.

Sources

Prometheus: configuration, ordered target relabeling and file-discovery metadata, checked 2026-09-13.
Prometheus: jobs and instances, target identity and generated scrape metrics, checked 2026-09-13.
Prometheus: HTTP API, active and dropped target inspection, checked 2026-09-13.