Secrets · XB Field Notes

Kubernetes Secrets require explicit access boundaries

Sat, 21 Aug 2027 00:00:00 +0000

A Kubernetes Secret is an API object for confidential values that workloads need. It gives operators a dedicated object type, consumption paths, and access-control targets. It does not make data unreadable to every other actor. Design the boundary around who can retrieve the object, create consumers, inspect their output, and administer the nodes that run them.

Encoding and protection are different layers

In the JSON API representation, Secret data values use base64; stringData is a write convenience for supplying strings. Base64 can be reversed without a key and is not encryption. The representation does not prove how a particular cluster protects its backing storage. Kubernetes documentation describes Secrets as unencrypted at rest by default unless encryption is configured. Verify the actual API-server and storage setup rather than inferring it from encoded output.

Storage encryption also does not prevent an authorized API reader from obtaining the data. RBAC rules for get, list, and watch can expose Secret contents, so a broad listing permission is not merely access to object names. Protect backups and diagnostic output too. A working application eventually receives usable values, which means a compromised authorized consumer remains inside the data’s trust boundary.

Secrets are intended for small values, with a one-MiB object limit. Mount only required keys and restrict access to the relevant containers. Unix file modes can narrow access within a workload, but do not replace API authorization or control of workload creation. Environment injection also has an exposure and lifecycle tradeoff; choose it from the application’s needs rather than assuming it is automatically safer than a file.

When it breaks

Denying get on Secrets does not necessarily deny indirect access. A subject allowed to create Pods in a namespace can normally name a Secret for a volume in its Pod specification. Kubelet then supplies the data to that workload. Permissions to create suitable controllers can have similar consequences. Review workload-creation rights and applicable admission controls alongside direct data-reading rules.

Broadly mounting every key expands the set of values a process can read. Selecting one key through volume items limits that particular projection, but a privileged creator may be able to request a different projection. Do not mistake a narrow manifest for proof that the actor writing it has equally narrow authority. The intended consumer and the authority to define consumers are separate parts of the boundary.

Rotation requires more than editing an API object. Delivery, application reload, credential issuance, and revocation have their own timing. Environment snapshots and subPath mounts do not automatically refresh like a normal Secret projection. An updated file also does not force an application to discard a value it already cached. Test the issuing system and consumers together before claiming that old credentials are no longer accepted.

Check it quickly

Use a disposable Kubernetes cluster with administrator access for test impersonation and python:3.11-slim-bookworm available on its nodes. Save this as secret-access.py and run python3 secret-access.py KUBECONFIG_PATH CONTEXT against its dedicated configuration. Kubernetes 1.34.11 was used for the recorded run. Every value in this fixture is deliberately synthetic, and output reports checks without printing the stored value.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

import base64
namespace = 'notes-secrets-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 demo = 'DEMO_ONLY_NOT_A_CREDENTIAL'
 k('create', '-n', namespace, '-f', '-', payload={'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'example'},
 'type': 'Opaque', 'stringData': {'needed': demo, 'unused': 'UNUSED_DEMO'}})
 stored = obj(namespace, 'secret', 'example')
 assert base64.b64decode(stored['data']['needed']).decode() == demo
 apply(namespace, {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'creator'},
 'automountServiceAccountToken': False})
 apply(namespace, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'Role',
 'metadata': {'name': 'pod-creator'},
 'rules': [{'apiGroups': [''], 'resources': ['pods'], 'verbs': ['create', 'get']},
 {'apiGroups': [''], 'resources': ['pods/log'], 'verbs': ['get']}]})
 apply(namespace, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'RoleBinding',
 'metadata': {'name': 'pod-creator'},
 'subjects': [{'kind': 'ServiceAccount', 'name': 'creator', 'namespace': namespace}],
 'roleRef': {'apiGroup': 'rbac.authorization.k8s.io', 'kind': 'Role', 'name': 'pod-creator'}})
 identity = 'system:serviceaccount:' + namespace + ':creator'
 denied = k('--as', identity, 'get', 'secret', 'example', '-n', namespace, expected=None)
 assert denied.returncode != 0 and 'Forbidden' in denied.stderr
 probe = "from pathlib import Path\nimport json,time\nroot=Path('/example')\nprint(json.dumps({'synthetic_value_matches':(root/'one').read_text()=='DEMO_ONLY_NOT_A_CREDENTIAL',\n 'unused_key_absent':not (root/'unused').exists(),\n 'service_account_token_absent':not Path('/var/run/secrets/kubernetes.io/serviceaccount/token').exists()}),flush=True)\ntime.sleep(600)"
 podspec = spec(probe)
 podspec['serviceAccountName'] = 'creator'
 podspec['volumes'] = [{'name': 'example', 'secret': {'secretName': 'example',
 'items': [{'key': 'needed', 'path': 'one'}], 'defaultMode': 0o400}}]
 podspec['containers'][0]['volumeMounts'] = [{'name': 'example', 'mountPath': '/example', 'readOnly': True}]
 definition = {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'reader', 'namespace': namespace},
 'spec': podspec}
 k('--as', identity, 'create', '-f', '-', payload=definition)
 wait_for(lambda: obj(namespace, 'pod', 'reader'), pod_ready)
 output = wait_for(lambda: k('--as', identity, 'logs', '-n', namespace, 'reader').stdout.strip(), bool)
 observed = json.loads(output)
 assert all(observed.values())
 print(json.dumps({'api_base64_decodes_to_synthetic_value': True,
 'direct_secret_get_forbidden': True, 'same_identity_can_create_consuming_pod': True,
 'mount_observation': observed, 'etcd_encryption_tested': False}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The script creates a Secret with two dummy keys and confirms that its API base64 representation decodes to the supplied synthetic text. It binds a service account to a Role permitting Pod creation and inspection of Pod logs, without permitting Secret reads. A request impersonating that account to get the Secret returns Forbidden.

The same impersonated subject then creates a Pod that mounts only the needed key. Its own program verifies the expected dummy value, absence of the unused key, and absence of an automatically mounted service-account token. The subject can read those boolean results through its permitted log access. All checks pass, demonstrating the indirect workload path despite the direct denial.

The namespace is removed afterward. No real credential, issued service-account token, backing-store encryption, or rotation process was tested. The experiment establishes authorization and projection behavior, not a complete secret-management system. Assess both direct readers and actors who can create or alter readers before describing a Secret as narrowly scoped.

Sources

Kubernetes: Secrets, API representation, consumption and access implications, checked 2026-09-13.
Kubernetes: Good practices for Secrets, encryption, least privilege and workload access, checked 2026-09-13.
Kubernetes: RBAC authorization, resource verbs, role bindings and namespace scope, checked 2026-09-13.

Credential expiry needs an enforcing verifier

Sun, 27 Sep 2026 00:00:00 +0000

A credential expires only when the system deciding access checks an applicable deadline. Writing an expiry date in an inventory does not change what a database or API accepts. A short lifetime can reduce the period during which a stolen credential starts new access, but its effect depends on the authentication mechanism and on what happens to sessions or other credentials already created with it.

When it breaks

Expiry does not undo earlier actions. Data already copied remains copied, and a session established before the deadline may continue working. An attacker may also have created another access path while authorized. Treat lifetime as one limit on future use, alongside scope, revocation, session management, and investigation of actual effects. It does not put a universal ceiling on the damage from exposure.

PostgreSQL’s VALID UNTIL applies to a role’s password, not to the role’s existence or all authentication methods. A connection using a non-password method does not gain that deadline check. Testing expiry through a local trust rule can therefore misleadingly suggest that the deadline is broken. Verify the authentication path actually used by the client.

Rotation and renewal are separate changes. Replacing a password does not necessarily move its expiry date, and extending a deadline does not replace the value. A refresh design that updates only one can leave applications unable to reconnect or leave the old value usable. Successful access through an existing pool connection does not prove that replacement credentials can establish fresh connections.

Check it quickly

Use an isolated Linux environment with PostgreSQL 18 programs on PATH, running as a non-root user, and an unused loopback port 55477. Save this as a shell script and run it with sh. It initializes a disposable cluster, uses explicitly invented passwords, binds TCP only to loopback, and removes its temporary directory on exit. Local trust is included solely to demonstrate a different authentication path:

#!/bin/sh
set -eu
lab_root=$(mktemp -d /tmp/credential-expiry.XXXXXX)
cleanup() {
 if test -f "$lab_root/data/postmaster.pid"; then
 pg_ctl -D "$lab_root/data" -m immediate -w stop >/dev/null
 fi
 rm -rf "$lab_root"
}
trap cleanup EXIT
initdb -D "$lab_root/data" --locale=C --auth-local=trust --auth-host=scram-sha-256 >/dev/null
pg_ctl -D "$lab_root/data" -l "$lab_root/server.log" \
 -o "-k $lab_root -p 55477 -c listen_addresses=127.0.0.1" -w start >/dev/null
admin() {
 psql -X -qAt -v ON_ERROR_STOP=1 -h "$lab_root" -p 55477 -d postgres "$@"
}
# Both passwords below are invented, disposable lab fixtures.
admin <<'SQL'
SET password_encryption='scram-sha-256';
CREATE ROLE article_expiry LOGIN PASSWORD 'LAB-ONLY-OLD-PASSWORD';
SQL
mkfifo "$lab_root/input"
PGPASSWORD=LAB-ONLY-OLD-PASSWORD psql -X -qAt -v ON_ERROR_STOP=1 \
 -h 127.0.0.1 -p 55477 -U article_expiry -d postgres \
 <"$lab_root/input" >"$lab_root/held.out" 2>"$lab_root/held.err" &
held_pid=$!
exec 3>"$lab_root/input"
printf "SELECT 'ready';\n" >&3
attempt=0
until grep -qx ready "$lab_root/held.out"; do
 attempt=$((attempt + 1))
 test "$attempt" -lt 100
 sleep 0.05
done
admin <<'SQL'
SELECT format('ALTER ROLE article_expiry VALID UNTIL %L',
 clock_timestamp()+interval '1 second') \gexec
SELECT pg_sleep(1.2);
SELECT 'deadline passed='||(rolvaliduntil<clock_timestamp())
FROM pg_authid WHERE rolname='article_expiry';
SQL
if PGPASSWORD=LAB-ONLY-OLD-PASSWORD psql -X -qAt -h 127.0.0.1 -p 55477 \
 -U article_expiry -d postgres -c 'SELECT 1' >"$lab_root/new.out" 2>"$lab_root/new.err"; then
 exit 1
fi
grep -q 'password authentication failed' "$lab_root/new.err"
printf '%s\n' 'fresh SCRAM login rejected after expiry'
printf "SELECT 'held after expiry='||current_user;\n" >&3
psql -X -qAt -v ON_ERROR_STOP=1 -h "$lab_root" -p 55477 \
 -U article_expiry -d postgres -c "SELECT 'local trust login='||current_user"
admin <<'SQL'
SET password_encryption='scram-sha-256';
ALTER ROLE article_expiry PASSWORD 'LAB-ONLY-NEW-PASSWORD';
SQL
if PGPASSWORD=LAB-ONLY-NEW-PASSWORD psql -X -qAt -h 127.0.0.1 -p 55477 \
 -U article_expiry -d postgres -c 'SELECT 1' >"$lab_root/rotated.out" 2>"$lab_root/rotated.err"; then
 exit 1
fi
grep -q 'password authentication failed' "$lab_root/rotated.err"
printf '%s\n' 'rotating password alone preserves expired deadline'
admin <<'SQL'
SELECT format('ALTER ROLE article_expiry VALID UNTIL %L',
 clock_timestamp()+interval '1 hour') \gexec
SQL
if PGPASSWORD=LAB-ONLY-OLD-PASSWORD psql -X -qAt -h 127.0.0.1 -p 55477 \
 -U article_expiry -d postgres -c 'SELECT 1' >"$lab_root/old.out" 2>"$lab_root/old.err"; then
 exit 1
fi
grep -q 'password authentication failed' "$lab_root/old.err"
printf '%s\n' 'old password rejected after rotation'
PGPASSWORD=LAB-ONLY-NEW-PASSWORD psql -X -qAt -v ON_ERROR_STOP=1 \
 -h 127.0.0.1 -p 55477 -U article_expiry -d postgres \
 -c "SELECT 'new password login='||current_user"
printf "SELECT 'held after rotation='||current_user;\n" >&3
printf '\\q\n' >&3
exec 3>&-
wait "$held_pid"
grep -qx 'held after expiry=article_expiry' "$lab_root/held.out"
grep -qx 'held after rotation=article_expiry' "$lab_root/held.out"
cat "$lab_root/held.out"

The recorded PostgreSQL 18.6 run reports that the deadline has passed. It waits on the database clock after setting a deadline one second ahead, so this is an actual expiry crossing rather than merely a string comparison against a fabricated date. A fresh SCRAM password login is then rejected.

The connection opened before expiry remains usable. Its input pipe keeps the same client and database session alive across the deadline and password replacement. The final output confirms queries under that role after both events. Closing or revoking existing sessions requires an explicit policy beyond this password attribute.

A new local trust login succeeds despite the expired password, as the documented authentication distinction predicts. Replacing the password alone still leaves a fresh password login rejected because the old deadline remains. Once the script supplies a new future deadline, the replacement password connects and the original password is rejected.

These checks isolate three independent observations: the deadline affects new password authentication, the chosen authentication method determines whether that rule applies, and existing sessions are not automatically terminated. The example does not implement a production credential broker or demonstrate revocation of every access path. The private cluster and both fixture passwords are discarded afterward.

Refresh before consumers need another connection

Choose lifetimes with enough allowance for obtaining a replacement, distributing it, refreshing consumers, and recovering from a bounded interruption. Monitor remaining validity at the verifier and the age of the value actually loaded by each consumer. An inventory showing a new version is insufficient if a running process still holds the old one.

Validate replacement credentials through fresh authentication before retiring an old version where the platform’s transition model permits overlap. Confirm that retirement denies new use of the old credential, and separately decide whether sessions must be drained or terminated. During a compromise, emergency revocation may take priority over an uninterrupted transition; expiry should not become a reason to wait for the clock.

Keep clocks synchronized and avoid silently extending lifetimes whenever renewal fails. Record which principal, version, verifier, and deadline were tested without recording secret values. The useful evidence is acceptance before the intended boundary, rejection afterward on the intended path, and an understood effect on existing sessions, not just a date displayed in a management interface.

Sources

PostgreSQL 18 CREATE ROLE, password-only validity and non-password authentication, checked 2026-09-12.
PostgreSQL 18 ALTER ROLE, independent password and validity changes, checked 2026-09-12.
PostgreSQL 18 host authentication, matching authentication rules and trust versus SCRAM, checked 2026-09-12.
OWASP secrets management, expiry, rotation, revocation, and lifecycle control, checked 2026-09-12.