A Kubernetes Secret is an API object for confidential values that workloads need. It gives operators a dedicated object type, consumption paths, and access-control targets. It does not make data unreadable to every other actor. Design the boundary around who can retrieve the object, create consumers, inspect their output, and administer the nodes that run them.

Encoding and protection are different layers

In the JSON API representation, Secret data values use base64; stringData is a write convenience for supplying strings. Base64 can be reversed without a key and is not encryption. The representation does not prove how a particular cluster protects its backing storage. Kubernetes documentation describes Secrets as unencrypted at rest by default unless encryption is configured. Verify the actual API-server and storage setup rather than inferring it from encoded output.

Storage encryption also does not prevent an authorized API reader from obtaining the data. RBAC rules for get, list, and watch can expose Secret contents, so a broad listing permission is not merely access to object names. Protect backups and diagnostic output too. A working application eventually receives usable values, which means a compromised authorized consumer remains inside the data’s trust boundary.

Secrets are intended for small values, with a one-MiB object limit. Mount only required keys and restrict access to the relevant containers. Unix file modes can narrow access within a workload, but do not replace API authorization or control of workload creation. Environment injection also has an exposure and lifecycle tradeoff; choose it from the application’s needs rather than assuming it is automatically safer than a file.

When it breaks

Denying get on Secrets does not necessarily deny indirect access. A subject allowed to create Pods in a namespace can normally name a Secret for a volume in its Pod specification. Kubelet then supplies the data to that workload. Permissions to create suitable controllers can have similar consequences. Review workload-creation rights and applicable admission controls alongside direct data-reading rules.

Broadly mounting every key expands the set of values a process can read. Selecting one key through volume items limits that particular projection, but a privileged creator may be able to request a different projection. Do not mistake a narrow manifest for proof that the actor writing it has equally narrow authority. The intended consumer and the authority to define consumers are separate parts of the boundary.

Rotation requires more than editing an API object. Delivery, application reload, credential issuance, and revocation have their own timing. Environment snapshots and subPath mounts do not automatically refresh like a normal Secret projection. An updated file also does not force an application to discard a value it already cached. Test the issuing system and consumers together before claiming that old credentials are no longer accepted.

Check it quickly

Use a disposable Kubernetes cluster with administrator access for test impersonation and python:3.11-slim-bookworm available on its nodes. Save this as secret-access.py and run python3 secret-access.py KUBECONFIG_PATH CONTEXT against its dedicated configuration. Kubernetes 1.34.11 was used for the recorded run. Every value in this fixture is deliberately synthetic, and output reports checks without printing the stored value.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
    result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
                            capture_output=True, text=True, timeout=150)
    if expected is not None:
        assert result.returncode == expected, result.stderr
    return result


def obj(namespace, kind, name=None, selector=None):
    args = ['get', kind, '-n', namespace, '-o', 'json']
    if name:
        args.append(name)
    if selector:
        args += ['-l', selector]
    return json.loads(k(*args).stdout)


def apply(namespace, data):
    return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
    until = time.monotonic() + seconds
    while time.monotonic() < until:
        value = read()
        if ready(value):
            return value
        time.sleep(0.5)
    raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
    return any(c['type'] == 'Ready' and c['status'] == 'True'
               for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
    return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
            'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
                            'command': ['python3', '-c', script],
                            'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
                                          'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

import base64
namespace = 'notes-secrets-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
    demo = 'DEMO_ONLY_NOT_A_CREDENTIAL'
    k('create', '-n', namespace, '-f', '-', payload={'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'example'},
                      'type': 'Opaque', 'stringData': {'needed': demo, 'unused': 'UNUSED_DEMO'}})
    stored = obj(namespace, 'secret', 'example')
    assert base64.b64decode(stored['data']['needed']).decode() == demo
    apply(namespace, {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'creator'},
                      'automountServiceAccountToken': False})
    apply(namespace, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'Role',
        'metadata': {'name': 'pod-creator'},
        'rules': [{'apiGroups': [''], 'resources': ['pods'], 'verbs': ['create', 'get']},
                  {'apiGroups': [''], 'resources': ['pods/log'], 'verbs': ['get']}]})
    apply(namespace, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'RoleBinding',
        'metadata': {'name': 'pod-creator'},
        'subjects': [{'kind': 'ServiceAccount', 'name': 'creator', 'namespace': namespace}],
        'roleRef': {'apiGroup': 'rbac.authorization.k8s.io', 'kind': 'Role', 'name': 'pod-creator'}})
    identity = 'system:serviceaccount:' + namespace + ':creator'
    denied = k('--as', identity, 'get', 'secret', 'example', '-n', namespace, expected=None)
    assert denied.returncode != 0 and 'Forbidden' in denied.stderr
    probe = "from pathlib import Path\nimport json,time\nroot=Path('/example')\nprint(json.dumps({'synthetic_value_matches':(root/'one').read_text()=='DEMO_ONLY_NOT_A_CREDENTIAL',\n                  'unused_key_absent':not (root/'unused').exists(),\n                  'service_account_token_absent':not Path('/var/run/secrets/kubernetes.io/serviceaccount/token').exists()}),flush=True)\ntime.sleep(600)"
    podspec = spec(probe)
    podspec['serviceAccountName'] = 'creator'
    podspec['volumes'] = [{'name': 'example', 'secret': {'secretName': 'example',
        'items': [{'key': 'needed', 'path': 'one'}], 'defaultMode': 0o400}}]
    podspec['containers'][0]['volumeMounts'] = [{'name': 'example', 'mountPath': '/example', 'readOnly': True}]
    definition = {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'reader', 'namespace': namespace},
                  'spec': podspec}
    k('--as', identity, 'create', '-f', '-', payload=definition)
    wait_for(lambda: obj(namespace, 'pod', 'reader'), pod_ready)
    output = wait_for(lambda: k('--as', identity, 'logs', '-n', namespace, 'reader').stdout.strip(), bool)
    observed = json.loads(output)
    assert all(observed.values())
    print(json.dumps({'api_base64_decodes_to_synthetic_value': True,
                      'direct_secret_get_forbidden': True, 'same_identity_can_create_consuming_pod': True,
                      'mount_observation': observed, 'etcd_encryption_tested': False}))
finally:
    k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The script creates a Secret with two dummy keys and confirms that its API base64 representation decodes to the supplied synthetic text. It binds a service account to a Role permitting Pod creation and inspection of Pod logs, without permitting Secret reads. A request impersonating that account to get the Secret returns Forbidden.

The same impersonated subject then creates a Pod that mounts only the needed key. Its own program verifies the expected dummy value, absence of the unused key, and absence of an automatically mounted service-account token. The subject can read those boolean results through its permitted log access. All checks pass, demonstrating the indirect workload path despite the direct denial.

The namespace is removed afterward. No real credential, issued service-account token, backing-store encryption, or rotation process was tested. The experiment establishes authorization and projection behavior, not a complete secret-management system. Assess both direct readers and actors who can create or alter readers before describing a Secret as narrowly scoped.

Sources