cgroups enforce container budgets within a hierarchy

Fri, 11 Feb 2028 00:00:00 +0000

Kubernetes resource fields become useful constraints only when the node’s execution stack applies them. On Linux, kubelet and the container runtime use cgroups to organize processes and configure resource controllers. The result is a hierarchy, not merely one number attached to a Pod. Check the relevant container group and its inherited constraints before deciding what budget a process can actually consume.

Read the controller that enforces the resource

In cgroup v2, cpu.max expresses CPU bandwidth as quota and period in microseconds. A value of 10000 100000 permits one tenth of a CPU’s time per period for that group. It does not pin the process to one core or reduce the processor’s clock speed. Several runnable threads can consume a group’s allowance faster within the same period.

CPU weight serves a different purpose: it influences relative distribution under contention among eligible groups at the applicable hierarchy level. A Kubernetes CPU request can inform that weight, as well as scheduling placement. A request is not itself a hard runtime ceiling, and a weight does not guarantee an application’s wall-clock latency or exclusive ownership of a CPU.

The memory controller exposes limits and usage separately. memory.max describes the group’s hard memory boundary, while memory.current reports accounted use. Reclaim and OOM behavior are memory mechanisms, not CPU-style periodic throttling. Application heap size alone does not explain all accounted memory, so compare runtime memory metrics with the controller’s view when investigating a limit.

When it breaks

Hierarchical constraints can be tighter than one local file suggests. A child group with no local CPU cap can still be constrained by an ancestor. Container cgroup namespaces may hide parts of that hierarchy from the container’s view. Do not infer unrestricted node capacity from a local max value without checking the node-level context and effective parent constraints.

A multi-container Pod can have separate container limits. One container’s unused allowance is not automatically a transferable promise that another container may exceed its own cap. Pod-level resource features introduce additional configuration and version considerations; they should not be silently substituted for per-container settings. The experiment below deliberately uses only container resources and confirms that distinction in the admitted Pod.

High throttling can explain latency even when a host-wide CPU average looks modest. Conversely, one throttling counter cannot identify every source of delay. Read changes over a relevant interval, correlate workload demand and request latency, and remember that ancestor contention and other resources can also matter. A short synthetic loop establishes enforcement, not an application capacity plan.

Controller files also depend on the environment. cgroup v1 uses different interfaces, and CPU-manager policies or runtime configuration can alter placement and enforcement details. Verify the actual node setup rather than copying a pathname and assuming its meaning. The fixture requires the cgroup v2 interface and reads settings without changing node controller files.

Check it quickly

Use a disposable Linux Kubernetes cluster with cgroup v2, kubectl and python:3.11-slim-bookworm available on the nodes. Save the script as cgroup-enforcement.py and run python3 cgroup-enforcement.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. Namespace creation and deletion permissions are required. The recorded server was Kubernetes 1.34.11 using the kind container runtime setup.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-cgroup-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 podspec = spec()
 podspec['containers'] = []
 for name, request, limit, memory in [('small', '25m', '100m', '64Mi'), ('large', '50m', '200m', '96Mi')]:
 container = spec()['containers'][0]
 container['name'] = name
 container['resources'] = {'requests': {'cpu': request, 'memory': '24Mi'},
 'limits': {'cpu': limit, 'memory': memory}}
 podspec['containers'].append(container)
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'budgets'}, 'spec': podspec})
 wait_for(lambda: obj(namespace, 'pod', 'budgets'), pod_ready)
 snapshot = '''from pathlib import Path
import json,os
p=Path('/sys/fs/cgroup')
assert (p/'cgroup.controllers').exists(), 'this example requires cgroup v2'
print(json.dumps({'cpu_max':(p/'cpu.max').read_text().strip(),
 'cpu_weight':int((p/'cpu.weight').read_text()),
 'memory_max':int((p/'memory.max').read_text()),
 'group_inode':os.stat(p).st_ino}))
'''
 observations = {}
 for name, expected_cpu, expected_memory in [('small', 0.1, 64), ('large', 0.2, 96)]:
 data = json.loads(k('exec', '-n', namespace, 'budgets', '-c', name, '--', 'python3', '-c', snapshot).stdout)
 quota, period = map(int, data['cpu_max'].split())
 assert quota / period == expected_cpu
 assert data['memory_max'] == expected_memory * 1024 * 1024
 observations[name] = data
 assert observations['small']['group_inode'] != observations['large']['group_inode']
 load = '''from pathlib import Path
import json,time
p=Path('/sys/fs/cgroup/cpu.stat')
def counters(): return {k:int(v) for k,v in (line.split() for line in p.read_text().splitlines())}
before=counters(); begin=time.monotonic(); cpu=time.process_time(); value=0
while time.monotonic()-begin < 3: value=(value+1)%1000003
after=counters()
print(json.dumps({'wall_seconds':time.monotonic()-begin,'process_cpu_seconds':time.process_time()-cpu,
 'nr_throttled_delta':after['nr_throttled']-before['nr_throttled'],
 'throttled_usec_delta':after['throttled_usec']-before['throttled_usec']}))
'''
 throttling = json.loads(k('exec', '-n', namespace, 'budgets', '-c', 'small', '--', 'python3', '-c', load).stdout)
 assert throttling['nr_throttled_delta'] > 0 and throttling['throttled_usec_delta'] > 0
 final = obj(namespace, 'pod', 'budgets')
 assert pod_ready(final) and all(c['restartCount'] == 0 for c in final['status']['containerStatuses'])
 assert 'resources' not in final['spec']
 print(json.dumps({'container_groups': observations, 'measured_throttling': throttling,
 'container_restarts': 0, 'pod_level_resource_field_used': False}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

Two containers share one Pod but expose different cgroup directory identities. The smaller container reports cpu.max of 10000 100000 and memory.max of 67108864. The larger reports 20000 100000 and 100663296. Observed CPU weights are six and eleven; those values are runtime mappings, not portable constants that every installation must reproduce.

A three-second busy loop in the smaller container records thirty additional throttled periods and about 2.707 seconds of throttled time. Its process CPU time is about 0.298 seconds over 3.007 seconds elapsed. Both containers remain Ready with zero restarts. All assertions passed, and cleanup removed the namespace.

The test verifies distinct container settings and actual CPU throttling. It does not measure fairness between contending workloads, expose every ancestor setting, or exercise Pod-level resource allocation. Keep those boundaries explicit when turning low-level controller observations into operational conclusions.

Sources

Kubernetes: About cgroup v2, Linux resource-controller integration and prerequisites, checked 2026-09-13.
Kubernetes: Resource Management, container requests and limits passed to the runtime, checked 2026-09-13.
Linux: Control Group v2, hierarchy, CPU bandwidth, weights and memory accounting, checked 2026-09-13.

Cgroup · XB Field Notes

cgroups enforce container budgets within a hierarchy

Read the controller that enforces the resource

When it breaks

Check it quickly

Sources