Limits · XB Field Notes

cgroups enforce container budgets within a hierarchy

Fri, 11 Feb 2028 00:00:00 +0000

Kubernetes resource fields become useful constraints only when the node’s execution stack applies them. On Linux, kubelet and the container runtime use cgroups to organize processes and configure resource controllers. The result is a hierarchy, not merely one number attached to a Pod. Check the relevant container group and its inherited constraints before deciding what budget a process can actually consume.

Read the controller that enforces the resource

In cgroup v2, cpu.max expresses CPU bandwidth as quota and period in microseconds. A value of 10000 100000 permits one tenth of a CPU’s time per period for that group. It does not pin the process to one core or reduce the processor’s clock speed. Several runnable threads can consume a group’s allowance faster within the same period.

CPU weight serves a different purpose: it influences relative distribution under contention among eligible groups at the applicable hierarchy level. A Kubernetes CPU request can inform that weight, as well as scheduling placement. A request is not itself a hard runtime ceiling, and a weight does not guarantee an application’s wall-clock latency or exclusive ownership of a CPU.

The memory controller exposes limits and usage separately. memory.max describes the group’s hard memory boundary, while memory.current reports accounted use. Reclaim and OOM behavior are memory mechanisms, not CPU-style periodic throttling. Application heap size alone does not explain all accounted memory, so compare runtime memory metrics with the controller’s view when investigating a limit.

When it breaks

Hierarchical constraints can be tighter than one local file suggests. A child group with no local CPU cap can still be constrained by an ancestor. Container cgroup namespaces may hide parts of that hierarchy from the container’s view. Do not infer unrestricted node capacity from a local max value without checking the node-level context and effective parent constraints.

A multi-container Pod can have separate container limits. One container’s unused allowance is not automatically a transferable promise that another container may exceed its own cap. Pod-level resource features introduce additional configuration and version considerations; they should not be silently substituted for per-container settings. The experiment below deliberately uses only container resources and confirms that distinction in the admitted Pod.

High throttling can explain latency even when a host-wide CPU average looks modest. Conversely, one throttling counter cannot identify every source of delay. Read changes over a relevant interval, correlate workload demand and request latency, and remember that ancestor contention and other resources can also matter. A short synthetic loop establishes enforcement, not an application capacity plan.

Controller files also depend on the environment. cgroup v1 uses different interfaces, and CPU-manager policies or runtime configuration can alter placement and enforcement details. Verify the actual node setup rather than copying a pathname and assuming its meaning. The fixture requires the cgroup v2 interface and reads settings without changing node controller files.

Check it quickly

Use a disposable Linux Kubernetes cluster with cgroup v2, kubectl and python:3.11-slim-bookworm available on the nodes. Save the script as cgroup-enforcement.py and run python3 cgroup-enforcement.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. Namespace creation and deletion permissions are required. The recorded server was Kubernetes 1.34.11 using the kind container runtime setup.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-cgroup-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 podspec = spec()
 podspec['containers'] = []
 for name, request, limit, memory in [('small', '25m', '100m', '64Mi'), ('large', '50m', '200m', '96Mi')]:
 container = spec()['containers'][0]
 container['name'] = name
 container['resources'] = {'requests': {'cpu': request, 'memory': '24Mi'},
 'limits': {'cpu': limit, 'memory': memory}}
 podspec['containers'].append(container)
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'budgets'}, 'spec': podspec})
 wait_for(lambda: obj(namespace, 'pod', 'budgets'), pod_ready)
 snapshot = '''from pathlib import Path
import json,os
p=Path('/sys/fs/cgroup')
assert (p/'cgroup.controllers').exists(), 'this example requires cgroup v2'
print(json.dumps({'cpu_max':(p/'cpu.max').read_text().strip(),
 'cpu_weight':int((p/'cpu.weight').read_text()),
 'memory_max':int((p/'memory.max').read_text()),
 'group_inode':os.stat(p).st_ino}))
'''
 observations = {}
 for name, expected_cpu, expected_memory in [('small', 0.1, 64), ('large', 0.2, 96)]:
 data = json.loads(k('exec', '-n', namespace, 'budgets', '-c', name, '--', 'python3', '-c', snapshot).stdout)
 quota, period = map(int, data['cpu_max'].split())
 assert quota / period == expected_cpu
 assert data['memory_max'] == expected_memory * 1024 * 1024
 observations[name] = data
 assert observations['small']['group_inode'] != observations['large']['group_inode']
 load = '''from pathlib import Path
import json,time
p=Path('/sys/fs/cgroup/cpu.stat')
def counters(): return {k:int(v) for k,v in (line.split() for line in p.read_text().splitlines())}
before=counters(); begin=time.monotonic(); cpu=time.process_time(); value=0
while time.monotonic()-begin < 3: value=(value+1)%1000003
after=counters()
print(json.dumps({'wall_seconds':time.monotonic()-begin,'process_cpu_seconds':time.process_time()-cpu,
 'nr_throttled_delta':after['nr_throttled']-before['nr_throttled'],
 'throttled_usec_delta':after['throttled_usec']-before['throttled_usec']}))
'''
 throttling = json.loads(k('exec', '-n', namespace, 'budgets', '-c', 'small', '--', 'python3', '-c', load).stdout)
 assert throttling['nr_throttled_delta'] > 0 and throttling['throttled_usec_delta'] > 0
 final = obj(namespace, 'pod', 'budgets')
 assert pod_ready(final) and all(c['restartCount'] == 0 for c in final['status']['containerStatuses'])
 assert 'resources' not in final['spec']
 print(json.dumps({'container_groups': observations, 'measured_throttling': throttling,
 'container_restarts': 0, 'pod_level_resource_field_used': False}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

Two containers share one Pod but expose different cgroup directory identities. The smaller container reports cpu.max of 10000 100000 and memory.max of 67108864. The larger reports 20000 100000 and 100663296. Observed CPU weights are six and eleven; those values are runtime mappings, not portable constants that every installation must reproduce.

A three-second busy loop in the smaller container records thirty additional throttled periods and about 2.707 seconds of throttled time. Its process CPU time is about 0.298 seconds over 3.007 seconds elapsed. Both containers remain Ready with zero restarts. All assertions passed, and cleanup removed the namespace.

The test verifies distinct container settings and actual CPU throttling. It does not measure fairness between contending workloads, expose every ancestor setting, or exercise Pod-level resource allocation. Keep those boundaries explicit when turning low-level controller observations into operational conclusions.

Sources

Kubernetes: About cgroup v2, Linux resource-controller integration and prerequisites, checked 2026-09-13.
Kubernetes: Resource Management, container requests and limits passed to the runtime, checked 2026-09-13.
Linux: Control Group v2, hierarchy, CPU bandwidth, weights and memory accounting, checked 2026-09-13.

Set file limits in the process that starts the workload

Mon, 19 Jul 2027 00:00:00 +0000

A shell’s ulimit builtin changes resource limits for that shell and the processes it subsequently starts. For file descriptors, RLIMIT_NOFILE has a current soft value and a hard ceiling. Understanding where those values are inherited is essential when a terminal reports a generous limit but an independently managed service still fails to open files or accept connections.

When it breaks

The soft value is the limit normally enforced for new descriptor allocation. An unprivileged process can raise its soft value only within the existing hard ceiling and can lower its hard limit. Raising a hard limit requires the relevant authority and remains subject to kernel constraints. A low hard value inherited at startup can therefore prevent the application from applying its intended soft setting.

Changing a child’s limit does not change the parent that launched it. Likewise, a command in a newly opened terminal does not retroactively configure a daemon started by systemd. The service’s actual parent and launch configuration determine its initial limits. A shell profile only has an effect when the launch path actually reads and applies that profile.

Bash option selection matters too. Reading ulimit -n normally reports the soft value, but assigning a value without selecting -S or -H changes both soft and hard limits. An operator intending a temporary soft reduction can accidentally lower the hard ceiling as well. Check the shell’s documented behavior instead of treating every similar-looking command as equivalent.

Check it quickly

Run this Python script as an administrator in a disposable Linux container with Bash and Python installed. It launches one experiment under an unprivileged numeric identity and leaves the administrator controller’s limits unchanged. The recorded environment used Linux 6.12.76, Bash 5.2.15, and Python 3.11.2. No existing account configuration or service unit is edited:

import resource,subprocess

before=resource.getrlimit(resource.RLIMIT_NOFILE)
assert before[0]>=64 and before[1]>=64
script=r'''
set -eu
ulimit -S -n 32
ulimit -H -n 64
python3 - <<'CHILD'
import resource
assert resource.getrlimit(resource.RLIMIT_NOFILE)==(32,64)
resource.setrlimit(resource.RLIMIT_NOFILE,(48,64))
assert resource.getrlimit(resource.RLIMIT_NOFILE)==(48,64)
print('first child inherits 32/64 and raises its own soft limit to 48')
CHILD
python3 - <<'CHILD'
import resource
assert resource.getrlimit(resource.RLIMIT_NOFILE)==(32,64)
print('second child still inherits shell values 32/64')
CHILD
if ulimit -S -n 65 2>/dev/null; then exit 11; fi
if ulimit -H -n 128 2>/dev/null; then exit 12; fi
(
 ulimit -n 16
 python3 - <<'CHILD'
import resource
assert resource.getrlimit(resource.RLIMIT_NOFILE)==(16,16)
print('Bash assignment without -S/-H changed both limits in this subshell')
CHILD
)
python3 - <<'CHILD'
import resource
assert resource.getrlimit(resource.RLIMIT_NOFILE)==(32,64)
print('parent shell limits survived child changes and rejected invalid increases')
CHILD
'''
# Run the experiment under an unprivileged numeric identity in this disposable container.
result=subprocess.run(['bash','--noprofile','--norc','-c',script],user=61001,group=61001,extra_groups=[],
 env={'PATH':'/usr/bin:/bin','LC_ALL':'C'},capture_output=True,text=True,timeout=10)
assert result.returncode==0,(result.returncode,result.stderr)
assert resource.getrlimit(resource.RLIMIT_NOFILE)==before
print(result.stdout,end='')
print('administrator controller limits remain unchanged')

The test shell starts by explicitly setting soft 32 and hard 64. Its first Python child inherits that pair, then successfully raises its own soft limit to 48 while keeping hard 64. This demonstrates an allowed increase within the ceiling; it does not require administrative authority inside the child.

A second Python child, launched by the same shell, still receives 32 and 64. The first child’s change stayed in that child, so there is no reverse propagation to the shell or its later children. Both observations read the actual resource-limit API rather than inferring values from the text of a startup script.

Attempts to set soft 65 above hard 64 and to raise the hard value to 128 are rejected in the unprivileged shell. The fixture checks the commands’ failure results. It does not suppress an error and then pretend the larger values were applied, and it does not rely on root accidentally making an otherwise forbidden operation succeed.

Within a subshell, ulimit -n 16 changes both values to 16. After that subshell ends, another child of the original test shell still sees 32 and 64. This separates two rules that often become confused: an assignment can affect both limit fields, while its process-local scope still prevents it from rewriting the parent.

Finally, the administrator controller compares its original limits with its current pair and confirms they are unchanged. All reductions and rejected increases occurred in disposable child processes. The exercise validates inheritance and permission rules, not a production server’s required descriptor budget or the behavior of every possible shell implementation.

Configure and inspect the real launch path

For a systemd service, configure the appropriate LimitNOFILE value in its effective unit settings and start a replacement process when that startup policy changes. For a shell-launched application, apply the intended setting in the launch path before execution. A one-time terminal command is not persistent configuration unless something deliberately runs it again on the next launch.

Inspect the target process’s limits, such as through /proc/PID/limits, and correlate them with the correct running instance. Reading the control shell’s limit answers a different question. Container runtime settings, user managers, and intermediate wrappers can each affect inheritance, so trace the actual process chain when expected values differ.

Choose a budget that includes sockets, ordinary files, pipes, event descriptors, and operational headroom. Raising the ceiling consumes no descriptors by itself, but allowing more live work can increase memory and downstream load. Validate the application’s event mechanism and ownership discipline as well as the numbers; higher limits are useful when the workload can use them correctly.

Sources

getrlimit(2), soft and hard resource limits, privilege checks, and inheritance, checked 2026-09-12.
GNU Bash builtins, ulimit soft/hard selection and assignment behavior, checked 2026-09-12.
systemd 252 execution environment, LimitNOFILE and service-manager execution limits, checked 2026-09-12.

Diagnose descriptor exhaustion in the process that fails

Tue, 27 Oct 2026 00:00:00 +0000

A process needs file descriptors for more than ordinary files. Network sockets, pipes, event interfaces, and duplicated references all compete for entries in its descriptor table. RLIMIT_NOFILE limits the descriptor numbers available to new allocations: the soft limit is one greater than the maximum permitted number. A service can therefore fail to accept work even while the machine has plenty of disk space and memory.

When it breaks

EMFILE reports that the calling process has reached its applicable descriptor limit. ENFILE identifies exhaustion of the system’s open-file resources. They require different investigations. Raising a machine-wide file setting does not by itself change the failed process’s soft limit, and increasing one application’s limit does not repair a system-wide resource shortage.

Exhaustion also damages diagnostics. Opening a log file or enumerating a directory may require another descriptor. An application that tries to inspect /proc/self/fd only after reaching its limit can fail during the inspection itself. Existing descriptors can still be usable, which makes an already established control pipe valuable for reporting the error without allocating another resource.

A rising descriptor count needs context. It can reflect growing legitimate concurrency, a pool warming up, or references whose owners never close them. The curve alone does not prove a leak. Compare descriptor types, workload, and the count after work drains; then trace ownership for references that remain beyond their intended lifetime.

Check it quickly

Save the following as exhaustion.py and run python3 exhaustion.py in a disposable Linux environment. The recorded test used Linux 6.12.76 and Python 3.11.2. The controller starts a separate child, which lowers only its own soft limit to 32. Both inherited limit values must initially permit that setting:

import os,resource,subprocess,sys

before=resource.getrlimit(resource.RLIMIT_NOFILE)
child=r'''
import errno,os,resource,socket,sys
soft,hard=resource.getrlimit(resource.RLIMIT_NOFILE)
assert soft>=32 and hard>=32
resource.setrlimit(resource.RLIMIT_NOFILE,(32,hard))
opened=[]
def emfile(operation):
 try:operation()
 except OSError as error:assert error.errno==errno.EMFILE,error
 else:raise AssertionError('expected per-process descriptor exhaustion')
try:
 while True:
 try:opened.append(os.open('/dev/null',os.O_RDONLY))
 except OSError as error:
 assert error.errno==errno.EMFILE,error;break
 assert opened and max(opened)==31
 emfile(lambda:socket.socket(socket.AF_UNIX,socket.SOCK_STREAM))
 emfile(lambda:os.dup(0))
 emfile(lambda:os.listdir('/proc/self/fd'))
 print('exhausted',flush=True)
 assert sys.stdin.readline()=='continue\n'
 released=opened.pop();os.close(released)
 replacement=os.open('/dev/null',os.O_RDONLY);opened.append(replacement)
 assert replacement==released
 print('EMFILE affects open, socket, dup, and self-enumeration; releasing one slot permits one open')
finally:
 for fd in opened:os.close(fd)
assert os.listdir('/proc/self/fd')
print('after cleanup, descriptor inspection succeeds again')
'''
process=subprocess.Popen([sys.executable,'-c',child],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
try:
 assert process.stdout.readline().strip()=='exhausted'
 descriptor=os.open('/dev/null',os.O_RDONLY);os.close(descriptor)
 print('controller can still open files while child is exhausted')
 process.stdin.write('continue\n');process.stdin.flush()
 output,error=process.communicate(timeout=10)
 assert process.returncode==0,error;print(output,end='')
finally:
 if process.poll() is None:process.kill();process.wait()
assert resource.getrlimit(resource.RLIMIT_NOFILE)==before

The child repeatedly opens /dev/null until allocation returns EMFILE. Its highest allocated number is 31, consistent with a soft limit of 32. The number of new opens is smaller than 32 because standard streams and any other inherited descriptors already occupy entries. Count the occupied table, rather than assuming the limit describes additional opens still available.

While full, the child checks socket creation, duplication of standard input, and enumeration of its own descriptor directory. All fail with EMFILE. The directory has not disappeared; reading its entries requires an open directory handle that this process can no longer allocate. This separates a diagnostic failure caused by resource exhaustion from a missing-path problem.

The child announces its state through the output pipe opened before exhaustion. The controller then successfully opens and closes /dev/null while the child remains full. That concurrent observation establishes the per-process boundary. The fixture does not intentionally exhaust the host’s system-wide file table, which would require a different, more disruptive experiment.

After receiving a continue message over its existing input pipe, the child closes one owned descriptor. The next open succeeds and reuses that available number. This demonstrates a precise recovery condition: one released slot permits one new allocation. It does not restore headroom for an arbitrary burst of incoming connections or a multi-step operation requiring several handles.

Finally, the child closes all descriptors it deliberately allocated and can enumerate /proc/self/fd again. The controller verifies its own resource limits are unchanged. Cleanup is in a finally block, and the controller terminates a child that fails to finish; the test leaves neither a permanently reduced parent limit nor a background workload consuming descriptors.

Recover capacity and repair ownership

During an incident, inspect the failing process’s limits and descriptor types from an authorized external process with available resources. Correlate the PID with the intended service instance. A shell’s ulimit output describes that shell, so it is insufficient evidence for a separately launched daemon’s budget.

Reduce new admissions when concurrency exceeds the intended budget, and release references when their work completes. If a leak is confirmed, repair its ownership path and exercise cancellation and error exits. Restarting can temporarily recover slots, but repeated exhaustion after similar work indicates that the underlying lifetime or capacity problem remains.

Set a launch-time limit with room for listeners, active work, files, pipes, and operational inspection. Validate the actual running process afterward. Increasing the allowance should follow a resource budget and measurements; it can permit more useful concurrency, but it can also move the bottleneck into memory, downstream connections, or the application’s event-handling mechanism.

Sources

getrlimit(2), RLIMIT_NOFILE, allocation limits, and EMFILE, checked 2026-09-12.
open(2), per-process EMFILE versus system-wide ENFILE, checked 2026-09-12.
dup(2), descriptor allocation and exhaustion when duplicating a reference, checked 2026-09-12.