Processes · XB Field Notes

A signal has a disposition, a mask, and delivery rules

Mon, 23 Nov 2026 00:00:00 +0000

Signals notify processes or threads of events using kernel-defined delivery rules. A signal’s disposition selects a default action, an installed handler, or ignoring it where permitted. A thread’s mask can block delivery of eligible signals while they remain pending. Separating disposition, masking, and pending state is essential when a program appears to miss repeated notifications or fails to stop as expected.

When it breaks

Ordinary signals are not a reliable event counter. Several occurrences of one standard signal can coalesce while it is already pending. An application that treats each SIGUSR1 as one queued work item can therefore lose that counting relationship. Real-time signals have queuing semantics, but their resources are finite and still require a deliberate application protocol.

Signal names do not promise application features. SIGHUP commonly requests reload in particular daemons, but its default action is termination. SIGTERM also terminates by default; it becomes a graceful shutdown request only when the application implements the necessary behavior. Successfully sending a signal does not prove that configuration reloaded or that in-flight work drained.

SIGKILL and SIGSTOP cannot be caught, ignored, or blocked. KILL provides no opportunity for application cleanup, and even it should not be described as guaranteeing instantaneous resource release in every kernel state. A supervisor must observe actual completion. Use symbolic signal names instead of assuming all numerical assignments are identical across platforms.

Check it quickly

Save the following as signals.py and run python3 signals.py in a disposable Linux environment. The recorded run used Linux 6.12.76 and Python 3.11.2. The script signals only itself and children it creates, restores its original mask, and confines cleanup markers to a temporary directory:

from pathlib import Path
from tempfile import TemporaryDirectory
import errno,os,signal,subprocess,sys

watched={signal.SIGUSR1,signal.SIGRTMIN}
old=signal.pthread_sigmask(signal.SIG_BLOCK,watched)
try:
 for _ in range(5):os.kill(os.getpid(),signal.SIGUSR1)
 assert signal.SIGUSR1 in signal.sigpending()
 assert signal.sigwait({signal.SIGUSR1})==signal.SIGUSR1
 assert signal.sigtimedwait({signal.SIGUSR1},0) is None
 for _ in range(3):os.kill(os.getpid(),signal.SIGRTMIN)
 for _ in range(3):assert signal.sigtimedwait({signal.SIGRTMIN},0).si_signo==signal.SIGRTMIN
 assert signal.sigtimedwait({signal.SIGRTMIN},0) is None
 print('five blocked standard signals coalesce into one; three blocked real-time signals remain queued')
finally:signal.pthread_sigmask(signal.SIG_SETMASK,old)
for sig in [signal.SIGKILL,signal.SIGSTOP]:
 try:signal.signal(sig,signal.SIG_IGN)
 except OSError as error:assert error.errno==errno.EINVAL
 else:raise AssertionError('uncatchable signal accepted a disposition change')
old=signal.pthread_sigmask(signal.SIG_BLOCK,{signal.SIGKILL,signal.SIGSTOP})
assert not {signal.SIGKILL,signal.SIGSTOP}&signal.pthread_sigmask(signal.SIG_BLOCK,set())
signal.pthread_sigmask(signal.SIG_SETMASK,old)
print('SIGKILL and SIGSTOP cannot be ignored or added to the blocked mask')
worker=r'''
from pathlib import Path
import signal,sys
if int(sys.argv[1]):signal.signal(signal.SIGTERM,lambda *_:sys.exit(0))
try:
 print('ready',flush=True)
 while True:signal.pause()
finally:Path(sys.argv[2]).write_text('cleanup')
'''
with TemporaryDirectory(prefix='xbcatm-signals-') as directory:
 for name,sig,handled,expected,cleaned in [
 ('hup-default',signal.SIGHUP,False,-signal.SIGHUP,False),
 ('term-default',signal.SIGTERM,False,-signal.SIGTERM,False),
 ('term-handled',signal.SIGTERM,True,0,True),
 ('kill',signal.SIGKILL,True,-signal.SIGKILL,False)]:
 marker=Path(directory)/name
 process=subprocess.Popen([sys.executable,'-c',worker,str(int(handled)),str(marker)],stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 try:
 assert process.stdout.readline().strip()=='ready'
 os.kill(process.pid,sig);output,error=process.communicate(timeout=5)
 assert process.returncode==expected,(name,process.returncode,error)
 assert marker.exists()==cleaned,name
 print(name,'returncode',process.returncode,'cleanup',cleaned)
 finally:
 if process.poll() is None:process.kill();process.wait()

First, the script blocks SIGUSR1, sends it to itself five times, and confirms it is pending. One synchronous wait consumes it; a second zero-time wait finds nothing. Blocking before sending makes this demonstration deterministic. It establishes coalescing of a pending standard signal, not a timing-dependent guess based on how many asynchronous callbacks happened to print.

The same experiment sends three instances of SIGRTMIN while blocked. Three waits retrieve three instances, followed by an empty wait. This contrasts the queuing rules using synchronous consumption. The test does not measure queue capacity, priority across different real-time signal numbers, or delivery of an application payload.

Attempts to ignore KILL and STOP fail with EINVAL. Adding them to the blocked mask also leaves them absent from the actual mask. These checks distinguish a rejected disposition change from a mask request whose unmaskable members are ignored; neither operation makes those signals controllable by the application.

Four child-process cases then separate sending from cleanup. Default HUP and default TERM terminate their children without creating the marker in Python’s finally block. A child with an explicit TERM handler exits through Python and creates the marker. A final child has that same TERM handler, but KILL terminates it without running the cleanup.

The recorded Python subprocess return codes are negative for signal termination: minus the terminating signal number. The handled TERM case exits normally with zero. Those values are Python’s process-status representation, not a universal shell exit-code convention. The fixture checks both the status and marker so neither one alone substitutes for observing the cleanup behavior.

Put work in the appropriate execution context

In native signal handlers, only async-signal-safe operations are suitable. Ordinary logging, allocation, and many library calls can deadlock or corrupt state if interrupted midway. A common design records a minimal notification and lets normal program flow perform coordinated shutdown. The exact mechanism should fit the runtime and its supported event loop.

Python handlers run through the interpreter’s signal machinery in the main thread, rather than executing arbitrary Python inside the low-level C handler. This is why the small TERM example can raise SystemExit. It is not a model for doing arbitrary work in a C handler, nor a guarantee that a long-running native operation will promptly yield to Python cleanup.

Define the expected response, an observation mechanism, and an appropriate deadline when controlling a service. A reload should expose evidence that the new configuration took effect; a shutdown should stop admissions, finish or abandon work according to policy, and report completion. Escalation changes termination behavior, but cannot retroactively supply cleanup the application never implemented.

Sources

signal(7), default actions, masks, standard coalescing, and real-time queues, checked 2026-09-12.
signal-safety(7), async-signal-safe restrictions for native signal handlers, checked 2026-09-12.
Python 3.11 signal documentation, Python handler execution, synchronous waiting, and signal masks, checked 2026-09-12.
kill(2), signal delivery permissions and return semantics, checked 2026-09-12.

Diagnose descriptor exhaustion in the process that fails

Tue, 27 Oct 2026 00:00:00 +0000

A process needs file descriptors for more than ordinary files. Network sockets, pipes, event interfaces, and duplicated references all compete for entries in its descriptor table. RLIMIT_NOFILE limits the descriptor numbers available to new allocations: the soft limit is one greater than the maximum permitted number. A service can therefore fail to accept work even while the machine has plenty of disk space and memory.

When it breaks

EMFILE reports that the calling process has reached its applicable descriptor limit. ENFILE identifies exhaustion of the system’s open-file resources. They require different investigations. Raising a machine-wide file setting does not by itself change the failed process’s soft limit, and increasing one application’s limit does not repair a system-wide resource shortage.

Exhaustion also damages diagnostics. Opening a log file or enumerating a directory may require another descriptor. An application that tries to inspect /proc/self/fd only after reaching its limit can fail during the inspection itself. Existing descriptors can still be usable, which makes an already established control pipe valuable for reporting the error without allocating another resource.

A rising descriptor count needs context. It can reflect growing legitimate concurrency, a pool warming up, or references whose owners never close them. The curve alone does not prove a leak. Compare descriptor types, workload, and the count after work drains; then trace ownership for references that remain beyond their intended lifetime.

Check it quickly

Save the following as exhaustion.py and run python3 exhaustion.py in a disposable Linux environment. The recorded test used Linux 6.12.76 and Python 3.11.2. The controller starts a separate child, which lowers only its own soft limit to 32. Both inherited limit values must initially permit that setting:

import os,resource,subprocess,sys

before=resource.getrlimit(resource.RLIMIT_NOFILE)
child=r'''
import errno,os,resource,socket,sys
soft,hard=resource.getrlimit(resource.RLIMIT_NOFILE)
assert soft>=32 and hard>=32
resource.setrlimit(resource.RLIMIT_NOFILE,(32,hard))
opened=[]
def emfile(operation):
 try:operation()
 except OSError as error:assert error.errno==errno.EMFILE,error
 else:raise AssertionError('expected per-process descriptor exhaustion')
try:
 while True:
 try:opened.append(os.open('/dev/null',os.O_RDONLY))
 except OSError as error:
 assert error.errno==errno.EMFILE,error;break
 assert opened and max(opened)==31
 emfile(lambda:socket.socket(socket.AF_UNIX,socket.SOCK_STREAM))
 emfile(lambda:os.dup(0))
 emfile(lambda:os.listdir('/proc/self/fd'))
 print('exhausted',flush=True)
 assert sys.stdin.readline()=='continue\n'
 released=opened.pop();os.close(released)
 replacement=os.open('/dev/null',os.O_RDONLY);opened.append(replacement)
 assert replacement==released
 print('EMFILE affects open, socket, dup, and self-enumeration; releasing one slot permits one open')
finally:
 for fd in opened:os.close(fd)
assert os.listdir('/proc/self/fd')
print('after cleanup, descriptor inspection succeeds again')
'''
process=subprocess.Popen([sys.executable,'-c',child],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
try:
 assert process.stdout.readline().strip()=='exhausted'
 descriptor=os.open('/dev/null',os.O_RDONLY);os.close(descriptor)
 print('controller can still open files while child is exhausted')
 process.stdin.write('continue\n');process.stdin.flush()
 output,error=process.communicate(timeout=10)
 assert process.returncode==0,error;print(output,end='')
finally:
 if process.poll() is None:process.kill();process.wait()
assert resource.getrlimit(resource.RLIMIT_NOFILE)==before

The child repeatedly opens /dev/null until allocation returns EMFILE. Its highest allocated number is 31, consistent with a soft limit of 32. The number of new opens is smaller than 32 because standard streams and any other inherited descriptors already occupy entries. Count the occupied table, rather than assuming the limit describes additional opens still available.

While full, the child checks socket creation, duplication of standard input, and enumeration of its own descriptor directory. All fail with EMFILE. The directory has not disappeared; reading its entries requires an open directory handle that this process can no longer allocate. This separates a diagnostic failure caused by resource exhaustion from a missing-path problem.

The child announces its state through the output pipe opened before exhaustion. The controller then successfully opens and closes /dev/null while the child remains full. That concurrent observation establishes the per-process boundary. The fixture does not intentionally exhaust the host’s system-wide file table, which would require a different, more disruptive experiment.

After receiving a continue message over its existing input pipe, the child closes one owned descriptor. The next open succeeds and reuses that available number. This demonstrates a precise recovery condition: one released slot permits one new allocation. It does not restore headroom for an arbitrary burst of incoming connections or a multi-step operation requiring several handles.

Finally, the child closes all descriptors it deliberately allocated and can enumerate /proc/self/fd again. The controller verifies its own resource limits are unchanged. Cleanup is in a finally block, and the controller terminates a child that fails to finish; the test leaves neither a permanently reduced parent limit nor a background workload consuming descriptors.

Recover capacity and repair ownership

During an incident, inspect the failing process’s limits and descriptor types from an authorized external process with available resources. Correlate the PID with the intended service instance. A shell’s ulimit output describes that shell, so it is insufficient evidence for a separately launched daemon’s budget.

Reduce new admissions when concurrency exceeds the intended budget, and release references when their work completes. If a leak is confirmed, repair its ownership path and exercise cancellation and error exits. Restarting can temporarily recover slots, but repeated exhaustion after similar work indicates that the underlying lifetime or capacity problem remains.

Set a launch-time limit with room for listeners, active work, files, pipes, and operational inspection. Validate the actual running process afterward. Increasing the allowance should follow a resource budget and measurements; it can permit more useful concurrency, but it can also move the bottleneck into memory, downstream connections, or the application’s event-handling mechanism.

Sources

getrlimit(2), RLIMIT_NOFILE, allocation limits, and EMFILE, checked 2026-09-12.
open(2), per-process EMFILE versus system-wide ENFILE, checked 2026-09-12.
dup(2), descriptor allocation and exhaustion when duplicating a reference, checked 2026-09-12.