Unix · XB Field Notes

Fork creates a process; exec replaces its program

Tue, 12 Oct 2027 00:00:00 +0000

Fork followed by exec is one common Unix process-launch model. fork creates a child process, while a successful execve replaces the calling process’s program image. They solve different problems and have different inheritance rules. Understanding the distinction matters when arranging input, output, environment, and lifecycle management for a command started by another program.

When it breaks

Fork does not make all subsequent state independent. Ordinary private memory diverges between parent and child, commonly through copy-on-write, but shared mappings remain shared. Inherited descriptors also refer to the same open file descriptions as their parent counterparts, so ordinary file offsets can move in both processes when either one reads. Copying the descriptor table does not independently reopen each file.

Exec does not simply preserve every launch setting. Descriptors marked close-on-exec close during successful execution, while other eligible descriptors survive. execve receives an explicit environment array instead of automatically reproducing the caller’s current environment. Caught signal dispositions reset and old memory mappings disappear, even though process identity and several other attributes continue.

Forking a multithreaded process needs particular care. The child contains only the calling thread but can inherit locks whose owning threads are absent. Arbitrary library work in the gap can deadlock. The allowed pre-exec operations are therefore constrained; a convenient single-threaded demonstration must not become an unrestricted recipe for manipulating a live threaded server after fork.

Check it quickly

Save this as forkexec.py and run python3 forkexec.py in a disposable Linux environment. The recorded run used Linux 6.12.76 and Python 3.11.2. This fixture is deliberately single-threaded. It uses one private Python value, one explicitly shared anonymous mapping, a temporary data file, and a pipe for a small result:

from pathlib import Path
from tempfile import TemporaryDirectory
import errno,json,mmap,os,sys

program=r'''
import errno,json,os,sys
original_pid,kept,dropped,report=map(int,sys.argv[1:])
assert os.getpid()==original_pid
try:os.fstat(dropped)
except OSError as error:assert error.errno==errno.EBADF
else:raise AssertionError('close-on-exec descriptor survived')
assert os.read(kept,2)==b'ef'
assert os.environ['XB_FORK_VALUE']=='child-value'
os.write(report,json.dumps({'pid_unchanged':True,'kept_descriptor':'ef','close_on_exec':'closed','explicit_environment':'child-value'}).encode())
'''
with TemporaryDirectory(prefix='xbcatm-fork-') as directory:
 data=Path(directory)/'data';data.write_bytes(b'abcdef')
 descriptor=os.open(data,os.O_RDONLY);dropped=os.dup(descriptor)
 read_end,write_end=os.pipe();os.set_inheritable(descriptor,True);os.set_inheritable(write_end,True)
 assert not os.get_inheritable(dropped)
 private=['parent'];shared=mmap.mmap(-1,8,flags=mmap.MAP_SHARED);shared[:]=b'parent!!'
 assert os.read(descriptor,2)==b'ab'
 sys.stdout.flush();pid=os.fork()
 if pid==0:
 try:
 os.close(read_end);private[0]='child';shared[:]=b'child!!!'
 assert os.read(descriptor,2)==b'cd'
 try:os.execve(str(Path(directory)/'missing-program'),['missing-program'],{})
 except FileNotFoundError:pass
 else:raise AssertionError('missing exec unexpectedly succeeded')
 os.execve(sys.executable,[sys.executable,'-c',program,str(os.getpid()),str(descriptor),str(dropped),str(write_end)],{'XB_FORK_VALUE':'child-value'})
 except BaseException as error:
 os.write(write_end,('ERROR '+repr(error)).encode());os._exit(1)
 os.close(write_end)
 try:
 waited,status=os.waitpid(pid,0);report=os.read(read_end,4096)
 assert waited==pid and os.WIFEXITED(status) and os.WEXITSTATUS(status)==0,report
 result=json.loads(report);assert result['pid_unchanged']
 assert private==['parent'] and shared[:]==b'child!!!'
 assert os.lseek(descriptor,0,os.SEEK_CUR)==6
 print('fork: private Python state diverges, MAP_SHARED changes and open-description offsets remain shared')
 print('missing exec returns to the child; successful exec keeps its PID, honors close-on-exec, and receives an explicit environment')
 print('parent reaped a successful exit with waitpid and retained its original private value')
 finally:
 shared.close()
 for fd in [descriptor,dropped,read_end]:os.close(fd)

Before fork, the parent reads ab from abcdef. The child changes its private Python list and the shared mapping, then reads cd through the inherited descriptor. After the child finishes, the parent’s list still contains its original value, but the shared mapping contains the child’s bytes. Both results are required to describe memory inheritance accurately.

The child first tries to execute a nonexistent file. That ordinary missing-path failure raises FileNotFoundError, and the child continues running its existing program. It then successfully executes Python again with a supplied program and an explicit environment. The example verifies this recoverable failure case; it does not generalize every rare execution failure into the same recovery behavior.

The replacement program compares its PID with the child’s PID passed before exec and finds them equal. Exec has replaced code within the child, not created a third process. The explicitly provided environment contains the fixture value, which the replacement reads directly; it does not depend on a login shell or an implicit profile-loading step.

The retained data descriptor reads ef, showing that the original parent’s read, the child’s read, and the replacement program’s read advanced one shared open-description offset. Meanwhile, the descriptor left non-inheritable fails inspection with EBADF in the replacement. The fixture deliberately marks only the descriptors needed across exec as inheritable.

The parent uses waitpid to collect the child’s status and confirms a normal zero exit. It then observes offset six through its own descriptor and closes its resources. Waiting obtains lifecycle information and reaps the exited child; the child’s numerical PID alone is not a substitute for collecting completion or verifying which launch succeeded.

Make launch ownership explicit

Define which descriptors a child needs, close unused pipe ends, and choose environment values deliberately. An extra inherited pipe writer can prevent a reader from seeing EOF even after the intended producer exits. Descriptor lifetime is therefore part of the communication protocol, not merely a cleanup detail after program startup.

Prefer the runtime’s supported subprocess or spawn interface when it expresses the needed setup. Such interfaces can use mechanisms other than a literal fork/exec pair. Their documented descriptor and environment behavior is the relevant application contract; assuming every launcher follows an identical internal sequence can produce incorrect portability and safety conclusions.

Distinguish launch failure, program failure, and failure to observe completion. A successful exec never returns to the previous program, whereas the replacement can later exit unsuccessfully. Record the actual child status and preserve enough context to identify the attempted command, without logging confidential environment values. That gives launch supervision a verifiable outcome rather than an inferred success.

Sources

fork(2), private memory, shared mappings, inherited descriptions, and thread restrictions, checked 2026-09-12.
execve(2), image replacement, explicit environment, preserved PID, and close-on-exec, checked 2026-09-12.
waitpid(2), collecting child status and reaping an exited child, checked 2026-09-12.

Setuid runs as the file owner

Sat, 02 Oct 2027 00:00:00 +0000

A setuid binary runs with the identity of its owner rather than of whoever started it. The kernel swaps credentials at exec time, which is how passwd rewrites /etc/shadow without an ordinary user holding write permission on it. The bit belongs to the file, and it grants the owner’s entire identity, not one narrow ability.

That is an odd trade to make on purpose: a whole program is trusted with an entire account, so any bug inside it is a privilege bug. Capabilities, user namespaces and socket activation now cover most of what setuid was for.

Identity follows the file

execve(2) states the rule directly: if the set-user-ID bit is set on the program file, the effective user ID of the calling process becomes that of the file’s owner. The real UID and real GID are unchanged, and the new effective UID is copied into the saved set-user-ID. Real, effective and saved are the whole mechanism.

The bits are visible before anything runs: stat reports /usr/bin/sudo as mode 4511 (-r-s--x--x) and /usr/bin/su as 4755, both owned by uid 0, while /usr/bin/passwd is plain 0755. Which binaries carry the bit is a per-host fact, so read it rather than assume it.

Four ways the bit does nothing

execve(2) lists the conditions under which the transition is skipped: no_new_privs is set on the calling thread, the filesystem is mounted nosuid, or the process is being ptraced. For a nosuid mount the man page warns that behaviour varies by kernel version — some refuse the exec with EPERM, others ignore the bits and run the file anyway. File capabilities are ignored under those same conditions.

Scripts are the fourth case: “Linux (like most other modern UNIX systems) ignores the set-user-ID and set-group-ID bits on scripts.” A shell script can list as -rwsr-xr-x forever and elevate nothing, because #! hands control to the interpreter and only the interpreter’s own bits count. On the macOS host here, chmod 4755 as owner returned EPERM while chmod 1755 set the sticky bit.

Dropping privilege is not symmetric

The setuid(2) syscall is not the file bit. A privileged caller holding CAP_SETUID moves real, effective and saved together, so a set-user-ID-root program calling setuid(1000) ends with all three at 1000 — and regaining root then fails with EPERM, because 0 is no longer the real UID or the saved set-user-ID. The man page says such a program must use seteuid(2) instead.

seteuid(2) moves the effective UID alone, and an unprivileged caller may only target the real UID, the effective UID, or the saved set-user-ID. The saved UID is therefore a deliberate escape hatch: the same process can drop to 1000 with seteuid and climb back to 0 afterwards. Permanent drops need setuid(2); reversible ones need seteuid(2).

The narrower replacements

Capabilities split one identity into about forty named powers. setcap cap_net_bind_service=+ep /usr/bin/webserver lets that binary bind port 80 with no setuid bit and no ability to read /etc/shadow. capabilities(7) notes that exec of a set-user-ID-root program still results in the full capability set being treated as enabled — exactly the blast radius the narrower grant avoids. User namespaces map uid 0 to the caller inside a namespace and grant nothing outside it, and socket activation keeps the listener owned by PID 1 while the service runs unprivileged.

When it breaks

The classic failure is the setuid script: someone rewrites a helper in shell or Python, keeps the mode, and the tool silently loses the privilege it needed. The listing still shows s, an audit still flags it, and nothing elevates. Prove elevation with id -u printed from inside the running program, never from the mode string.

The dangerous failure is a shell escape inside a setuid binary. The C library scrubs LD_PRELOAD and its relatives, but application-level inputs stay yours to sanitise: an unscrubbed PATH turns system("thumbnail ...") into an attacker’s binary, and a user-controlled config path redirects file reads. Because the whole process carries the owner’s identity, one such bug is a root hole. Keep the list short and give each binary an owner that owns nothing else of value.

Check it quickly

Run these on the host you are auditing:

find / -xdev -perm -4000 -type f -printf '%m %u %p\n' 2>/dev/null
findmnt -no TARGET,OPTIONS / /home /tmp /var | grep -o nosuid
getcap -r /usr/bin /usr/sbin 2>/dev/null
grep -E 'NoNewPrivs|CapEff' /proc/self/status
stat -c '%A %a %U %n' /usr/bin/passwd

The first command should return a short list; on a stock Debian host it is roughly a dozen paths, and every one should be a program you would trust with its owner’s whole identity. A nosuid on a filesystem holding a setuid binary explains a privilege that vanished, and anything in the getcap output has already migrated off setuid.

Sources

man7: execve, the set-user-ID transition at exec and the four conditions that skip it, checked 2026-09-16.
man7: setuid, CAP_SETUID, the saved set-user-ID, and why seteuid is the reversible path, checked 2026-09-16.
man7: capabilities, narrow capability grants as the replacement for a setuid-root binary, checked 2026-09-16.

The sticky bit restricts removal in a shared directory

Fri, 01 Oct 2027 00:00:00 +0000

The sticky bit on a Linux directory adds an ownership restriction to removing or renaming entries. It is commonly combined with broad directory write and search permissions so several users can create temporary files without freely removing one another’s entries. The directory permission check and the sticky ownership check both matter; sticky is an additional condition, not a replacement for normal access control.

When it breaks

Without sticky, a user with the necessary write and search permissions on a directory can often remove another user’s file even when that file’s mode grants no access. Unlink changes the directory’s name-to-object relationship. It does not require permission to read or overwrite the file contents, so making the file itself mode 000 does not protect its pathname in a broadly writable directory.

With sticky enabled, ordinary users must additionally own the entry or the containing directory to perform the relevant removal. A privileged process with the appropriate CAP_FOWNER capability can bypass that ownership restriction. Sticky therefore limits peers under a specific permissions model; it does not protect against every administrator or the directory’s own owner.

Sticky does not stop permitted content access. Another user may still read or overwrite a mode-0666 file while being unable to unlink it. The mode also does not automatically propagate to newly created subdirectories. Confusing pathname protection, content privacy, and inheritance can leave a shared tree with materially different behavior from its apparently protected top level.

Check it quickly

Save this as sticky.py and run python3 sticky.py as root in a disposable Linux container. The controller needs permission to assign three numeric identities and launch unprivileged children; it creates no accounts. The recorded environment used Linux 6.12.76 and Python 3.11.2. All paths are inside a temporary fixture directory:

from pathlib import Path
from tempfile import TemporaryDirectory
import errno,os,stat,subprocess,sys

directory_owner=61000;file_owner=61001;other_user=61002
probe_code=r'''
import os,sys
path=sys.argv[1];operation=sys.argv[2]
try:
 if operation=='unlink':os.unlink(path)
 elif operation=='rename':os.rename(path,path+'-renamed')
 elif operation=='write':
 fd=os.open(path,os.O_WRONLY|os.O_TRUNC)
 try:os.write(fd,b'other-user-write')
 finally:os.close(fd)
 elif operation=='read':
 fd=os.open(path,os.O_RDONLY);os.close(fd)
 elif operation=='mkdir':os.mkdir(path,0o777)
 else:raise AssertionError('unknown operation')
except OSError as error:print(error.errno)
else:print(0)
'''
def probe(uid,path,operation):
 result=subprocess.run([sys.executable,'-c',probe_code,str(path),operation],user=uid,group=uid,extra_groups=[],capture_output=True,text=True,timeout=5)
 assert result.returncode==0,result.stderr
 return int(result.stdout)
with TemporaryDirectory(prefix='xbcatm-sticky-') as directory:
 root=Path(directory);root.chmod(0o755);shared=root/'shared';shared.mkdir();shared.chmod(0o777)
 os.chown(shared,directory_owner,directory_owner);victim=shared/'victim'
 def make_victim(mode):
 victim.write_bytes(b'fixture');os.chown(victim,file_owner,file_owner);victim.chmod(mode)
 make_victim(0o000)
 assert probe(other_user,victim,'unlink')==0
 print('without sticky, directory write/search allows removing another owner’s mode-000 file')
 shared.chmod(0o1777);make_victim(0o666)
 assert probe(other_user,victim,'unlink')==errno.EPERM
 assert probe(other_user,victim,'rename')==errno.EPERM
 assert probe(other_user,victim,'write')==0 and victim.read_bytes()==b'other-user-write'
 victim.chmod(0o600);assert probe(other_user,victim,'read')==errno.EACCES
 print('sticky blocks outsider unlink/rename, but file permissions still govern reading and writing')
 assert probe(file_owner,victim,'unlink')==0
 make_victim(0o600);assert probe(directory_owner,victim,'unlink')==0
 print('both the entry owner and the directory owner may remove it')
 child=shared/'child';assert probe(other_user,child,'mkdir')==0
 assert not child.stat().st_mode&stat.S_ISVTX
 shared.chmod(0o1766);assert stat.filemode(shared.stat().st_mode).endswith('T')
 assert probe(other_user,shared/'blocked','mkdir')==errno.EACCES
 shared.chmod(0o1777);assert stat.filemode(shared.stat().st_mode).endswith('t')
 print('sticky is not inherited by a new subdirectory; uppercase T means other-execute is absent')

The first case uses a 0777 directory owned by one identity and a mode-000 file owned by another. A third user successfully unlinks that file. This demonstrates that directory write/search permissions can allow removal independently of the file’s content permissions. Root sets up the fixture, but the tested removal occurs under the third, unprivileged identity.

After setting the directory to 1777, the third user’s unlink and rename attempts fail with EPERM. The same user successfully overwrites the file when its mode is 0666. The write opens an existing file without O_CREAT, avoiding a different Linux protected-regular-file policy that could otherwise obscure which permission rule the exercise is measuring.

Changing the file to 0600 causes the third user’s read attempt to fail with EACCES. Sticky remained enabled throughout, so it is the file-permission change that altered content access. The two error classes arise from different checks and should not be collapsed into a claim that sticky makes each temporary file private.

The file’s owner successfully removes its entry, and a newly recreated file can also be removed by the directory owner. Both outcomes are checked using ordinary child identities. These cases demonstrate the two ownership exceptions directly instead of relying on root, whose capabilities would make the result ambiguous.

Next, the third user creates a subdirectory. It has no sticky bit automatically inherited from the parent. Changing the parent to 1766 produces an uppercase T in its permission display, and another creation attempt fails because directory search permission is missing. Restoring 1777 produces lowercase t, which combines sticky with the other-execute bit.

Design shared storage deliberately

Use sticky when the intended policy allows shared creation but restricts peer removal. Set file permissions or ACLs separately for the intended readers and writers. If each user’s data must remain confidential, verify access under another identity; the presence of a sticky directory alone supplies no evidence that its files are unreadable.

Treat directory ownership as authority over its entries. A directory owner who can remove files may disrupt other users even while unable to read their contents. In a hierarchy, inspect the permissions of the actual parent containing each entry, because a protected ancestor does not automatically apply its sticky behavior to descendants.

For temporary-file creation, use an API that creates a fresh, suitably restricted object without trusting a predictable preexisting name. Sticky helps with a particular shared-directory removal problem, but it does not solve symbolic-link substitution or every unsafe open pattern. Verify creation, subsequent content access, and cleanup under the actual identities and filesystem used by the application.

Sources

inode(7), sticky directories, file ownership, and displayed permission bits, checked 2026-09-12.
unlink(2), directory write/search permissions and sticky removal restrictions, checked 2026-09-12.
capabilities(7), CAP_FOWNER exceptions to sticky-directory ownership checks, checked 2026-09-12.
rename(2), rename permission checks in sticky directories, checked 2026-09-12.

Environment variables are a per-process thing

Sat, 25 Sep 2027 00:00:00 +0000

An environment is a collection of name-value strings supplied to a program when it starts. A child normally receives the environment selected by its parent; changing that child’s copy does not rewrite the parent’s or a sibling’s copy. This is a useful configuration boundary, but it is not a promise that the command has no side effects. The program can still modify shared files, databases, or other resources using those settings.

Export selects values for another program

A shell variable and an environment entry are related but distinct. Assigning DEMO_SETTING=local creates or updates a shell parameter. export DEMO_SETTING marks it for inclusion when the shell launches external commands. A new Bash process normally cannot see an unexported shell variable, even though the parent can print it. Merely placing an assignment in a configuration file does not load that file into every shell.

A value inherited through the environment is already marked for export by Bash. Changing its value retains that attribute, so a later grandchild can inherit the changed value without another export statement. unset removes the variable; Bash’s export -n removes its export attribute while retaining the shell value. The relevant question is which environment the immediate launcher supplied, not whether every ancestor repeated an export command.

A subshell is not a freshly executed shell

Parenthesized command groups and command substitution run in a subshell environment copied from the current shell. They can see ordinary shell variables that were never exported. Their assignments still do not flow back into the outer shell. This differs from executing a separate shell program, which receives exported entries rather than a complete copy of the parent’s shell parameter table.

Sourcing a file with . runs its commands in the current shell. Its assignments can therefore change the caller directly; it is not an exception to process isolation because no separate process is returning those changes. A sourced file is executable shell code, not passive configuration. Only source a file whose contents you intend to execute in that context.

Per-command settings have a defined lifetime

A prefix such as DEMO_SETTING=once external-command supplies a value for that invocation without permanently changing the surrounding shell’s value. This statement concerns an external command; assignment behavior around special builtins and functions has additional shell-specific rules. Do not generalize it into a claim that every command-looking construct creates an isolated process.

At the operating system interface, execve takes an explicit environment array and replaces the current process image. It does not itself create a new process. A launcher can preserve, filter, or replace the environment, which explains why privilege boundaries, service managers, and container configuration may produce values different from an interactive terminal. Process ancestry alone is insufficient to establish the actual settings delivered to an application.

When it breaks

Editing a terminal’s environment does not update an already running service. Likewise, changing an environment file has no effect until the relevant launcher rereads it and starts the process, unless the application provides its own reload mechanism. Check the configuration path used by that service and verify the resulting behavior in the process that actually handles requests.

Treat missing and empty values separately, because applications may assign them different meanings. Environment strings cannot contain an embedded NUL, and platforms limit the combined argument and environment size, so they are unsuitable for arbitrary binary payloads or unbounded documents. Inspect only the specific nonsecret keys needed for diagnosis; dumping an entire environment can expose credentials inherited by unrelated tools.

Check it quickly

Run this Bash example. It prints only a dedicated demonstration variable, uses no external configuration file, and leaves the surrounding terminal unchanged. The small here-document passed to . contains the complete trusted configuration used for the sourcing test.

BASH_ENV= bash --noprofile --norc <<'BASH'
unset DEMO_SETTING
DEMO_SETTING=local
printf 'shell=%s\n' "$DEMO_SETTING"
bash --noprofile --norc -c 'printf "unexported-child=%s\n" "${DEMO_SETTING-unset}"'
(printf 'subshell=%s\n' "$DEMO_SETTING")
export DEMO_SETTING
bash --noprofile --norc <<'CHILD'
printf 'exported-child=%s\n' "$DEMO_SETTING"
DEMO_SETTING=child
bash --noprofile --norc -c 'printf "grandchild=%s\n" "$DEMO_SETTING"'
CHILD
printf 'parent-after-child=%s\n' "$DEMO_SETTING"
DEMO_SETTING=once bash --noprofile --norc -c 'printf "one-command=%s\n" "$DEMO_SETTING"'
printf 'parent-after-command=%s\n' "$DEMO_SETTING"
. /dev/stdin <<'CONFIG'
DEMO_SETTING=sourced
CONFIG
printf 'parent-after-source=%s\n' "$DEMO_SETTING"
BASH

The unexported child prints unset, while the parenthesized subshell sees local. After export, the new child sees local, changes its own copy, and passes child to its grandchild. The outer shell remains local. The one-command override prints once, and the following parent check is still local.

Finally, sourcing changes that outer child shell’s value to sourced. Together these observations distinguish shell parameters, exported entries, subshell copies, per-command overrides, and execution in the current shell. They establish variable scope, not the absence of application side effects or the correctness of a service’s deployment configuration.

Sources

Bash execution environment, exported entries versus the state copied into subshells, checked 2026-09-07.
Bash environment, export attributes and per-command assignments, checked 2026-09-07.
POSIX exec functions, process-image replacement and explicit environment arrays, checked 2026-09-07.

A bare command the shell cannot find

Thu, 20 May 2027 00:00:00 +0000

“Command not found” identifies a lookup failure, but lookup does not always mean walking every directory in PATH. In ordinary Bash, a command word without a slash can resolve to a function, builtin, cached executable path, or a newly searched executable. An alias may already have changed the command while the input was read. Inspect the shell that actually reported the failure before reinstalling a tool that may already exist.

Know which lookup layer answered

After expansion, Bash checks functions and then builtins before searching for an external command. type name describes the current resolution, and type -a name exposes additional definitions or candidates. command -v name is useful for testing availability in that shell, but its result need not be a filesystem path. It can identify a function or builtin, so do not blindly treat the output as a file to copy.

For external commands, directories in PATH are searched in order. An earlier executable can shadow a newer version later in the list. Bash also caches successful paths; hash -t name inspects a cached entry and hash -r clears the table. A changed file at an already cached location and a different path choice are separate issues. Starting a new terminal can hide this distinction by resetting shell state.

A slash selects a path directly

Any slash in the command word bypasses the external PATH search. ./tool addresses a file in the current directory; /opt/example/bin/tool addresses an absolute location. A bare tool does not automatically search the current directory. Empty or relative entries in PATH can make it do so, which allows the current working directory to change the selected executable.

Use explicit, trusted directories when assembling a service’s environment. A working interactive terminal does not establish that cron, a service manager, or another user has the same path or startup files. Compare their actual environments and working directories. Changing a login configuration file will not update an already running process; apply the intended environment to the process that launches the command.

When it breaks

Bash normally reports 127 when a command cannot be found and 126 when a located command cannot be executed. These values guide investigation, but they are not a complete diagnosis: an existing script can fail because the interpreter named by its shebang is missing. Read the diagnostic and inspect the selected file’s permissions and interpreter instead of repeatedly appending directories.

Do not “fix” lookup by adding writable project directories ahead of trusted tools or by using administrator privileges indiscriminately. First establish whether a function, an alias, or a cached location is responsible. command name bypasses a same-named function in Bash; an explicit executable path bypasses path selection. Neither makes the selected program itself trustworthy or repairs its missing dependencies.

Check it quickly

This Bash example creates two disposable executables with the same name. Its temporary directories are the only additions to PATH, and all changes stay inside the child shell. Cleanup names each file explicitly. No installed executable is modified.

Setup uses errexit to stop if fixture creation fails, then disables it before deliberately testing failures. On Bash 3.2.57, used to check this example, keeping that option enabled makes these execution-failure branches report 1 instead of the ordinary 126 and 127. Recording the shell version and options is therefore part of interpreting an unexpected status, not merely a reproducibility detail.

bash --noprofile --norc <<'BASH'
set -eu
demo_dir=$(mktemp -d)
trap 'rm -f "$demo_dir/one/demo-tool" "$demo_dir/two/demo-tool" "$demo_dir/blocked"; rmdir "$demo_dir/one" "$demo_dir/two" "$demo_dir"' EXIT
mkdir "$demo_dir/one" "$demo_dir/two"
printf '#!/bin/sh\nprintf "one\\n"\n' > "$demo_dir/one/demo-tool"
printf '#!/bin/sh\nprintf "two\\n"\n' > "$demo_dir/two/demo-tool"
chmod +x "$demo_dir/one/demo-tool" "$demo_dir/two/demo-tool"
PATH="$demo_dir/one:$demo_dir/two:$PATH"
demo-tool
hash -p "$demo_dir/two/demo-tool" demo-tool
demo-tool
hash -r
demo-tool
demo-tool() { printf 'function\n'; }
demo-tool
command demo-tool
unset -f demo-tool
"$demo_dir/two/demo-tool"
: > "$demo_dir/blocked"
chmod 600 "$demo_dir/blocked"
set +e
if "$demo_dir/blocked" 2>/dev/null; then
 printf 'unexpected execution\n'
else
 printf 'not-executable=%s\n' "$?"
fi
if "$demo_dir/missing" 2>/dev/null; then
 printf 'unexpected file\n'
else
 printf 'missing=%s\n' "$?"
fi
BASH

The first three lines should be one, two, one: normal search, an explicitly installed cache entry, and a fresh search after clearing it. Next come function, one, and two, distinguishing function dispatch, function bypass, and an explicit path. Finally, the nonexecutable file returns 126 and the absent path returns 127.

This proves selection behavior with controlled files. It does not reproduce every operating system loader error or service environment. For a real incident, record the resolved command and its version in the failing environment, then verify the corrected invocation there. A successful lookup in an unrelated terminal is insufficient evidence that the original failure is repaired.

Sources

POSIX shell command search, command search rules and the not-found exit status, checked 2026-06-01.
Bash reference manual: command search and execution, function and builtin precedence plus cached executable paths, checked 2026-09-07.

Graceful shutdown stops work, not life

Sat, 19 Dec 2026 00:00:00 +0000

Graceful shutdown is a transition in how a service accepts and finishes work. Receiving a signal is only the trigger. The useful guarantee is narrower than “nothing is lost”: stop admitting new business work, account for what was already accepted, allow bounded completion, and report honestly when that bound expires.

A signal begins coordination

SIGTERM can be handled so an application starts its shutdown sequence. SIGKILL cannot be caught, blocked, or ignored. Verify which process receives the stop signal: a handler inside a worker does not help if a wrapper receives the signal and never forwards it.

Keep the signal handler small. Python executes its handlers in the main interpreter thread and warns against synchronization locks inside them. The fixture only sets a flag; ordinary control flow changes readiness and admission under a lock. It does not perform network I/O or wait for workers inside the handler.

Readiness should be withdrawn early enough for routing changes to begin. Admission still needs its own local gate, because existing connections and delayed routing updates can deliver requests afterward. Changing the gate and registering accepted work must use a consistent boundary.

The deadline covers a lifecycle

Track work until its real completion point. A caller timeout is not proof that a worker stopped, and a sent response is not proof that a business transaction is durable. Select the completion boundary appropriate to the operation instead of treating every closed socket as success.

Use a monotonic deadline for local elapsed time. Leave room for cleanup within the supervisor’s actual allowance. When the deadline expires, remaining work needs an explicit cancellation, abandonment, or durable handoff policy. Merely breaking out of a wait loop does not cancel a running thread.

Successful draining and incomplete shutdown should remain distinguishable in records. A particular nonzero exit code is a service contract, not a universal rule that guarantees correct recovery. A client losing a response may still need idempotency or reconciliation before retrying.

Coordinate with the supervisor

In systemd, TimeoutStopSec governs configured stop-command and service-stop waits. KillMode determines the processes targeted, not how much drain time they receive. With mixed mode, the initial termination signal targets the main process and final escalation targets remaining control-group processes; early main-process exit can also trigger that escalation. Review stop hooks and notification-based extensions in the real unit.

Kubernetes begins the termination grace period before running a configured preStop hook. The hook does not receive an independent extra drain budget. EndpointSlice termination updates and local shutdown proceed alongside each other; they are not an instantaneous global barrier after which no request can arrive.

The normal stop signal can depend on image or runtime configuration. Test the deployed process tree and stop contract rather than assuming the behavior of a locally executed program also proves container, sidecar, and proxy coordination.

When it breaks

A shutdown flag that only hides readiness can leave a keep-alive connection admitting new work forever. Conversely, closing every active socket immediately can discard requests that had enough time to finish. Stopping admission and finishing accepted work are separate decisions.

An unbounded join can exceed the supervisor’s allowance. Daemon threads allow process exit without waiting, but that is abandonment, not safe cancellation or transaction rollback. Libraries have their own shutdown contracts: Python’s socketserver shutdown must be called from a different thread than serve_forever or it deadlocks.

Observe accepted, completed, refused, and unfinished work, together with shutdown duration and exit outcome. A clean-looking exit without those distinctions can hide dropped work. The goal is a verified lifecycle, not just a registered signal handler.

Check it quickly

Save this block as shutdown_check.py and run python3 shutdown_check.py with Python 3.10 or newer on a Unix-like system. It ran on macOS with Python 3.14.4. Each case creates its own child process and loopback HTTP server; SIGTERM targets only that child. The POST release route is test coordination, not a production administration API.

from concurrent.futures import ThreadPoolExecutor
from http.client import HTTPConnection, RemoteDisconnected
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import selectors
import signal
import subprocess
import sys
from threading import Condition, Event, Thread
import time


def child(grace):
 state = {"draining": False, "active": 0}
 condition, release = Condition(), Event()
 stop_requested = False

 def stop(signum, frame):
 nonlocal stop_requested
 stop_requested = True # No lock, I/O, or shutdown call in the signal handler.

 class Handler(BaseHTTPRequestHandler):
 def log_message(self, *args):
 pass

 def reply(self, status, value):
 body = json.dumps(value).encode()
 self.send_response(status)
 self.send_header("Content-Type", "application/json")
 self.send_header("Content-Length", str(len(body)))
 self.send_header("Cache-Control", "no-store")
 self.end_headers()
 self.wfile.write(body)

 def do_GET(self):
 if self.path == "/state":
 with condition:
 snapshot = dict(state)
 self.reply(200, snapshot)
 elif self.path == "/readyz":
 with condition:
 ready = not state["draining"]
 self.reply(200 if ready else 503, {"ready": ready})
 elif self.path == "/work":
 with condition:
 admitted = not state["draining"]
 if admitted:
 state["active"] += 1
 if not admitted:
 self.reply(503, {"error": "draining"})
 return
 try:
 if release.wait(timeout=5):
 self.reply(200, {"done": True})
 else:
 self.reply(503, {"error": "fixture-timeout"})
 finally:
 with condition:
 state["active"] -= 1
 condition.notify_all()
 else:
 self.reply(404, {"error": "not-found"})

 def do_POST(self):
 if self.path == "/release": # Loopback-only test coordination, not workload admission.
 self.reply(200, {"released": True})
 release.set() # Finish the control reply before work can complete and the child exits.
 else:
 self.reply(404, {"error": "not-found"})

 signal.signal(signal.SIGTERM, stop)
 server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
 server.daemon_threads = True # Explicitly permits abandonment in the deadline case.
 serving = Thread(target=server.serve_forever, kwargs={"poll_interval": 0.01})
 serving.start()
 print(json.dumps({"port": server.server_port}), flush=True)
 while not stop_requested:
 time.sleep(0.005)
 deadline = time.monotonic() + grace
 with condition:
 state["draining"] = True
 while state["active"] and time.monotonic() < deadline:
 condition.wait(timeout=max(0, deadline - time.monotonic()))
 unfinished = state["active"]
 server.shutdown() # Called outside the thread running serve_forever.
 serving.join(timeout=2)
 server.server_close() # Does not wait for daemon request threads.
 assert not serving.is_alive()
 print(json.dumps({"unfinished": unfinished, "deadline_expired": unfinished > 0}), flush=True)
 return 3 if unfinished else 0 # Fixture policy, not a universal exit-code convention.


def request(port, route, method="GET"):
 connection = HTTPConnection("127.0.0.1", port, timeout=3)
 try:
 connection.request(method, route)
 response = connection.getresponse()
 return response.status, json.loads(response.read())
 finally:
 connection.close()


def wait_state(port, predicate):
 deadline = time.monotonic() + 3
 while time.monotonic() < deadline:
 status, state = request(port, "/state")
 assert status == 200
 if predicate(state):
 return state
 time.sleep(0.01)
 raise AssertionError("child did not reach the expected state")


def exercise(finish):
 process = subprocess.Popen(
 [sys.executable, __file__, "child", "3" if finish else "1"],
 stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
 )
 try:
 with selectors.DefaultSelector() as selector:
 selector.register(process.stdout, selectors.EVENT_READ)
 assert selector.select(timeout=5), "child did not announce its local port"
 port = json.loads(process.stdout.readline())["port"]
 assert request(port, "/readyz") == (200, {"ready": True})
 with ThreadPoolExecutor(max_workers=1) as pool:
 active = pool.submit(request, port, "/work")
 wait_state(port, lambda state: state["active"] == 1)
 process.send_signal(signal.SIGTERM)
 wait_state(port, lambda state: state["draining"])
 assert request(port, "/readyz") == (503, {"ready": False})
 assert request(port, "/work") == (503, {"error": "draining"})
 assert request(port, "/state")[1]["active"] == 1
 if finish:
 assert request(port, "/release", "POST")[0] == 200
 assert active.result(timeout=4) == (200, {"done": True})
 output, errors = process.communicate(timeout=5)
 assert errors == ""
 record = json.loads(output)
 assert process.returncode == (0 if finish else 3)
 assert record == {"unfinished": 0 if finish else 1, "deadline_expired": not finish}
 if not finish:
 try:
 active.result(timeout=4)
 except (OSError, RemoteDisconnected):
 pass
 else:
 raise AssertionError("abandoned work must not report a successful reply")
 print("completed drain" if finish else "expired drain", record, "exit", process.returncode)
 finally:
 if process.poll() is None:
 process.kill() # Safety cleanup targets only this fixture's child.
 process.wait(timeout=5)
 process.stdout.close()
 process.stderr.close()


if __name__ == "__main__":
 if sys.argv[1:2] == ["child"]:
 raise SystemExit(child(float(sys.argv[2])))
 exercise(True)
 exercise(False)

Both cases admit one request before SIGTERM. After the drain flag becomes visible, readiness returns 503, a new business request receives 503, and the admitted request remains counted.

Releasing that request lets it return 200 before the child exits zero with no unfinished work. Without release, the one-second drain allowance expires; the child reports one unfinished request, exits three, and the waiting client receives no successful reply.

The latter case deliberately abandons a daemon worker. It does not prove cooperative cancellation, durability, SIGKILL cleanup, or safe replay. No systemd unit, Kubernetes workload, real transaction, or production traffic was exercised.

Sources

Linux signal(7), termination signals and the uncatchable SIGKILL boundary, checked 2026-09-09.
Python signals, main-thread handlers and the warning against locks in handlers, checked 2026-09-09.
Python socketserver, shutdown-thread requirements and daemon request-thread behavior, checked 2026-09-09.
systemd.service, TimeoutStopSec stop waits and notification extensions, checked 2026-09-09.
systemd.kill, KillMode process scope and termination escalation, checked 2026-09-09.
Kubernetes: Pod lifecycle, preStop, termination grace, stop signals, and EndpointSlice transitions, checked 2026-09-09.

Give durable state and scratch space different lifecycles

Thu, 03 Dec 2026 00:00:00 +0000

Durable application state and disposable scratch files need different retention, capacity, and recovery policies. Separating their paths makes those policies easier to express, but directories alone do not create persistence or isolate disk capacity. The storage backing each path determines what survives process restart, container replacement, host failure, or cleanup. A reliable layout connects each class of data to an explicit lifecycle.

When it breaks

A cleanup job that treats one mixed directory as disposable can remove committed records alongside temporary downloads. The reverse problem is backing up large rebuildable caches while missing the small database that actually matters. File extensions and age are weak substitutes for knowing which application owns a path and whether its contents can be reconstructed from an authoritative source.

Container restart and replacement also differ. Stopping and starting the same container normally retains its writable filesystem layer. Removing that container discards the layer, while an explicitly retained named volume can survive. A tmpfs mount has another lifecycle and loses its contents when the container stops. Calling all three locations temporary hides distinctions that affect real recovery procedures.

Two separate directories on the same full filesystem still compete for that filesystem’s free space. Moving scratch from /data/tmp to /scratch without changing storage policy does not automatically protect database writes. Capacity isolation requires an appropriate mount, quota, or other enforceable budget, together with monitoring that distinguishes scratch growth from durable-state growth.

Check it quickly

Run this script using host Python and Docker after obtaining the pinned image. The recorded environment used Docker 29.3.1 and Linux arm64 Python 3.11.16 with its SQLite module. It creates one private named volume and an eight-mebibyte tmpfs scratch mount. Only its own containers and volume are removed; the small scratch exhaustion stays inside the experiment:

import subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def execute(container,code):return docker('exec',container,'python','-c',code).stdout.strip()

name='xbcatm-state-'+uuid.uuid4().hex[:10];volume=name+'-data';containers=[name+'-one',name+'-replacement']
try:
 docker('volume','create',volume)
 def start(container):
 docker('run','-d','--name',container,'--network','none','--mount','type=volume,src='+volume+',dst=/state','--tmpfs','/scratch:rw,size=8m',BASE,'python','-c','import time;time.sleep(180)')
 start(containers[0])
 execute(containers[0],'from pathlib import Path;import sqlite3;c=sqlite3.connect("/state/data.sqlite");c.execute("create table records(value text)");c.execute("insert into records values (?)",("committed",));c.commit();c.close();Path("/ephemeral").write_text("container-layer");Path("/scratch/crumb").write_text("scratch")')
 docker('stop','--time','1',containers[0]);docker('start',containers[0])
 execute(containers[0],'from pathlib import Path;import sqlite3;assert Path("/ephemeral").exists();assert not Path("/scratch/crumb").exists();c=sqlite3.connect("/state/data.sqlite");assert c.execute("select value from records").fetchall()==[("committed",)];c.close()')
 print('stop/start retains the container writable layer and volume data, while the tmpfs scratch mount loses its file')
 docker('rm','-f',containers[0]);start(containers[1])
 execute(containers[1],'from pathlib import Path;import sqlite3;assert not Path("/ephemeral").exists();c=sqlite3.connect("/state/data.sqlite");assert c.execute("select value from records").fetchall()==[("committed",)];c.close()')
 print('replacement loses the old writable layer but retains the named-volume database')
 execute(containers[1],'''import errno,os,sqlite3
fd=os.open('/scratch/fill',os.O_CREAT|os.O_WRONLY,0o600)
try:
 try:
 while True:os.write(fd,b'x'*1048576)
 except OSError as error:assert error.errno==errno.ENOSPC
finally:os.close(fd)
source=sqlite3.connect('/state/data.sqlite');source.execute('insert into records values (?)',('after-scratch-full',));source.commit()
backup=sqlite3.connect('/state/check-copy.sqlite');source.backup(backup);backup.close();source.close()
restored=sqlite3.connect('/state/check-copy.sqlite');assert restored.execute('select value from records order by rowid').fetchall()==[('committed',),('after-scratch-full',)];restored.close()
''')
 print('bounded scratch reaches ENOSPC while state still commits; a SQLite backup copy reopens with both committed records')
finally:
 docker('rm','-f',*containers,check=False);docker('volume','rm',volume,check=False)

The first container creates a SQLite database under /state, commits one record, writes /ephemeral in its ordinary writable layer, and puts a crumb under /scratch. The three writes have distinct storage locations even though the process accesses all of them through ordinary filesystem paths. SQLite commit is used for the durable application record rather than relying on an unclosed output buffer.

After stopping and starting that same container, the database record and /ephemeral still exist, but the scratch crumb is absent. The container command does not recreate these files during startup, so the observation is about storage lifetime rather than initialization hiding the loss. This distinguishes a process restart within a retained container from destruction of its writable layer.

The original container is then removed, and a replacement mounts the same named volume. Its database still contains committed, while /ephemeral is absent. The replacement gets a new writable layer but reconnects to the retained volume. This demonstrates the volume’s independence from one container’s lifecycle, without claiming protection against losing the Docker host or deleting the volume itself.

Next, the replacement fills its bounded scratch mount until a write reports ENOSPC. It can still insert and commit after-scratch-full into the database on the volume. The successful commit demonstrates the particular capacity boundary configured here. It does not imply unlimited durable storage or protection from exhausting some other shared resource, such as total host memory.

Finally, SQLite’s backup API creates a database copy that is closed, reopened, and checked for both committed records in order. This uses the database’s copying interface instead of assuming that copying one live database filename captures every required file consistently. The copy remains on the same experimental volume, so it validates copy readability and content, not an independent disaster-recovery backup.

Make policy follow the data contract

Use a stable state directory for authoritative application data and a separately managed scratch location for reconstructable work. The Filesystem Hierarchy Standard distinguishes application state under /var/lib; container paths can differ while preserving that semantic distinction. Name the directories in application configuration and document which component owns creation, permissions, cleanup, and migration.

Set scratch budgets deliberately and handle exhaustion without deleting unrelated state. A cache is disposable only if its source and reconstruction procedure are actually available. Some apparently temporary files are required for an in-progress transaction or recovery, so cleanup needs the application’s contract rather than an indiscriminate age rule.

Keep independent backups with retention appropriate to the data, and test restoring into a fresh environment. A volume surviving replacement is persistence evidence, not backup evidence; a same-volume copy cannot survive loss of that volume. Monitor the actual backing stores, verify restore contents, and make destructive volume removal an explicit lifecycle decision.

Sources

Docker volumes, data lifecycle independent of container writable layers, checked 2026-09-12.
Docker tmpfs mounts, temporary in-memory mounts and size limits, checked 2026-09-12.
Filesystem Hierarchy Standard: /var/lib, variable state information retained for applications, checked 2026-09-12.
SQLite online backup API, consistent database copying through the database API, checked 2026-09-12.

Named config files favor clarity

Sat, 21 Nov 2026 00:00:00 +0000

Configuration that is easy to find, read, and diff is more than a preference; it is how a change stays reviewable. Naming a file with a clear convention is the first step to environment-specific, single-source config.

A file is a record of intent

A named, versioned config file is a record of intent: the deployed values become explicit, reviewable, and reproducible just like code. When the file lives with the project, a change gets history, a blame, and a pull request, so anyone can answer ‘what actually shipped and who changed it’. A clear naming convention (app.conf for application knobs, app.env for environment shape) makes each file’s purpose obvious at a glance, and keeps the settings from being scattered into ad-hoc flags nobody can find.

Env and files solve different problems

Environment variables and files solve different problems. Variables are ideal for deploying the same build into many places without editing an image — you vary the value at run time per environment. Files are better for rich, structured settings a variable string would mangle. Most systems combine both: files keep the structure, the environment injects the values that differ per deploy. The rule that keeps it honest is that the source of truth stays explicit, and the boundary between structure and injectable values is drawn deliberately rather than by accident.

Secrets still move outside

Putting a config file in a repo is not a license to commit secrets into it. Anything that is a credential should be injected or referenced from a secrets store at run time, never baked into the file you commit and diff. The clean split is shape in files, secrets in a store: the committed config stays safe to show and the real values are encrypted where they live. If a value would be embarrassing or dangerous to publish, it does not belong in the committed file — that rule is exactly what tells you which are secrets.

When it breaks

Config names drift and the loader picks the wrong file. A deploy writes app-prod.json while the program opens app.prod.json, so the expected file is missing and the loader quietly falls back to a stale development file. The service starts, talks to the wrong database, and nothing in the diff explains it, because the real mistake is the filename rather than any value. Fail closed instead: resolve one explicit path at startup, log that path, and exit with file-not-found rather than guessing a sibling.

Environment overrides fail silently in the other direction. An empty APP_PORT= or a misspelled APP-PORT either overrides a good file value with an empty string or overrides nothing at all, depending on the loader, and the running value no longer matches any committed file. Because the change leaves no diff, the team re-reads the file and concludes nothing changed. Record the winning source for every key — file, environment, or flag — print it at startup, and reject empty values instead of treating them as real input.

Secrets leak when a convenient file becomes the real store. Someone commits a working .env carrying a production credential to fix staging once, removes it in the next commit, and assumes the problem is gone. It is not: the secret lives on in history, mirrors, and backups, and every clone made in between still carries it. Committed files should hold shape with placeholders such as password = "from-store", real values should arrive only from a secrets store or the runtime environment, and a scanner should block the commit before it lands.

Check it quickly

Save the following as config_precedence_probe.py and run python3 config_precedence_probe.py. It uses only the standard library, writes two temporary JSON files, and never touches your real config. The good sample starts from {"host": "127.0.0.1", "port": 8000, "mode": "dev"}, applies APP_PORT=9001 from the environment, then --mode prod from the command line, and asserts both the merged result and the per-key source map.

"""Config precedence probe: file < environment < CLI (stdlib only)."""
import json
import os
import sys
import tempfile

REQUIRED = ("host", "port", "mode")
ENV_MAP = {"host": "APP_HOST", "port": "APP_PORT", "mode": "APP_MODE"}

def parse_cli(argv):
 """Support --host/--port/--mode and --set key=value; returns dict."""
 out = {}
 i = 0
 while i < len(argv):
 arg = argv[i]
 if arg == "--set" and i + 1 < len(argv):
 i += 1
 kv = argv[i]
 if "=" not in kv:
 raise ValueError(f"bad --set (want key=value): {kv!r}")
 k, v = kv.split("=", 1)
 out[k.strip()] = v
 elif arg in ("--host", "--port", "--mode") and i + 1 < len(argv):
 i += 1
 out[arg[2:]] = argv[i]
 elif arg.startswith("--") and "=" in arg:
 k, v = arg[2:].split("=", 1)
 if k in REQUIRED:
 out[k] = v
 else:
 raise ValueError(f"unknown flag: {arg!r}")
 else:
 raise ValueError(f"unknown argument: {arg!r}")
 i += 1
 return out

def coerce(key, value):
 if key == "port":
 return int(str(value).strip())
 return str(value).strip()

def load_config(path, env, cli_args):
 """Merge file < env < CLI; raise KeyError listing missing fields."""
 with open(path, encoding="utf-8") as fh:
 file_cfg = json.load(fh)
 merged, source = {}, {}
 for key in REQUIRED:
 if key in file_cfg and str(file_cfg[key]).strip() != "":
 merged[key] = coerce(key, file_cfg[key])
 source[key] = "file"
 for key in REQUIRED:
 name = ENV_MAP[key]
 if name in env and str(env[name]).strip() != "":
 merged[key] = coerce(key, env[name])
 source[key] = "env"
 for key, raw in parse_cli(cli_args).items():
 if key in REQUIRED and str(raw).strip() != "":
 merged[key] = coerce(key, raw)
 source[key] = "cli"
 missing = [k for k in REQUIRED if k not in merged]
 if missing:
 raise KeyError(f"missing required fields: {', '.join(missing)}")
 return merged, source

def main():
 with tempfile.TemporaryDirectory() as tmp:
 good = os.path.join(tmp, "app.prod.json")
 with open(good, "w", encoding="utf-8") as fh:
 json.dump({"host": "127.0.0.1", "port": 8000, "mode": "dev"}, fh)
 # GOOD: env beats file, CLI beats env.
 merged, source = load_config(good, {"APP_PORT": "9001"}, ["--mode", "prod"])
 assert merged == {"host": "127.0.0.1", "port": 9001, "mode": "prod"}, merged
 assert source == {"host": "file", "port": "env", "mode": "cli"}, source
 print(f"GOOD {merged} via {source}")
 # BAD 1: missing field is an error, not a silent default.
 bad = os.path.join(tmp, "app.empty.json")
 with open(bad, "w", encoding="utf-8") as fh:
 json.dump({"host": "127.0.0.1"}, fh)
 try:
 load_config(bad, {}, [])
 except KeyError as exc:
 print(f"BAD-MISSING correctly rejected: {exc}")
 else:
 raise AssertionError("empty config should have failed")
 # BAD 2: drifted filename must fail closed, never fall back silently.
 try:
 load_config(os.path.join(tmp, "app-prod.json"), {}, [])
 except FileNotFoundError as exc:
 print(f"BAD-DRIFT correctly failed closed: {exc.filename}")
 else:
 raise AssertionError("drifted filename should have failed")
 print("PASS: file < env < cli, missing fields error, drift fails closed")

if __name__ == "__main__":
 try:
 main()
 except (KeyError, ValueError, FileNotFoundError, AssertionError) as exc:
 print(f"FAIL: {exc}", file=sys.stderr)
 raise SystemExit(1)

The bad samples prove the guardrails. An almost-empty file is rejected with missing required fields for port and mode instead of running on silent defaults, and a drifted filename such as app-prod.json raises file-not-found instead of falling back to another file. Expect three lines starting with good, bad-missing, and bad-drift, followed by a pass line confirming file beats nothing, environment beats file, and flags beat environment. If the good merge differs, the precedence order is wrong; if either bad sample passes silently, the loader is too forgiving.

Sources

12 Factor: config and environment, environment-driven configuration, checked 2026-11-19.
The Twelve-Factor App: build, release, run, separating stages, checked 2026-11-19.
man page: environment variables, how env shapes a process, checked 2026-11-19.

systemd supervises process lifecycles

Sun, 01 Nov 2026 00:00:00 +0000

A service manager turns process startup, failure, and shutdown into observable state transitions. With systemd, a unit can name the foreground command, its restart policy, and its output destination. That removes the need for a shell loop that repeatedly launches a daemon without recording why it stopped. It does not prove that the running application can answer requests or that restarting repairs the underlying fault.

When it breaks

Choose a service type that matches the program’s startup contract. Type=simple considers startup successful early, before the executable has necessarily been invoked successfully. Type=exec waits for successful execution of the program, so a missing executable can fail the start job directly. Neither type waits for the application to finish connecting to its database or loading its state.

A program that backgrounds itself can confuse a unit expecting a foreground main process. Use the program’s foreground mode and let the manager own its lifetime, or choose the documented service type for its actual daemonization protocol. Avoid guessing a PID from a process-name search: names can collide, and wrappers can leave the wrong process designated as the service’s main process.

Restart policy also needs explicit limits. Restart=on-failure covers selected failure exits, not every possible termination signal or every unhealthy condition. A hung process that never exits will not automatically become a restart event. Combine meaningful application health observation with a bounded restart policy rather than equating an active unit with successful service delivery.

Check it quickly

Run the following Python script as an administrator in a disposable Linux environment whose PID 1 is systemd. It writes only named temporary runtime units and a temporary working directory. The recorded environment was Debian 12 with systemd 252.39 in an isolated container. Confirm the two unit filenames are unused before executing:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,time

unit='xbcatm-supervise-lab.service';bad='xbcatm-exec-failure-lab.service'
unit_path=Path('/run/systemd/system')/unit;bad_path=unit_path.with_name(bad)
assert not unit_path.exists() and not bad_path.exists()
def run(args,success=True):
 result=subprocess.run(args,text=True,capture_output=True,timeout=10)
 assert (result.returncode==0)==success,(args,result.returncode,result.stderr)
 return result.stdout.strip()
def value(name,key):return run(['systemctl','show',name,'-p',key,'--value'])
def wait_for(check):
 deadline=time.monotonic()+5
 while not check():
 assert time.monotonic()<deadline,'state transition timed out'
 time.sleep(0.02)
with TemporaryDirectory(prefix='xbcatm-supervise-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py';ready=root/'ready'
 worker.write_text("""from pathlib import Path
import os,signal,sys
signal.signal(signal.SIGTERM,lambda *_: sys.exit(0))
Path(__file__).with_name('ready').write_text(str(os.getpid()))
print('worker ready',flush=True)
while True: signal.pause()
""")
 unit_path.write_text(f'''[Unit]
Description=Disposable supervision exercise
StartLimitIntervalSec=10s
StartLimitBurst=3
[Service]
Type=exec
ExecStart=/usr/bin/python3 -u {worker}
Restart=on-failure
RestartSec=200ms
TimeoutStopSec=2s
StandardOutput=journal
StandardError=journal
''')
 bad_path.write_text(f'[Service]\nType=exec\nExecStart={root}/missing-program\n')
 try:
 run(['systemd-analyze','verify',str(unit_path)])
 run(['systemctl','daemon-reload']);run(['systemctl','start',unit])
 wait_for(lambda: ready.exists() and ready.read_text()==value(unit,'MainPID'))
 first=value(unit,'MainPID');assert first!='0'
 first_invocation=value(unit,'InvocationID')
 run(['systemctl','kill','--kill-who=main','--signal=KILL',unit])
 wait_for(lambda: value(unit,'MainPID') not in ('0',first) and ready.read_text()==value(unit,'MainPID'))
 assert int(value(unit,'NRestarts'))==1
 print('SIGKILL: replacement PID confirmed; NRestarts=1')
 run(['journalctl','--sync'])
 for invocation in (first_invocation,value(unit,'InvocationID')):
 logs=run(['journalctl','_SYSTEMD_INVOCATION_ID='+invocation,'--no-pager','-o','cat'])
 assert 'worker ready' in logs
 print('journal contains both application startups')
 run(['systemctl','stop',unit]);time.sleep(0.3)
 assert value(unit,'ActiveState')=='inactive' and value(unit,'MainPID')=='0'
 print('manager stop leaves service inactive')
 run(['systemctl','daemon-reload']);run(['systemctl','start',bad],success=False)
 assert value(bad,'Result')=='exit-code' and value(bad,'ExecMainStatus')=='203'
 print('Type=exec reports missing executable: status 203/EXEC')
 finally:
 run(['systemctl','stop',unit,bad])
 for name in (unit,bad):
 if value(name,'ActiveState')=='failed':
 run(['systemctl','reset-failed',name])
 unit_path.unlink();bad_path.unlink(missing_ok=True)
 run(['systemctl','daemon-reload'])

The worker stays in the foreground and publishes its own startup marker. The script waits until that marker agrees with the unit’s main PID, then sends SIGKILL specifically to the managed main process. A different PID and one automatic restart confirm that the manager actually replaced the terminated process.

Journal checks use the two invocation identifiers, so older entries from an earlier run cannot satisfy the assertion. Each invocation must contain the worker’s startup message. This checks both lifecycle replacement and attribution of the output produced by the two actual workers, rather than simply finding a familiar phrase somewhere in the unit’s accumulated history.

An explicit manager stop leaves the unit inactive with no main PID after the restart delay has passed. This is an intentional stop, not another crash to undo. The separate failing unit uses a nonexistent executable; its start command fails and records status 203/EXEC. That failure concerns process execution, not a database connection or application-level health probe.

The script verifies the valid unit definition before starting it and removes both runtime unit files afterward. Syntax verification and successful execution are separate evidence: the first checks the manager’s interpretation of configuration, while the second observes the behavior of a real process. Neither substitutes for checking a production endpoint.

Make the lifecycle operationally useful

Keep the installed unit definition, runtime configuration, and application readiness signal consistent. Ordering another unit after this service orders manager startup completion; if a dependent application needs a stronger readiness promise, use a supported notification protocol or an explicit application-level check. Retrying transient dependencies may still be part of the application’s own startup behavior.

Use systemctl show for specific state properties and the journal for the corresponding invocation. A unit’s current PID can change between observations, so identify the instance when diagnosing repeated failures. Track both restart frequency and whether useful work resumes; a successful restart counter is not an availability metric by itself.

Runtime files under /run/systemd/system are suitable for this disposable exercise. A deployed service also needs an intentional installation and enablement policy, appropriate user permissions, and a tested shutdown path. Changing a unit definition requires reloading the manager’s configuration; starting a service once does not automatically arrange for it to start at the next boot.

Sources

systemd 252 service units, startup types and restart semantics, checked 2026-09-12.
systemd 252 execution environment, output routing and execution exit codes, checked 2026-09-12.
systemd 252 unit configuration, runtime unit paths and start-rate limits, checked 2026-09-12.
systemd 252 journalctl, querying recorded fields and synchronizing journal writes, checked 2026-09-12.

Scheduled jobs need to be safe to rerun

Sat, 15 Aug 2026 00:00:00 +0000

A scheduler starts attempts, while the business usually cares about completing a logical task. One daily credit may be attempted several times after an error, a restart, or an operator retry. Giving every attempt a new identifier makes those duplicates indistinguishable from new work. Use a key that identifies the intended task, such as its name and nominal scheduled time, and define what a repeated key means before changing business state.

Scheduling and execution are separate

Do not assume every scheduler catches up missed work. Ordinary cron behavior varies by implementation and does not provide a universal catch-up guarantee. A systemd calendar timer with Persistent=true can trigger after an inactive interval when at least one activation was missed; that does not mean replaying every missed occurrence. Kubernetes documents that a CronJob may create duplicate Jobs or miss an occurrence under some circumstances.

Choose the required business rule explicitly: skip stale work, process each missing period, or reconcile the current state. Use the nominal logical period in the deduplication key, with an unambiguous time zone. An actual start timestamp changes across retries and therefore cannot identify the same intended occurrence. Keep an attempt identifier separately for diagnostics without confusing it with business identity.

When it breaks

A check followed by an unrelated update has a race. Two workers can both observe that a key is absent and both apply the effect. A process lock may prevent overlap on one machine but does not make a business update atomic with a marker file. If the process fails after the update and before writing the marker, a retry repeats the effect. Writing the marker first can instead lose the work.

When both state and completion record live in one transactional database, commit them together under a uniqueness constraint. A failure before commit should preserve neither change; a retry after commit should see the completed logical key. In SQLite, BEGIN IMMEDIATE acquires the write transaction early, and competing writers may wait or fail with a busy error. A timeout bounds that wait; it does not make every workload contention-free.

This boundary does not cover an external email, payment, or HTTP call. A local rollback cannot undo an effect already accepted elsewhere. Such workflows need an appropriate remote idempotency contract or a durable delivery design with deduplication at the receiver. Reusing a key with a different payload also requires a defined rejection policy; the example below always applies the same fixed credit.

Check it quickly

Save the code as idempotent-jobs.py and run python3 idempotent-jobs.py. It uses a temporary SQLite database and eight worker threads, each with its own database connection. The business effect adds ten units to a balance. A non-null primary key records which logical daily credit has completed within the same transaction.

from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from tempfile import TemporaryDirectory
from threading import Barrier
import json
import sqlite3

with TemporaryDirectory() as directory:
 database = str(Path(directory) / 'jobs.db')
 with sqlite3.connect(database) as connection:
 connection.executescript('''
CREATE TABLE balance (id INTEGER PRIMARY KEY, units INTEGER NOT NULL);
INSERT INTO balance VALUES (1, 0);
CREATE TABLE completed (logical_key TEXT PRIMARY KEY NOT NULL);
''')

 def run(key, fail=False, gate=None):
 connection = sqlite3.connect(database, timeout=10, isolation_level=None)
 try:
 if gate:
 gate.wait(timeout=5)
 connection.execute('BEGIN IMMEDIATE')
 inserted = connection.execute(
 'INSERT INTO completed VALUES (?) ON CONFLICT(logical_key) DO NOTHING', (key,))
 if not inserted.rowcount:
 connection.execute('COMMIT')
 return 'duplicate'
 connection.execute('UPDATE balance SET units = units + 10 WHERE id = 1')
 if fail:
 raise RuntimeError('injected failure before commit')
 connection.execute('COMMIT')
 return 'applied'
 except Exception:
 if connection.in_transaction:
 connection.execute('ROLLBACK')
 raise
 finally:
 connection.close()

 key = 'daily-credit:2026-09-13T00:00:00Z'
 try:
 run(key, fail=True)
 except RuntimeError as error:
 assert str(error) == 'injected failure before commit'
 with sqlite3.connect(database) as connection:
 assert connection.execute('SELECT units FROM balance').fetchone() == (0,)
 assert connection.execute('SELECT count(*) FROM completed').fetchone() == (0,)
 gate = Barrier(8)
 with ThreadPoolExecutor(max_workers=8) as pool:
 results = list(pool.map(lambda _: run(key, gate=gate), range(8)))
 assert results.count('applied') == 1 and results.count('duplicate') == 7
 assert run(key) == 'duplicate'
 assert run('daily-credit:2026-09-14T00:00:00Z') == 'applied'
 with sqlite3.connect(database) as connection:
 units = connection.execute('SELECT units FROM balance').fetchone()[0]
 completed = connection.execute('SELECT count(*) FROM completed').fetchone()[0]
 assert (units, completed) == (20, 2)
 print(json.dumps({'rolled_back_attempt_left_no_state': True,
 'concurrent_applied': results.count('applied'),
 'concurrent_duplicates': results.count('duplicate'),
 'final_units': units, 'completed_logical_jobs': completed}))

The first attempt intentionally raises after both inserts and updates but before commit. The script confirms that rollback leaves zero units and no completed key. Eight workers then start together for the same logical occurrence. Exactly one reports applied; seven report duplicate. Their unique-key decision and balance mutation are serialized by the actual database, not a mocked lock.

A further replay after successful commit remains a duplicate. A different nominal date is a new logical task and applies another ten units. The final result is twenty units and two completed keys. This verifies rollback and replay within one SQLite database, not scheduler delivery guarantees, process-kill recovery, or external exactly-once execution.

Keep failure and completion visible

Record the logical key, attempt outcome, duration, and last successful business period. Distinguish a harmless duplicate from a database error; catching every exception and labeling it completed would hide lost work. Define how long completion records must remain available before deleting them, since a retry after deletion can apply again. Operational evidence should tell an operator whether the intended period completed, rather than merely whether a timer launched a process.

Sources

SQLite: transactions, writer serialization, commit, and rollback, checked 2026-09-13.
SQLite: UPSERT, targeted handling of a uniqueness conflict, checked 2026-09-13.
systemd.timer, the scope of persistent calendar catch-up, checked 2026-09-13.
Kubernetes: CronJob, approximate scheduling and duplicate or missed jobs, checked 2026-09-13.

Least privilege is a threshold, not a feeling

Tue, 11 Aug 2026 00:00:00 +0000

“Run with the least privilege” sounds like a wish, but as an operational measure it only works when the capabilities are enumerated. A service user should be limited by explicit filesystem access, user and group setup, and systemd hardening instead of a general sense that “it does not need much”.

Define the minimum in writing

Least privilege is not a mood, it is a list, and a list you can write down. For a typical unit that is a read-only tree of the code itself, one writable directory for its own data, and its network port, nothing else. Say exactly that in the unit: User=, Group= for identity, a read-only mount for everything except the one data directory, and no grant for home, tmp, or the root filesystem. When a service needs to write only one place, making everything else read-only removes an entire category of mistakes. The sentence ‘it only needs a directory’ has to become a filesystem layout that proves it.

Filesystem it, verify it

A declared policy that is never checked is a wish. Start the service, then probe what it can actually reach: RestrictAddressFamilies, ProtectSystem=strict, PrivateDevices, NoNewPrivileges, and ProtectHome=yes are declarative guardrails in systemd that turn a wish into enforced boundaries. Where a single service must do one privileged thing, grant that narrow capability rather than giving it the root-granting whole; file ACLs or a specific capability let it expose one capability without root. The point is the boundary must be testable, so you can prove the box cannot read what it was not granted, not just believe it.

Why too-broad hurts later

The cost of ‘just give it everything now’ lands much later and compounds. A broadly privileged process turns a small bug in one component into a box-wide compromise, because the extra surface becomes a stepping stone when something else is broken. A narrow surface is the difference between one compromised service and erosion of the whole host. Treat privilege as an auditable, spelled-out property and re-review it whenever the service’s needs change, because the least-privilege threshold only stays true if you re-derive it as its day-to-day rights grow.

When it breaks

A wildcard RBAC grant deletes the threshold in one line. A rule with verbs: ["*"] on resources: ["*"] gives a deploy bot administrator reach: one leaked CI token can list Secrets and patch workloads. Every call succeeds until the credential is abused elsewhere, so the failure stays silent.

A long-lived token fails by outliving its purpose. A kubeconfig or token with admin scope is copied to laptops, runbooks, and logs, and still authenticates months later. A former teammate or an exported log keeps full access because no expiry or rotation cuts it off.

Permission drift without review is the slow version of the same breach. A temporary debug grant added during an incident is never removed, and without a re-review the live bindings diverge from the matrix. Each exception looks harmless alone, but together they restore the broad access the threshold was meant to remove.

Over-tightening breaks from the other side and creates shadow operations. Removing a write without offering the narrow path blocks a deploy, so the team routes around the control with root SSH or manual kubectl --as=system:admin. The audited path shows zero grants while the real work moves to an unaudited channel that is worse.

Check it quickly

Save the following as privilege_audit.py and run python3 privilege_audit.py. It models three roles as granted-action sets, replays seven requests through an allow-or-deny check, and compares each role against actions used in the review window. Only the standard library is used.

ROLES = {
 "api-reader": {"articles:read", "health:read"},
 "deployer": {"articles:read", "articles:publish", "health:read"},
 "ops-admin": {"articles:read", "articles:publish", "secrets:read", "users:write"},
}

# (role, action, should_allow)
REQUESTS = [
 ("api-reader", "articles:read", True),
 ("api-reader", "articles:publish", False),
 ("api-reader", "secrets:read", False),
 ("deployer", "articles:publish", True),
 ("deployer", "secrets:read", False),
 ("ops-admin", "secrets:read", True),
 ("ops-admin", "users:write", True),
]

USED = {"articles:read", "articles:publish", "health:read"}


def allowed(role, action):
 return action in ROLES[role]


for role, action, expected in REQUESTS:
 decision = allowed(role, action)
 assert decision is expected, (role, action, decision)
 print(f"{role:10} {action:17} -> {'allow' if decision else 'deny'}")

for role, granted in sorted(ROLES.items()):
 unused = sorted(granted - USED)
 print(f"{role:10} granted={len(granted)} unused={len(unused)} {unused}")
 if role == "ops-admin":
 assert unused == ["secrets:read", "users:write"], unused

print("PASS: 7 requests match the matrix; ops-admin holds 2 unused grants")

The seven requests pin the matrix: the reader reads articles but is denied publish and secret access, the deployer publishes but cannot read secrets, and the administrator reaches secrets and user writes. The usage comparison flags the drift: the first two roles have zero unused grants, while ops-admin holds two grants outside the used set.

Expected output ends with PASS: 7 requests match the matrix; ops-admin holds 2 unused grants, with one decision line per request and one usage line per role. The assertions passed with Python 3.12 on macOS. The script checks decision logic and grant hygiene, not live bindings, admission webhooks, or token lifetimes.

Sources

systemd.service man page, sandboxing options, the unit-level confinement options, checked 2026-08-12.
man page for capabilities, dividing root into fine-grained capabilities, checked 2026-08-12.
Open Web Application Security Project, least privilege, the developer-focused control, checked 2026-08-12.

Unix · XB Field Notes

Fork creates a process; exec replaces its program

When it breaks

Check it quickly

Make launch ownership explicit

Sources

Setuid runs as the file owner

Identity follows the file

Four ways the bit does nothing

Dropping privilege is not symmetric

The narrower replacements

When it breaks

Check it quickly

Sources

The sticky bit restricts removal in a shared directory

When it breaks

Check it quickly

Design shared storage deliberately

Sources

Environment variables are a per-process thing

Export selects values for another program

A subshell is not a freshly executed shell

Per-command settings have a defined lifetime

When it breaks

Check it quickly

Sources

A bare command the shell cannot find

Know which lookup layer answered

A slash selects a path directly

When it breaks

Check it quickly

Sources

Graceful shutdown stops work, not life

A signal begins coordination

The deadline covers a lifecycle

Coordinate with the supervisor

When it breaks

Check it quickly

Further reading

Sources

Give durable state and scratch space different lifecycles

When it breaks

Check it quickly

Make policy follow the data contract

Further reading

Sources

Named config files favor clarity

A file is a record of intent

Env and files solve different problems

Secrets still move outside

When it breaks

Check it quickly

Further reading

Sources

systemd supervises process lifecycles

When it breaks

Check it quickly

Make the lifecycle operationally useful

Further reading

Sources

Scheduled jobs need to be safe to rerun

Scheduling and execution are separate

When it breaks

Check it quickly

Keep failure and completion visible

Further reading

Sources

Least privilege is a threshold, not a feeling

Define the minimum in writing

Filesystem it, verify it

Why too-broad hurts later

When it breaks

Check it quickly

Further reading

Sources