Graceful shutdown is a transition in how a service accepts and finishes work. Receiving a signal is only the trigger. The useful guarantee is narrower than “nothing is lost”: stop admitting new business work, account for what was already accepted, allow bounded completion, and report honestly when that bound expires.
A signal begins coordination
SIGTERM can be handled so an application starts its shutdown sequence. SIGKILL cannot be caught, blocked, or ignored. Verify which process receives the stop signal: a handler inside a worker does not help if a wrapper receives the signal and never forwards it.
Keep the signal handler small. Python executes its handlers in the main interpreter thread and warns against synchronization locks inside them. The fixture only sets a flag; ordinary control flow changes readiness and admission under a lock. It does not perform network I/O or wait for workers inside the handler.
Readiness should be withdrawn early enough for routing changes to begin. Admission still needs its own local gate, because existing connections and delayed routing updates can deliver requests afterward. Changing the gate and registering accepted work must use a consistent boundary.
The deadline covers a lifecycle
Track work until its real completion point. A caller timeout is not proof that a worker stopped, and a sent response is not proof that a business transaction is durable. Select the completion boundary appropriate to the operation instead of treating every closed socket as success.
Use a monotonic deadline for local elapsed time. Leave room for cleanup within the supervisor’s actual allowance. When the deadline expires, remaining work needs an explicit cancellation, abandonment, or durable handoff policy. Merely breaking out of a wait loop does not cancel a running thread.
Successful draining and incomplete shutdown should remain distinguishable in records. A particular nonzero exit code is a service contract, not a universal rule that guarantees correct recovery. A client losing a response may still need idempotency or reconciliation before retrying.
Coordinate with the supervisor
In systemd, TimeoutStopSec governs configured stop-command and service-stop waits. KillMode determines the processes targeted, not how much drain time they receive. With mixed mode, the initial termination signal targets the main process and final escalation targets remaining control-group processes; early main-process exit can also trigger that escalation. Review stop hooks and notification-based extensions in the real unit.
Kubernetes begins the termination grace period before running a configured preStop hook. The hook does not receive an independent extra drain budget. EndpointSlice termination updates and local shutdown proceed alongside each other; they are not an instantaneous global barrier after which no request can arrive.
The normal stop signal can depend on image or runtime configuration. Test the deployed process tree and stop contract rather than assuming the behavior of a locally executed program also proves container, sidecar, and proxy coordination.
When it breaks
A shutdown flag that only hides readiness can leave a keep-alive connection admitting new work forever. Conversely, closing every active socket immediately can discard requests that had enough time to finish. Stopping admission and finishing accepted work are separate decisions.
An unbounded join can exceed the supervisor’s allowance. Daemon threads allow process exit without waiting, but that is abandonment, not safe cancellation or transaction rollback. Libraries have their own shutdown contracts: Python’s socketserver shutdown must be called from a different thread than serve_forever or it deadlocks.
Observe accepted, completed, refused, and unfinished work, together with shutdown duration and exit outcome. A clean-looking exit without those distinctions can hide dropped work. The goal is a verified lifecycle, not just a registered signal handler.
Check it quickly
Save this block as shutdown_check.py and run python3 shutdown_check.py with Python 3.10 or newer on a Unix-like system. It ran on macOS with Python 3.14.4. Each case creates its own child process and loopback HTTP server; SIGTERM targets only that child. The POST release route is test coordination, not a production administration API.
from concurrent.futures import ThreadPoolExecutor
from http.client import HTTPConnection, RemoteDisconnected
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import selectors
import signal
import subprocess
import sys
from threading import Condition, Event, Thread
import time
def child(grace):
state = {"draining": False, "active": 0}
condition, release = Condition(), Event()
stop_requested = False
def stop(signum, frame):
nonlocal stop_requested
stop_requested = True # No lock, I/O, or shutdown call in the signal handler.
class Handler(BaseHTTPRequestHandler):
def log_message(self, *args):
pass
def reply(self, status, value):
body = json.dumps(value).encode()
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.send_header("Cache-Control", "no-store")
self.end_headers()
self.wfile.write(body)
def do_GET(self):
if self.path == "/state":
with condition:
snapshot = dict(state)
self.reply(200, snapshot)
elif self.path == "/readyz":
with condition:
ready = not state["draining"]
self.reply(200 if ready else 503, {"ready": ready})
elif self.path == "/work":
with condition:
admitted = not state["draining"]
if admitted:
state["active"] += 1
if not admitted:
self.reply(503, {"error": "draining"})
return
try:
if release.wait(timeout=5):
self.reply(200, {"done": True})
else:
self.reply(503, {"error": "fixture-timeout"})
finally:
with condition:
state["active"] -= 1
condition.notify_all()
else:
self.reply(404, {"error": "not-found"})
def do_POST(self):
if self.path == "/release": # Loopback-only test coordination, not workload admission.
self.reply(200, {"released": True})
release.set() # Finish the control reply before work can complete and the child exits.
else:
self.reply(404, {"error": "not-found"})
signal.signal(signal.SIGTERM, stop)
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
server.daemon_threads = True # Explicitly permits abandonment in the deadline case.
serving = Thread(target=server.serve_forever, kwargs={"poll_interval": 0.01})
serving.start()
print(json.dumps({"port": server.server_port}), flush=True)
while not stop_requested:
time.sleep(0.005)
deadline = time.monotonic() + grace
with condition:
state["draining"] = True
while state["active"] and time.monotonic() < deadline:
condition.wait(timeout=max(0, deadline - time.monotonic()))
unfinished = state["active"]
server.shutdown() # Called outside the thread running serve_forever.
serving.join(timeout=2)
server.server_close() # Does not wait for daemon request threads.
assert not serving.is_alive()
print(json.dumps({"unfinished": unfinished, "deadline_expired": unfinished > 0}), flush=True)
return 3 if unfinished else 0 # Fixture policy, not a universal exit-code convention.
def request(port, route, method="GET"):
connection = HTTPConnection("127.0.0.1", port, timeout=3)
try:
connection.request(method, route)
response = connection.getresponse()
return response.status, json.loads(response.read())
finally:
connection.close()
def wait_state(port, predicate):
deadline = time.monotonic() + 3
while time.monotonic() < deadline:
status, state = request(port, "/state")
assert status == 200
if predicate(state):
return state
time.sleep(0.01)
raise AssertionError("child did not reach the expected state")
def exercise(finish):
process = subprocess.Popen(
[sys.executable, __file__, "child", "3" if finish else "1"],
stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
)
try:
with selectors.DefaultSelector() as selector:
selector.register(process.stdout, selectors.EVENT_READ)
assert selector.select(timeout=5), "child did not announce its local port"
port = json.loads(process.stdout.readline())["port"]
assert request(port, "/readyz") == (200, {"ready": True})
with ThreadPoolExecutor(max_workers=1) as pool:
active = pool.submit(request, port, "/work")
wait_state(port, lambda state: state["active"] == 1)
process.send_signal(signal.SIGTERM)
wait_state(port, lambda state: state["draining"])
assert request(port, "/readyz") == (503, {"ready": False})
assert request(port, "/work") == (503, {"error": "draining"})
assert request(port, "/state")[1]["active"] == 1
if finish:
assert request(port, "/release", "POST")[0] == 200
assert active.result(timeout=4) == (200, {"done": True})
output, errors = process.communicate(timeout=5)
assert errors == ""
record = json.loads(output)
assert process.returncode == (0 if finish else 3)
assert record == {"unfinished": 0 if finish else 1, "deadline_expired": not finish}
if not finish:
try:
active.result(timeout=4)
except (OSError, RemoteDisconnected):
pass
else:
raise AssertionError("abandoned work must not report a successful reply")
print("completed drain" if finish else "expired drain", record, "exit", process.returncode)
finally:
if process.poll() is None:
process.kill() # Safety cleanup targets only this fixture's child.
process.wait(timeout=5)
process.stdout.close()
process.stderr.close()
if __name__ == "__main__":
if sys.argv[1:2] == ["child"]:
raise SystemExit(child(float(sys.argv[2])))
exercise(True)
exercise(False)
Both cases admit one request before SIGTERM. After the drain flag becomes visible, readiness returns 503, a new business request receives 503, and the admitted request remains counted.
Releasing that request lets it return 200 before the child exits zero with no unfinished work. Without release, the one-second drain allowance expires; the child reports one unfinished request, exits three, and the waiting client receives no successful reply.
The latter case deliberately abandons a daemon worker. It does not prove cooperative cancellation, durability, SIGKILL cleanup, or safe replay. No systemd unit, Kubernetes workload, real transaction, or production traffic was exercised.
Further reading
Sources
- Linux signal(7), termination signals and the uncatchable SIGKILL boundary, checked 2026-09-09.
- Python signals, main-thread handlers and the warning against locks in handlers, checked 2026-09-09.
- Python socketserver, shutdown-thread requirements and daemon request-thread behavior, checked 2026-09-09.
- systemd.service, TimeoutStopSec stop waits and notification extensions, checked 2026-09-09.
- systemd.kill, KillMode process scope and termination escalation, checked 2026-09-09.
- Kubernetes: Pod lifecycle, preStop, termination grace, stop signals, and EndpointSlice transitions, checked 2026-09-09.
