A service manager turns process startup, failure, and shutdown into observable state transitions. With systemd, a unit can name the foreground command, its restart policy, and its output destination. That removes the need for a shell loop that repeatedly launches a daemon without recording why it stopped. It does not prove that the running application can answer requests or that restarting repairs the underlying fault.

When it breaks

Choose a service type that matches the program’s startup contract. Type=simple considers startup successful early, before the executable has necessarily been invoked successfully. Type=exec waits for successful execution of the program, so a missing executable can fail the start job directly. Neither type waits for the application to finish connecting to its database or loading its state.

A program that backgrounds itself can confuse a unit expecting a foreground main process. Use the program’s foreground mode and let the manager own its lifetime, or choose the documented service type for its actual daemonization protocol. Avoid guessing a PID from a process-name search: names can collide, and wrappers can leave the wrong process designated as the service’s main process.

Restart policy also needs explicit limits. Restart=on-failure covers selected failure exits, not every possible termination signal or every unhealthy condition. A hung process that never exits will not automatically become a restart event. Combine meaningful application health observation with a bounded restart policy rather than equating an active unit with successful service delivery.

Check it quickly

Run the following Python script as an administrator in a disposable Linux environment whose PID 1 is systemd. It writes only named temporary runtime units and a temporary working directory. The recorded environment was Debian 12 with systemd 252.39 in an isolated container. Confirm the two unit filenames are unused before executing:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,time

unit='xbcatm-supervise-lab.service';bad='xbcatm-exec-failure-lab.service'
unit_path=Path('/run/systemd/system')/unit;bad_path=unit_path.with_name(bad)
assert not unit_path.exists() and not bad_path.exists()
def run(args,success=True):
    result=subprocess.run(args,text=True,capture_output=True,timeout=10)
    assert (result.returncode==0)==success,(args,result.returncode,result.stderr)
    return result.stdout.strip()
def value(name,key):return run(['systemctl','show',name,'-p',key,'--value'])
def wait_for(check):
    deadline=time.monotonic()+5
    while not check():
        assert time.monotonic()<deadline,'state transition timed out'
        time.sleep(0.02)
with TemporaryDirectory(prefix='xbcatm-supervise-',dir='/run') as directory:
    root=Path(directory);worker=root/'worker.py';ready=root/'ready'
    worker.write_text("""from pathlib import Path
import os,signal,sys
signal.signal(signal.SIGTERM,lambda *_: sys.exit(0))
Path(__file__).with_name('ready').write_text(str(os.getpid()))
print('worker ready',flush=True)
while True: signal.pause()
""")
    unit_path.write_text(f'''[Unit]
Description=Disposable supervision exercise
StartLimitIntervalSec=10s
StartLimitBurst=3
[Service]
Type=exec
ExecStart=/usr/bin/python3 -u {worker}
Restart=on-failure
RestartSec=200ms
TimeoutStopSec=2s
StandardOutput=journal
StandardError=journal
''')
    bad_path.write_text(f'[Service]\nType=exec\nExecStart={root}/missing-program\n')
    try:
        run(['systemd-analyze','verify',str(unit_path)])
        run(['systemctl','daemon-reload']);run(['systemctl','start',unit])
        wait_for(lambda: ready.exists() and ready.read_text()==value(unit,'MainPID'))
        first=value(unit,'MainPID');assert first!='0'
        first_invocation=value(unit,'InvocationID')
        run(['systemctl','kill','--kill-who=main','--signal=KILL',unit])
        wait_for(lambda: value(unit,'MainPID') not in ('0',first) and ready.read_text()==value(unit,'MainPID'))
        assert int(value(unit,'NRestarts'))==1
        print('SIGKILL: replacement PID confirmed; NRestarts=1')
        run(['journalctl','--sync'])
        for invocation in (first_invocation,value(unit,'InvocationID')):
            logs=run(['journalctl','_SYSTEMD_INVOCATION_ID='+invocation,'--no-pager','-o','cat'])
            assert 'worker ready' in logs
        print('journal contains both application startups')
        run(['systemctl','stop',unit]);time.sleep(0.3)
        assert value(unit,'ActiveState')=='inactive' and value(unit,'MainPID')=='0'
        print('manager stop leaves service inactive')
        run(['systemctl','daemon-reload']);run(['systemctl','start',bad],success=False)
        assert value(bad,'Result')=='exit-code' and value(bad,'ExecMainStatus')=='203'
        print('Type=exec reports missing executable: status 203/EXEC')
    finally:
        run(['systemctl','stop',unit,bad])
        for name in (unit,bad):
            if value(name,'ActiveState')=='failed':
                run(['systemctl','reset-failed',name])
        unit_path.unlink();bad_path.unlink(missing_ok=True)
        run(['systemctl','daemon-reload'])

The worker stays in the foreground and publishes its own startup marker. The script waits until that marker agrees with the unit’s main PID, then sends SIGKILL specifically to the managed main process. A different PID and one automatic restart confirm that the manager actually replaced the terminated process.

Journal checks use the two invocation identifiers, so older entries from an earlier run cannot satisfy the assertion. Each invocation must contain the worker’s startup message. This checks both lifecycle replacement and attribution of the output produced by the two actual workers, rather than simply finding a familiar phrase somewhere in the unit’s accumulated history.

An explicit manager stop leaves the unit inactive with no main PID after the restart delay has passed. This is an intentional stop, not another crash to undo. The separate failing unit uses a nonexistent executable; its start command fails and records status 203/EXEC. That failure concerns process execution, not a database connection or application-level health probe.

The script verifies the valid unit definition before starting it and removes both runtime unit files afterward. Syntax verification and successful execution are separate evidence: the first checks the manager’s interpretation of configuration, while the second observes the behavior of a real process. Neither substitutes for checking a production endpoint.

Make the lifecycle operationally useful

Keep the installed unit definition, runtime configuration, and application readiness signal consistent. Ordering another unit after this service orders manager startup completion; if a dependent application needs a stronger readiness promise, use a supported notification protocol or an explicit application-level check. Retrying transient dependencies may still be part of the application’s own startup behavior.

Use systemctl show for specific state properties and the journal for the corresponding invocation. A unit’s current PID can change between observations, so identify the instance when diagnosing repeated failures. Track both restart frequency and whether useful work resumes; a successful restart counter is not an availability metric by itself.

Runtime files under /run/systemd/system are suitable for this disposable exercise. A deployed service also needs an intentional installation and enablement policy, appropriate user permissions, and a tested shutdown path. Changing a unit definition requires reloading the manager’s configuration; starting a service once does not automatically arrange for it to start at the next boot.

Further reading

Sources