systemd restart policies need explicit failure and rate rules

Sun, 07 Nov 2027 00:00:00 +0000

A restart policy decides whether a service manager should try another process after the current one ends. It can recover from a transient failure, but it cannot correct a missing configuration or a deterministic bug. The useful contract has three parts: which endings qualify, how long to wait, and how many starts are permitted within a time window. Those settings answer different questions.

When it breaks

Restart=on-failure does not mean restart after every signal. For service types other than oneshot, ordinary SIGHUP, SIGINT, SIGTERM, and SIGPIPE terminations are normally considered clean. A nonzero exit or an unclean signal such as SIGKILL qualifies under the default success rules. SuccessExitStatus and the restart-specific exception directives can change classification, so inspect the effective unit.

on-abnormal is not simply a stricter synonym for on-failure: it excludes ordinary nonzero process exit codes while covering abnormal signals and certain timeout or watchdog failures. Choose according to how the program reports recoverable and intentional endings. A process that remains alive but stops doing useful work needs a separately configured detection mechanism; exit-based restart logic cannot infer that condition.

RestartSec supplies a delay before restarting. In the systemd 252 exercise below, it is a fixed delay, not an exponentially increasing backoff. StartLimitIntervalSec and StartLimitBurst limit starts within an interval, including manual starts. They are not a lifetime retry counter, and reaching the limit does not schedule an automatic recovery attempt at the next interval boundary.

Check it quickly

Run this Python script as an administrator in a disposable Linux systemd environment. It creates a uniquely named runtime unit and a temporary worker, then removes them. The recorded run used Debian 12 with systemd 252.39. The worker initially exits with status seven and records every actual invocation:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,time,uuid

unit='xbcatm-restart-lab-'+uuid.uuid4().hex[:8]+'.service'
path=Path('/run/systemd/system')/unit
assert not path.exists()
def ctl(*args):
 result=subprocess.run(['systemctl',*args],text=True,capture_output=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def value(key):return ctl('show',unit,'-p',key,'--value')
def wait_for(check):
 deadline=time.monotonic()+5
 while not check():
 assert time.monotonic()<deadline,'state transition timed out'
 time.sleep(0.02)
with TemporaryDirectory(prefix='xbcatm-restart-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py';calls=root/'calls';healthy=root/'healthy'
 worker.write_text("""from pathlib import Path
import sys,time
root=Path(__file__).parent
with (root/'calls').open('a') as output: output.write('started\\n')
if not (root/'healthy').exists(): sys.exit(7)
time.sleep(60)
""")
 path.write_text(f'''[Unit]
StartLimitIntervalSec=10s
StartLimitBurst=3
[Service]
Type=exec
ExecStart=/usr/bin/python3 {worker}
Restart=on-failure
RestartSec=100ms
''')
 try:
 ctl('daemon-reload');ctl('start',unit)
 wait_for(lambda: value('ActiveState')=='failed')
 assert calls.read_text().splitlines()==['started']*3
 assert value('Result')=='exit-code'
 subprocess.run(['journalctl','--sync'],check=True)
 logs=subprocess.run(['journalctl','-u',unit,'--no-pager','-o','cat'],capture_output=True,text=True,check=True).stdout
 assert 'Start request repeated too quickly' in logs
 print('three starts; failed/exit-code with start-rate-limit journal message')
 healthy.touch();ctl('reset-failed',unit);ctl('start',unit)
 wait_for(lambda: len(calls.read_text().splitlines())==4)
 first=value('MainPID');assert first!='0'
 ctl('kill','--kill-who=main','--signal=TERM',unit)
 wait_for(lambda: value('ActiveState')=='inactive')
 assert value('NRestarts')=='0'
 print('on-failure treats ordinary SIGTERM as clean; no restart')
 ctl('start',unit);first=value('MainPID')
 ctl('kill','--kill-who=main','--signal=KILL',unit)
 wait_for(lambda: value('MainPID') not in ('0',first))
 assert value('NRestarts')=='1'
 print('SIGKILL triggers one restart')
 ctl('stop',unit);time.sleep(0.2)
 assert value('ActiveState')=='inactive'
 print('explicit stop remains stopped')
 finally:
 ctl('stop',unit)
 if value('ActiveState')=='failed':ctl('reset-failed',unit)
 path.unlink();ctl('daemon-reload')

The first phase permits three starts in ten seconds with a short fixed restart delay. The worker’s own file contains exactly three entries, so the count represents actual process launches rather than a counter whose meaning was assumed. The manager then leaves the unit failed and records that the start request was repeated too quickly.

On the tested version, the terminal Result remains exit-code, reflecting the preceding process failure, even though the journal identifies start-rate limiting. Requiring a literal start-limit-hit result would miss this observed case. Use state, application invocation evidence, and the corresponding journal message together rather than relying on one expected diagnostic label across versions and failure paths.

The script repairs the fixture by creating its healthy marker, clears the failed state and rate counter, and starts it again. This time the process remains alive. Sending an ordinary SIGTERM produces an inactive unit with no automatic restart under on-failure. A later SIGKILL does cause one replacement process.

Finally, an explicit systemctl stop leaves the service inactive after the restart delay. Manager-requested shutdown is not treated as a crash that policy should immediately reverse. The cleanup handles failed state only when present, since an inactive unreferenced unit can be unloaded by the manager.

Recover the service, not just its counter

Read the failure before resetting its evidence. Inspect the configured command, exit status, recent invocation logs, and any dependency errors. Correct the cause or deploy a known working version, then reset the limiter if an immediate retry is justified. Repeatedly clearing the counter without changing the failure simply converts a bounded crash loop into an operator-driven one.

Alert on sustained unavailability and excessive restarts using the monitoring system actually responsible for the service. The unit settings shown here do not create an alert destination. Retain enough diagnostic context to distinguish failed execution, an application exit, resource exhaustion, and a deliberate stop without logging sensitive configuration.

Select the delay and burst to match startup cost and dependency behavior. A rapid retry may recover quickly from one fault but amplify load during a shared outage. Verify both recovery and termination in the deployed configuration, including any drop-ins. A policy is useful when the application resumes useful work and remains controllable, not merely when a PID repeatedly changes.

Sources

systemd 252 service restart rules, exit classification, fixed delay, and restart exceptions, checked 2026-09-12.
systemd 252 unit start limits, interval limits, manual starts, and counter reset, checked 2026-09-12.
systemd 252 systemctl, stop, state inspection, and reset-failed operations, checked 2026-09-12.

Restart · XB Field Notes

systemd restart policies need explicit failure and rate rules

When it breaks

Check it quickly

Recover the service, not just its counter

Sources