Automation · XB Field Notes

Cron repeats on the calendar

Thu, 30 Sep 2027 00:00:00 +0000

Cron matches calendar fields and launches commands at matching times. It does not inherently wait for an earlier invocation of the same command to finish. If a minute-level task takes seventy seconds, the next minute can launch another copy. Treat trigger policy, concurrency policy, and successful completion as separate responsibilities when operating recurring work.

Read fields as calendar selections

A conventional user crontab entry contains minute, hour, day of month, month, day of week, then the command. Asterisks select all values in their fields. A step such as */15 in minutes selects minute values within each hour; it is not a timer measuring fifteen minutes since the previous completion. A step that does not divide the field range evenly can produce unequal gaps across boundaries.

Day matching has a special rule in the Debian/Vixie format: when day of month and day of week are both restricted, either matching day can permit execution, while the minute, hour, and month constraints still apply. This is not a disjunction between month and weekday. Write representative dates down and verify the intended calendar before installing a terse expression.

Timezone, daylight-saving changes, and downtime affect calendar scheduling. Implementations differ in how they handle skipped or repeated times. Debian cron documents special handling for certain small clock changes and fixed-time jobs, so rules copied from another scheduler may be wrong here. This experiment does not change the clock or test those transition cases. A calendar trigger is also not a durable queue that automatically guarantees every missed run will be replayed.

When it breaks

Overlapping jobs can duplicate side effects, compete for resources, or overwrite shared files. If one invocation must run at a time, implement that policy explicitly. A nonblocking advisory lock can let a later invocation record a skip; a blocking lock can instead leave waiting processes. Neither choice provides exactly-once business effects or repairs a partially completed operation.

A local file lock only coordinates participants using the same relevant locking facility. It does not automatically serialize jobs on unrelated hosts, and network filesystem semantics require separate examination. Keep the lock held for the work’s lifetime, and make skipped work visible. Silently dropping every later trigger while one worker hangs can look like a healthy schedule unless completion age is monitored.

Cron is not universally log-free. Debian logs job starts to syslog by default and offers additional logging controls. Command output may be mailed when a working mail path exists, but a minimal container often lacks that path. Record useful start, completion, failure, and skip outcomes through an intentional destination. A start message proves dispatch, not completion of the application’s task.

Check it quickly

Use a fresh disposable Debian 12 Docker container as root, with no existing root crontab or cron daemon. Install python3, cron, and procps using apt, save the following as cron-overlap.py, and run python3 cron-overlap.py. Allow roughly two minutes: the script waits for actual calendar ticks rather than replacing cron with a simulated scheduler.

from pathlib import Path
from tempfile import TemporaryDirectory
import json
import os
import subprocess
import time

assert os.geteuid() == 0 and Path('/.dockerenv').exists(), 'use a disposable root Linux container'
assert subprocess.run(['pgrep', '-x', 'cron'], capture_output=True).returncode == 1
assert subprocess.run(['crontab', '-l'], capture_output=True).returncode != 0

with TemporaryDirectory(prefix='cron-overlap-') as directory:
 root = Path(directory)
 events_path = root / 'events.jsonl'
 stop = root / 'stop'
 worker = root / 'worker.py'
 worker.write_text('''from pathlib import Path
import fcntl,json,os,sys,time
root=Path(__file__).parent
kind=sys.argv[1]
def record(event):
 data=json.dumps({'event':event,'kind':kind,'pid':os.getpid(),'monotonic':time.monotonic()})+'\\n'
 fd=os.open(root/'events.jsonl',os.O_WRONLY|os.O_CREAT|os.O_APPEND,0o600)
 try: os.write(fd,data.encode())
 finally: os.close(fd)
lock=None
if kind=='locked':
 lock=open(root/'job.lock','a')
 try: fcntl.flock(lock,fcntl.LOCK_EX|fcntl.LOCK_NB)
 except BlockingIOError:
 record('skip')
 raise SystemExit(0)
record('start')
try:
 until=time.monotonic()+70
 while time.monotonic()<until and not (root/'stop').exists(): time.sleep(0.1)
finally:
 record('end')
 if lock: lock.close()
''')
 table = ''.join(f'* * * * * /usr/bin/python3 {worker} {kind} >>{root}/worker.log 2>&1\n'
 for kind in ('plain', 'locked'))
 subprocess.run(['crontab', '-'], input=table, text=True, check=True)
 daemon = subprocess.Popen(['cron', '-f'], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
 def read_events():
 if not events_path.exists():
 return []
 return [json.loads(line) for line in events_path.read_text().splitlines()]
 try:
 deadline = time.monotonic() + 140
 while time.monotonic() < deadline:
 assert daemon.poll() is None, 'cron exited'
 events = read_events()
 plain = [e for e in events if (e['kind'], e['event']) == ('plain', 'start')]
 locked = [e for e in events if (e['kind'], e['event']) == ('locked', 'start')]
 skipped = [e for e in events if (e['kind'], e['event']) == ('locked', 'skip')]
 if len(plain) >= 2 and skipped:
 break
 time.sleep(0.2)
 else:
 raise AssertionError('two real cron ticks were not observed')
 assert len(plain) == 2 and len(locked) == 1 and len(skipped) == 1
 assert plain[1]['monotonic'] - plain[0]['monotonic'] < 70
 assert not any(e['event'] == 'end' and e['pid'] == plain[0]['pid'] for e in events)
 print(json.dumps({'actual_cron_starts': len(plain), 'overlap_observed': True,
 'seconds_between_starts': round(plain[1]['monotonic']-plain[0]['monotonic'], 3),
 'locked_starts': len(locked), 'locked_skips': len(skipped)}), flush=True)
 finally:
 daemon.terminate()
 stop.touch()
 daemon.communicate(timeout=5)
 subprocess.run(['crontab', '-r'], check=True)
 deadline = time.monotonic() + 5
 while time.monotonic() < deadline:
 events = read_events()
 started = {e['pid'] for e in events if e['event'] == 'start'}
 ended = {e['pid'] for e in events if e['event'] == 'end'}
 if started <= ended:
 break
 time.sleep(0.1)
 else:
 raise AssertionError('test workers did not finish')

Two every-minute entries run workers that would last seventy seconds. The plain worker has no lock. The locked worker uses a shared file and a nonblocking exclusive flock, recording a skip when acquisition fails. Each event includes the worker PID and a monotonic timestamp, so distinct invocations and their overlap can be checked directly.

On Debian cron 3.0pl1-162, the plain worker started twice, 60.003 seconds apart, before its first invocation ended. The guarded worker started once and recorded one skip. All assertions passed. Cleanup stopped cron, signaled the test workers through their stop file, waited for their end records, and removed the temporary crontab. The experiment verifies local overlap and explicit exclusion; it does not implement durable retries, distributed locking, recovery after reboot, or exactly-once effects. Select those policies from the job’s actual business contract instead of assuming the calendar expression supplies them.

Sources

Debian 12: crontab(5), calendar fields, steps and the two day fields, checked 2026-09-13.
Debian 12: cron(8), daemon execution, logging and clock-change behavior, checked 2026-09-13.
Linux: flock(2), advisory locks and nonblocking acquisition, checked 2026-09-13.

GitHub Actions workflows gate the build

Thu, 02 Sep 2027 00:00:00 +0000

CI that shows on every pull request keeps reviews honest. The workflow YAML in the repo is that contract, versioned with the code it checks.

YAML is the contract

The workflow YAML says both when something runs and with what rights, so it is the whole contract rather than just a recipe. A on: block names the triggers, a permissions: block pins what the workflow can touch, and a jobs: section declares the runners and steps. Because on is read before any job runs, a workflow is also where you decide that a pull_request must pass tests to be mergeable or a schedule can run it unsupervised. Version the file with the code it checks, so a branch’s CI always reflects the branch’s own contract.

Jobs and steps run in order

Jobs default to running in parallel, while steps inside a job run sequentially, so the graph you get is the combination of those two rules. A failing step makes the rest of its job conditional; whether later jobs still run is controlled by needs/if, and by the job-level concurrency and matrix settings. This ordering is what turns a workflow into a gate that blocks a merge on red, yet it is fully readable from the file. The practical habit is to keep the fast checks early and the expensive ones downstream, so the gate fails cheap before it fails slowly.

Secrets stay close to the runner

Secrets are injected on the runner, scoped to the workflow, and never printed, but a workflow with broad permissions can still read the repository, so the smallest effective grant is the safety mile. Give each job the permissions it needs instead of the default wider set, and scope environments so secrets reach only the jobs that truly run there. Validate the effective permissions the way you would a production role: a workflow is code that runs with your credentials on every trigger, so a malicious or outdated entry is a long-term credential you are checking in. Rotate anything that must live outside the branch.

When it breaks

A green workflow means nothing if branch protection does not require it. The classic failure is renaming the gate job from test to verify while the rule still waits for the old name: requests block forever on a check that never reports, or an administrator allows bypass and the gate becomes decoration. Fork requests add a second variant: secrets stay read-only, a job needing a deploy token is skipped, and a queue that treats skipped as passed ships untested code. Audit the required list after every rename.

The other failures look like flakes until you read the file. A cache keyed only on runner.os restores last week’s node_modules after the lockfile changed, so green means stale, and a cache written by a fork request can poison main when keys collide. Secrets leak quietly: set -x, verbose curl, or echo ${{ secrets.TOKEN }} prints the value into logs every reader keeps. Two pushes sharing one concurrency group cancel each other, the deploy job dies halfway, and the retry assumes the first run finished.

Check it quickly

Save the following as gate_check.py in a scratch directory and run python3 gate_check.py. It uses only the standard library and writes no files. The bad sample is a negative control, not a template.

"""Minimal gate check for a GitHub Actions workflow (standard library only)."""
import re

GOOD = """name: gate
on:
 pull_request:
permissions:
 contents: read
jobs:
 test:
 runs-on: ubuntu-latest
 permissions:
 contents: read
 steps:
 - run: npm test
 gate:
 needs: [test]
 if: github.event_name == 'pull_request'
 runs-on: ubuntu-latest
 permissions: {}
 steps:
 - run: echo gate ok
"""

BAD = """name: gate
on: [push]
jobs:
 test:
 runs-on: ubuntu-latest
 steps:
 - run: echo ${{ secrets.DEPLOY_TOKEN }}
 deploy:
 runs-on: ubuntu-latest
 permissions:
 contents: write
 steps:
 - run: echo deploy
"""


def check_workflow(text):
 """Return a list of gate violations found in a workflow string."""
 errors = []
 if not re.search(r"(?m)^permissions\s*:", text):
 errors.append("missing top-level permissions: pin contents: read at the top")
 else:
 top = text.split("jobs:")[0] if "jobs:" in text else text
 if re.search(r"write-all|write", top):
 errors.append("top-level permissions too broad: use contents: read")
 if "jobs:" not in text:
 return errors + ["missing jobs: section"]
 jobs = {}
 current = None
 buf = []
 for line in text.split("jobs:", 1)[1].splitlines():
 head = re.match(r"^ ([A-Za-z0-9_-]+):\s*$", line)
 if head:
 if current:
 jobs[current] = "\n".join(buf)
 current = head.group(1)
 buf = [line]
 elif current is not None:
 buf.append(line)
 if current:
 jobs[current] = "\n".join(buf)
 if not jobs:
 return errors + ["no jobs found under jobs:"]
 if not any(re.search(r"(?m)^\s+needs\s*:", chunk) for chunk in jobs.values()):
 errors.append("missing needs: no job waits for tests, the gate cannot order checks")
 if not any(re.search(r"(?m)^\s+if\s*:", chunk) for chunk in jobs.values()):
 errors.append("missing if: no job guards on the event, costly work runs everywhere")
 for name, chunk in jobs.items():
 if not re.search(r"(?m)^\s+permissions\s*:", chunk):
 errors.append(f"job '{name}' missing permissions: grant each job least privilege")
 elif re.search(r"write-all|contents\s*:\s*write", chunk):
 errors.append(f"job '{name}' permissions too broad: keep contents: read or {{}}")
 for line in text.splitlines():
 if "secrets." in line and re.search(r"\becho\b|set\s+-x|curl\s+.*-v", line):
 errors.append(f"possible secret leak to logs: {line.strip()[:80]}")
 break
 return errors


def check(ok, message):
 if not ok:
 raise AssertionError(message)


def main():
 good = check_workflow(GOOD)
 bad = check_workflow(BAD)
 print(f"good workflow: {len(good)} problems")
 for item in good:
 print(f" unexpected: {item}")
 print(f"bad workflow: {len(bad)} problems")
 for item in bad:
 print(f" found: {item}")
 check(not good, f"good sample must pass, got: {good}")
 check(len(bad) >= 3, f"bad sample must fail visibly, got: {bad}")
 check(any("needs" in item for item in bad), "bad sample must expose missing needs")
 check(any("permissions" in item for item in bad), "bad sample must expose permissions")
 check(any("secret" in item for item in bad), "bad sample must expose secret leak")
 print("PASS: gate waits on tests, guards on the event, and stays least-privilege")


if __name__ == "__main__":
 main()

The bad sample fails the three gate rules at once. A missing top-level permissions block and a job without its own grant fail least privilege, and contents: write fails it explicitly. No job declares needs, so nothing waits for tests, and no job declares if, so costly work runs on every push. Echoing secrets.DEPLOY_TOKEN trips the leak check. The good sample passes because test stays read-only and gate waits on it behind an event guard.

The assertions passed with Python 3.12 on macOS. The fixture matches indentation with regular expressions, not a real YAML parser, so it misses anchors and reusable workflows. It also cannot see branch protection or secret rotation. Before deployment, check a real workflow and confirm the required name in the branch settings.

Sources

GitHub: workflow syntax, the workflow schema, checked 2027-08-15.
GitHub: understanding actions, the event-driven model, checked 2027-08-15.

A systemd timer schedules a service, not its success

Mon, 21 Jun 2027 00:00:00 +0000

A systemd timer separates the scheduling rule from the service that performs the work. The timer decides when to request activation; the service supplies the command, execution identity, environment, timeout, and result. This separation makes a maintenance job inspectable using normal unit tools. It does not turn a scheduled attempt into proof that a backup, cleanup, or report completed correctly.

When it breaks

An active timer can coexist with a failed service. Monitoring only the timer’s active state therefore misses failures in the actual work. Likewise, a recent trigger timestamp identifies scheduling activity, not successful business output. Check the service’s result and the artifact or state change the job was supposed to produce, with enough context to distinguish successive activations.

A timer normally activates the service with the matching basename, unless Unit selects another target. If that service is already active, the timer does not launch another independent copy. This prevents overlapping activations of that one unit, but it is not a distributed lock or protection against another service running the same command. A oneshot kept active by RemainAfterExit=yes can consequently prevent later scheduled executions.

Stopping a timer stops future scheduling. It does not automatically stop a service invocation already in progress. That distinction matters when pausing maintenance during an incident: inspect the target service and decide deliberately whether current work should finish or be interrupted. Interrupting a command also does not roll back its earlier filesystem or database changes.

Check it quickly

Run this Python script as an administrator in a disposable Linux system with systemd as PID 1. The recorded run used Debian 12 and systemd 252.39. It creates a unique timer/service pair and a temporary worker. The worker records real process entries and completion times; explicit release files hold successful jobs open so the scheduling behavior is observable:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,time,uuid

name='xbcatm-timer-'+uuid.uuid4().hex[:8];service=name+'.service';timer=name+'.timer'
paths={unit:Path('/run/systemd/system')/unit for unit in (service,timer)}
assert all(not p.exists() for p in paths.values())
def run(*args):
 result=subprocess.run(args,text=True,capture_output=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def value(unit,key):return run('systemctl','show',unit,'-p',key,'--value')
def wait_for(check):
 deadline=time.monotonic()+5
 while not check():
 assert time.monotonic()<deadline,'transition timed out'
 time.sleep(0.02)
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py'
 worker.write_text(r'''from pathlib import Path
import json,sys,time
root=Path(__file__).parent;number=len(list(root.glob('start-*.json')))+1
def record(kind):
 temp=root/'record.tmp';temp.write_text(json.dumps(time.monotonic()))
 temp.replace(root/(kind+'-'+str(number)+'.json'))
record('start')
if (root/'fail').exists():sys.exit(7)
while not (root/('release-'+str(number))).exists():time.sleep(0.01)
record('end')
''')
 paths[service].write_text(f'[Service]\nType=oneshot\nTimeoutStartSec=10s\nExecStart=/usr/bin/python3 {worker}\n')
 definition='[Timer]\nOnActiveSec=100ms\nOnUnitInactiveSec=10s\nAccuracySec=1ms\n'
 paths[timer].write_text(definition);(root/'fail').touch()
 try:
 run('systemd-analyze','verify',*[str(p) for p in paths.values()])
 run('systemctl','daemon-reload');run('systemctl','start',timer)
 wait_for(lambda:value(service,'ActiveState')=='failed')
 assert value(service,'ExecMainStatus')=='7' and value(timer,'ActiveState')=='active'
 run('systemctl','stop',timer)
 print('timer active while its first job failed with exit status 7')
 (root/'fail').unlink();run('systemctl','reset-failed',service)
 paths[timer].write_text(definition.replace('OnUnitInactiveSec=10s','OnUnitInactiveSec=250ms'))
 run('systemctl','daemon-reload');run('systemctl','start',timer)
 wait_for(lambda:(root/'start-2.json').exists())
 time.sleep(0.65)
 assert len(list(root.glob('start-*.json')))==2
 (root/'release-2').touch();wait_for(lambda:(root/'start-3.json').exists())
 gap=json.loads((root/'start-3.json').read_text())-json.loads((root/'end-2.json').read_text())
 assert gap>=0.25,gap
 print('no overlapping launch; next job starts at least 250ms after prior deactivation')
 pid=value(service,'MainPID');assert pid!='0'
 run('systemctl','stop',timer)
 assert value(service,'MainPID')==pid and value(service,'ActiveState')=='activating'
 print('stopping timer leaves its already running oneshot alive')
 (root/'release-3').touch();wait_for(lambda:value(service,'ActiveState')=='inactive')
 time.sleep(0.4)
 assert len(list(root.glob('start-*.json')))==3 and value(service,'Result')=='success'
 print('released job completes; stopped timer schedules no further run')
 finally:
 run('systemctl','stop',timer,service)
 for unit,path in paths.items():
 if value(unit,'ActiveState')=='failed':run('systemctl','reset-failed',unit)
 path.unlink()
 run('systemctl','daemon-reload')

The first timer activation launches a worker that deliberately exits with status seven. The service becomes failed while the timer remains active. Its next completion-relative interval is initially ten seconds, allowing the script to inspect this state and stop the timer before another attempt. The output confirms a scheduler can be functioning while the scheduled work is failing.

After correcting the fixture’s failure condition, the script clears the failed state, changes the interval to 250 milliseconds, and starts scheduling again. The second worker waits at its release gate. Even after more than two nominal intervals pass, no third worker has started. OnUnitInactiveSec is measured from deactivation, so unfinished work has not yet supplied that reference event.

Releasing the second worker lets it complete. The third worker then starts at least one configured interval after the recorded completion marker; actual unit deactivation occurs after that marker. The assertion verifies the completion-relative relationship without requiring a fragile exact launch timestamp. AccuracySec=1ms reduces coalescing in this short exercise, but it is not a real-time execution guarantee.

The script stops the timer while the third worker is still waiting. Its PID remains unchanged and the oneshot remains activating. Only releasing that worker lets the service complete successfully. After another interval has passed, the invocation count is unchanged, confirming that the stopped timer no longer schedules work. Both runtime unit files and the temporary worker are removed afterward.

Migrate the execution contract too

When moving a cron job, inspect more than its time expression. Use the intended working directory, explicit command paths, required environment, suitable user, and an appropriate runtime limit. systemd command lines are not implicitly shell programs. A pipeline, redirection, or shell expansion needs an intentional shell invocation or a separate script whose behavior is tested.

Installation and activation are also separate. Starting a runtime timer once does not enable it at boot, and enabling the service itself can introduce an unintended startup run. Install and enable the timer according to the deployment’s policy, then verify the loaded definitions and next activation. Keep the service independently runnable for controlled diagnosis.

Choose how failures should be retried and reported instead of assuming scheduling provides that policy. A later timer activation may try again, while service restart settings introduce their own rules and limits. Make repeated side effects safe and preserve a last-success signal tied to the actual result. The operational objective is completed useful work within its required window, not merely a timer that remains active.

Sources

systemd 252 systemd.timer, target selection, active-target behavior, and completion-relative intervals, checked 2026-09-12.
systemd 252 systemd.service, oneshot lifecycle, retained state, execution failure, and timeouts, checked 2026-09-12.
systemd 252 systemctl, starting, stopping, enabling, and inspecting timer and service units, checked 2026-09-12.

An Ansible playbook declares the state you want, not the steps

Sun, 17 Jan 2027 00:00:00 +0000

An Ansible playbook records the state a machine should have. That distinction matters after a partial deploy: the next run observes what is really present and repairs only the missing pieces, rather than blindly replaying every command from the beginning. A playbook is an executable contract between the inventory, the modules, and the operator who reviews the change.

Say the end, not each step

An imperative shell script says “run these commands in this order.” A playbook instead says “this package is installed, this file has these bytes, and this service is enabled and running.” A module compares the current host with that target before it acts. Matching tasks report ok; only a mismatch reports changed. The declaration can therefore be applied to a fresh host, a partly configured host, or one edited by hand.

The target is not vague intent. Module arguments are the contract: state: present, an exact template, an owner and mode, or a service’s enabled/running pair. Keep shared values in variables and roles, but make the final value visible to a reviewer. A hidden default or a command that mutates state outside the module makes the contract harder to reason about.

Modules make reruns converge

Most built-in modules are designed to be idempotent. ansible.builtin.copy compares content and metadata before replacing a file. ansible.builtin.package asks the package manager for the requested state. ansible.builtin.service changes enablement or running state only when necessary. A second invocation should be a quiet confirmation, not another restart, duplicate line, or appended record.

Some actions are not naturally idempotent. A raw command that creates a user or calls an API repeats its side effect unless guarded. Prefer a purpose-built module; otherwise use when with a registered read result, or precise creates/removes conditions. changed_when controls reporting and handler notification, not execution. Setting it to false cannot prevent a repeated mutation.

Inventory is the map

The playbook describes outcomes, while inventory selects hosts and variables select context. ansible_user, an SSH agent, or Vault can supply credentials without putting secrets in the repository. Group variables let one declaration cover a fleet while preserving host-specific values such as a port or region. Run with an explicit inventory in CI so “localhost” does not accidentally mean the runner itself.

Check mode (--check) asks modules to predict changes without applying them, and --diff shows content differences for modules that support it. Prediction is not proof: modules calling external systems may not implement check mode, and secrets should not be printed in a diff. Treat a check-mode report as review material, then run normal mode with the same inventory and keep the result with the deployment record.

When it breaks

A playbook can drift when a task shells out, a module has incomplete check-mode support, or an unpinned package resolves a different version tomorrow. Serial execution can leave half a group changed when a later host fails; use serial, handlers, and a rollback plan where the service cannot tolerate that window. A task depending on a fact gathered earlier must state that dependency instead of relying on incidental order.

Scope confusion is another failure: an inventory containing localhost, with a local connection targets the runner, while all selects every matching inventory host. Syntax checking cannot detect the wrong target. Review inventory and --list-hosts; inspect effective variables, including command-line overrides. Protect secrets with no_log and diff: false. Database migrations and cloud APIs still need their own transaction and retry semantics.

Check it quickly

Save the YAML as site.yml and Python as verify_ansible.py in one directory, then run these commands. The local connection modifies only a temporary file.

python3 -m venv .venv-ansible
.venv-ansible/bin/python -m pip install ansible-core==2.21.4
.venv-ansible/bin/python verify_ansible.py

- name: Converge a disposable local file
 hosts: localhost
 connection: local
 gather_facts: false
 tasks:
 - name: Declare the banner content and mode
 ansible.builtin.copy:
 dest: "{{ fixture }}"
 content: "managed by ansible\n"
 mode: "0644"

import json
import os
from pathlib import Path
import re
import stat
import subprocess
import sys
import tempfile

playbook = Path(__file__).with_name("site.yml").resolve()
ansible = Path(sys.executable).parent / "ansible-playbook"
print(subprocess.check_output([str(ansible), "--version"], text=True).splitlines()[0])
with tempfile.TemporaryDirectory(prefix="ansible-state-") as directory:
 work = Path(directory)
 banner = work / "banner"
 config = work / "ansible.cfg"
 config.write_text("[defaults]\nstdout_callback=default\n", encoding="utf-8")
 env = dict(os.environ, ANSIBLE_CONFIG=str(config), ANSIBLE_NOCOLOR="1",
 ANSIBLE_LOCAL_TEMP=str(work / "local-tmp"))
 variables = {"fixture": str(banner), "ansible_python_interpreter": sys.executable,
 "ansible_remote_tmp": str(work / "module-tmp")}
 command = [str(ansible), "-i", "localhost,", str(playbook),
 "--extra-vars", json.dumps(variables)]

 def run(*options):
 result = subprocess.run(command + list(options), cwd=work, env=env,
 check=True, text=True, capture_output=True, timeout=60)
 print(result.stdout)
 return result.stdout

 def changed(expected, *options):
 output = run(*options)
 recap = re.search(r"localhost\s+:\s+ok=(\d+)\s+changed=(\d+)\s+"
 r"unreachable=(\d+)\s+failed=(\d+)", output)
 assert recap and tuple(map(int, recap.groups())) == (1, expected, 0, 0)
 return output

 run("--syntax-check")
 assert "managed by ansible" in changed(1, "--check", "--diff")
 assert not banner.exists(), "check mode created the absent file"
 changed(1)
 assert banner.read_bytes() == b"managed by ansible\n"
 assert stat.S_IMODE(banner.stat().st_mode) == 0o644
 changed(0)
 banner.write_bytes(b"manual drift\n")
 before = (banner.read_bytes(), banner.stat().st_mtime_ns, banner.stat().st_mode)
 diff = changed(1, "--check", "--diff")
 assert "manual drift" in diff and "managed by ansible" in diff
 assert before == (banner.read_bytes(), banner.stat().st_mtime_ns, banner.stat().st_mode)
 changed(1)
 assert banner.read_bytes() == b"managed by ansible\n"
 changed(0)
print("PASS: syntax, absent-file preview, convergence, drift preview, and repair")

With Ansible Core 2.21.4, syntax checking succeeds. Check mode reports changed=1 without creating the absent file; normal execution creates it and a rerun reports changed=0. After manual drift, the preview shows both versions without changing bytes, mode, or modification time. Normal execution repairs the file and converges again. These assertions test this copy task; other modules and upgrades need their own checks.

Sources

Ansible playbooks introduction, playbook structure, tasks, and desired state, checked 2026-09-09.
Ansible check mode and diff mode, prediction and change previews, checked 2026-09-09.
Ansible error handling, changed reporting and handler notification, checked 2026-09-10.

Choose the clock and catch-up rule for a systemd timer

Fri, 20 Nov 2026 00:00:00 +0000

A schedule must specify what time means. “At nine each morning in New York” refers to a civil-time calendar. “A quarter hour after this job finishes” refers to an elapsed interval and an event. systemd timers support both, but the two clocks and reference points have different behavior around time-zone changes, suspension, and periods when the timer is inactive.

When it breaks

OnCalendar follows wall-clock calendar expressions. Omitting a time zone generally uses the system’s local zone, which may differ between machines. A daily local time can move relative to UTC when daylight-saving rules change. Preview the intended expression with a fixed reference date instead of assuming that today’s UTC offset applies throughout the year.

Monotonic options name their reference event explicitly. OnActiveSec refers to activation of the timer itself, OnStartupSec to the service manager’s startup, and OnUnitInactiveSec to the target unit’s deactivation. These are not interchangeable ways to say “every few minutes.” Ordinary monotonic timers pause with system suspension; WakeSystem changes clock selection and depends on platform support.

Persistent=yes affects calendar timers. It records trigger timing and requests a catch-up activation when the timer becomes active after missing at least one event. It does not replay every missed occurrence or certify successful job completion. The catch-up can also be delayed by configured randomization. A trigger history is not a durable queue of all business work that should have happened.

Check it quickly

Run this Python script as an administrator in a disposable Linux system with systemd as PID 1 and the time-zone database installed. The recorded run used Debian 12 with systemd 252.39. It first previews a calendar expression, then installs two unique timer/service pairs using the same three near-future calendar events. It never adjusts the machine’s clock:

from pathlib import Path
from tempfile import TemporaryDirectory
from datetime import datetime,timezone
import subprocess,time,uuid

name='xbcatm-calendar-'+uuid.uuid4().hex[:8]
names=[name,name+'-plain'];paths=[]
def run(*args):
 result=subprocess.run(args,text=True,capture_output=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
preview=run('systemd-analyze','calendar','--base-time=2026-03-07 00:00:00 UTC',
 '--iterations=3','*-*-* 09:00:00 America/New_York')
for expected in ('2026-03-07 14:00:00 UTC','2026-03-08 13:00:00 UTC','2026-03-09 13:00:00 UTC'):
 assert expected in preview,preview
print('09:00 New York: 14:00 UTC before spring DST, 13:00 UTC after it')
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py'
 worker.write_text("from pathlib import Path\nimport sys\np=Path(sys.argv[1])\nwith p.open('a') as output:output.write('run\\n')\n")
 now=int(time.time());events=[now+3,now+4,now+5]
 calendar=''.join('OnCalendar='+datetime.fromtimestamp(t,timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')+'\n' for t in events)
 try:
 for index,prefix in enumerate(names):
 service=Path('/run/systemd/system')/(prefix+'.service');timer=service.with_suffix('.timer')
 assert not service.exists() and not timer.exists()
 service.write_text(f'[Service]\nType=oneshot\nExecStart=/usr/bin/python3 {worker} {root}/{index}.log\n');paths.append(service)
 timer.write_text('[Timer]\n'+calendar+'AccuracySec=1ms\nRandomizedDelaySec=0\nPersistent='+('yes' if index==0 else 'no')+'\n');paths.append(timer)
 run('systemd-analyze','verify',*[str(p) for p in paths]);run('systemctl','daemon-reload')
 run('systemctl','start',name+'.timer');run('systemctl','stop',name+'.timer')
 assert not (root/'0.log').exists() and time.time()<events[0]
 while time.time()<events[-1]+0.15:time.sleep(0.05)
 run('systemctl','start',name+'.timer',name+'-plain.timer')
 deadline=time.monotonic()+3
 while not (root/'0.log').exists():
 assert time.monotonic()<deadline,'catch-up did not run';time.sleep(0.02)
 time.sleep(0.15)
 assert (root/'0.log').read_text()=='run\n' and not (root/'1.log').exists()
 print('three elapsed calendar events: persistent timer catches up once; plain timer does not')
 run('systemctl','stop',name+'.timer');run('systemctl','start',name+'.timer')
 time.sleep(0.2);assert (root/'0.log').read_text()=='run\n'
 print('reactivating with no newly missed event does not replay the job')
 print('timer inactivity was tested without changing the system clock or rebooting')
 finally:
 if paths:
 run('systemctl','stop',*[p.name for p in paths])
 run('systemctl','clean','--what=state',name+'.timer')
 for path in paths:path.unlink(missing_ok=True)
 run('systemctl','daemon-reload')

The preview asks for nine in the morning in New York, starting from a fixed date before the March 2026 daylight-saving transition. The first occurrence is 14:00 UTC; the following two are 13:00 UTC. The local clock time remains nine. This is a schedule interpretation check, not a request to change the host’s time zone or travel to those dates.

The persistent timer is started and stopped before the first laboratory event. Starting it establishes its timestamp state. While it is inactive, all three specified events pass. On reactivation it launches its service exactly once, producing one line in the result file. The nonpersistent timer, activated after the same events, produces no result file.

This experiment observes an actual inactive period rather than editing a timestamp file to impersonate a missed run. It demonstrates the catch-up decision without rebooting the machine. Three elapsed events become one activation; an application that needs to process three distinct accounting periods must discover and track those periods itself.

Stopping and starting the persistent timer again produces no extra line because no new event was missed. Cleanup stops the runtime units and removes the persistent timer’s state through systemctl clean --what=state. The nonpersistent timer has no matching persistent state to clean. Configuration files and the harmless result directory are then removed.

AccuracySec=1ms and zero randomized delay keep the short experiment interpretable. Neither setting promises exact process-start timing under arbitrary load. The code waits for observable completion within a bounded interval and asserts the resulting invocation count, rather than equating a predicted deadline with actual execution.

Decide which missed work matters

For a backup, one fresh snapshot after downtime may satisfy the recovery policy. For billing, each missing period might require separate reconciliation. Put that distinction into application state and idempotency rules. A service already active when a timer elapses is not automatically duplicated, and retained active oneshot state can prevent the command from running again.

Use the timer’s loaded configuration and next/last trigger properties to inspect scheduling, and the target service’s result to inspect execution. Calendar ordering after time-related targets does not by itself prove that a remote time source has synchronized the clock; the actual time-service configuration must supply that promise when it is required.

Test representative dates and lifecycle transitions before relying on a schedule. Include time-zone boundaries, timer disablement, and the task’s expected duration. Choose a monotonic reference for elapsed-delay requirements and a named calendar zone for civil-time requirements. Preserve evidence of completed useful work separately from evidence that the manager issued a trigger.

Sources

systemd 252 systemd.timer, clock references, calendar persistence, accuracy, and randomization, checked 2026-09-12.
systemd 252 systemd.time, calendar expressions and named time-zone syntax, checked 2026-09-12.
systemd 252 systemd-analyze, calendar previews from a fixed base time and multiple iterations, checked 2026-09-12.
systemd 252 systemctl, timer activation and removal of persistent timer state, checked 2026-09-12.

Scheduled jobs need to be safe to rerun

Sat, 15 Aug 2026 00:00:00 +0000

A scheduler starts attempts, while the business usually cares about completing a logical task. One daily credit may be attempted several times after an error, a restart, or an operator retry. Giving every attempt a new identifier makes those duplicates indistinguishable from new work. Use a key that identifies the intended task, such as its name and nominal scheduled time, and define what a repeated key means before changing business state.

Scheduling and execution are separate

Do not assume every scheduler catches up missed work. Ordinary cron behavior varies by implementation and does not provide a universal catch-up guarantee. A systemd calendar timer with Persistent=true can trigger after an inactive interval when at least one activation was missed; that does not mean replaying every missed occurrence. Kubernetes documents that a CronJob may create duplicate Jobs or miss an occurrence under some circumstances.

Choose the required business rule explicitly: skip stale work, process each missing period, or reconcile the current state. Use the nominal logical period in the deduplication key, with an unambiguous time zone. An actual start timestamp changes across retries and therefore cannot identify the same intended occurrence. Keep an attempt identifier separately for diagnostics without confusing it with business identity.

When it breaks

A check followed by an unrelated update has a race. Two workers can both observe that a key is absent and both apply the effect. A process lock may prevent overlap on one machine but does not make a business update atomic with a marker file. If the process fails after the update and before writing the marker, a retry repeats the effect. Writing the marker first can instead lose the work.

When both state and completion record live in one transactional database, commit them together under a uniqueness constraint. A failure before commit should preserve neither change; a retry after commit should see the completed logical key. In SQLite, BEGIN IMMEDIATE acquires the write transaction early, and competing writers may wait or fail with a busy error. A timeout bounds that wait; it does not make every workload contention-free.

This boundary does not cover an external email, payment, or HTTP call. A local rollback cannot undo an effect already accepted elsewhere. Such workflows need an appropriate remote idempotency contract or a durable delivery design with deduplication at the receiver. Reusing a key with a different payload also requires a defined rejection policy; the example below always applies the same fixed credit.

Check it quickly

Save the code as idempotent-jobs.py and run python3 idempotent-jobs.py. It uses a temporary SQLite database and eight worker threads, each with its own database connection. The business effect adds ten units to a balance. A non-null primary key records which logical daily credit has completed within the same transaction.

from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from tempfile import TemporaryDirectory
from threading import Barrier
import json
import sqlite3

with TemporaryDirectory() as directory:
 database = str(Path(directory) / 'jobs.db')
 with sqlite3.connect(database) as connection:
 connection.executescript('''
CREATE TABLE balance (id INTEGER PRIMARY KEY, units INTEGER NOT NULL);
INSERT INTO balance VALUES (1, 0);
CREATE TABLE completed (logical_key TEXT PRIMARY KEY NOT NULL);
''')

 def run(key, fail=False, gate=None):
 connection = sqlite3.connect(database, timeout=10, isolation_level=None)
 try:
 if gate:
 gate.wait(timeout=5)
 connection.execute('BEGIN IMMEDIATE')
 inserted = connection.execute(
 'INSERT INTO completed VALUES (?) ON CONFLICT(logical_key) DO NOTHING', (key,))
 if not inserted.rowcount:
 connection.execute('COMMIT')
 return 'duplicate'
 connection.execute('UPDATE balance SET units = units + 10 WHERE id = 1')
 if fail:
 raise RuntimeError('injected failure before commit')
 connection.execute('COMMIT')
 return 'applied'
 except Exception:
 if connection.in_transaction:
 connection.execute('ROLLBACK')
 raise
 finally:
 connection.close()

 key = 'daily-credit:2026-09-13T00:00:00Z'
 try:
 run(key, fail=True)
 except RuntimeError as error:
 assert str(error) == 'injected failure before commit'
 with sqlite3.connect(database) as connection:
 assert connection.execute('SELECT units FROM balance').fetchone() == (0,)
 assert connection.execute('SELECT count(*) FROM completed').fetchone() == (0,)
 gate = Barrier(8)
 with ThreadPoolExecutor(max_workers=8) as pool:
 results = list(pool.map(lambda _: run(key, gate=gate), range(8)))
 assert results.count('applied') == 1 and results.count('duplicate') == 7
 assert run(key) == 'duplicate'
 assert run('daily-credit:2026-09-14T00:00:00Z') == 'applied'
 with sqlite3.connect(database) as connection:
 units = connection.execute('SELECT units FROM balance').fetchone()[0]
 completed = connection.execute('SELECT count(*) FROM completed').fetchone()[0]
 assert (units, completed) == (20, 2)
 print(json.dumps({'rolled_back_attempt_left_no_state': True,
 'concurrent_applied': results.count('applied'),
 'concurrent_duplicates': results.count('duplicate'),
 'final_units': units, 'completed_logical_jobs': completed}))

The first attempt intentionally raises after both inserts and updates but before commit. The script confirms that rollback leaves zero units and no completed key. Eight workers then start together for the same logical occurrence. Exactly one reports applied; seven report duplicate. Their unique-key decision and balance mutation are serialized by the actual database, not a mocked lock.

A further replay after successful commit remains a duplicate. A different nominal date is a new logical task and applies another ten units. The final result is twenty units and two completed keys. This verifies rollback and replay within one SQLite database, not scheduler delivery guarantees, process-kill recovery, or external exactly-once execution.

Keep failure and completion visible

Record the logical key, attempt outcome, duration, and last successful business period. Distinguish a harmless duplicate from a database error; catching every exception and labeling it completed would hide lost work. Define how long completion records must remain available before deleting them, since a retry after deletion can apply again. Operational evidence should tell an operator whether the intended period completed, rather than merely whether a timer launched a process.

Sources

SQLite: transactions, writer serialization, commit, and rollback, checked 2026-09-13.
SQLite: UPSERT, targeted handling of a uniqueness conflict, checked 2026-09-13.
systemd.timer, the scope of persistent calendar catch-up, checked 2026-09-13.
Kubernetes: CronJob, approximate scheduling and duplicate or missed jobs, checked 2026-09-13.

Automation · XB Field Notes

Cron repeats on the calendar

Read fields as calendar selections

When it breaks

Check it quickly

Sources

GitHub Actions workflows gate the build

YAML is the contract

Jobs and steps run in order

Secrets stay close to the runner

When it breaks

Check it quickly

Further reading

Sources

A systemd timer schedules a service, not its success

When it breaks

Check it quickly

Migrate the execution contract too

Further reading

Sources

An Ansible playbook declares the state you want, not the steps

Say the end, not each step

Modules make reruns converge

Inventory is the map

When it breaks

Check it quickly

Further reading

Sources

Choose the clock and catch-up rule for a systemd timer

When it breaks

Check it quickly

Decide which missed work matters

Further reading

Sources

Scheduled jobs need to be safe to rerun

Scheduling and execution are separate

When it breaks

Check it quickly

Keep failure and completion visible

Further reading

Sources