A scheduler starts attempts, while the business usually cares about completing a logical task. One daily credit may be attempted several times after an error, a restart, or an operator retry. Giving every attempt a new identifier makes those duplicates indistinguishable from new work. Use a key that identifies the intended task, such as its name and nominal scheduled time, and define what a repeated key means before changing business state.

Scheduling and execution are separate

Do not assume every scheduler catches up missed work. Ordinary cron behavior varies by implementation and does not provide a universal catch-up guarantee. A systemd calendar timer with Persistent=true can trigger after an inactive interval when at least one activation was missed; that does not mean replaying every missed occurrence. Kubernetes documents that a CronJob may create duplicate Jobs or miss an occurrence under some circumstances.

Choose the required business rule explicitly: skip stale work, process each missing period, or reconcile the current state. Use the nominal logical period in the deduplication key, with an unambiguous time zone. An actual start timestamp changes across retries and therefore cannot identify the same intended occurrence. Keep an attempt identifier separately for diagnostics without confusing it with business identity.

When it breaks

A check followed by an unrelated update has a race. Two workers can both observe that a key is absent and both apply the effect. A process lock may prevent overlap on one machine but does not make a business update atomic with a marker file. If the process fails after the update and before writing the marker, a retry repeats the effect. Writing the marker first can instead lose the work.

When both state and completion record live in one transactional database, commit them together under a uniqueness constraint. A failure before commit should preserve neither change; a retry after commit should see the completed logical key. In SQLite, BEGIN IMMEDIATE acquires the write transaction early, and competing writers may wait or fail with a busy error. A timeout bounds that wait; it does not make every workload contention-free.

This boundary does not cover an external email, payment, or HTTP call. A local rollback cannot undo an effect already accepted elsewhere. Such workflows need an appropriate remote idempotency contract or a durable delivery design with deduplication at the receiver. Reusing a key with a different payload also requires a defined rejection policy; the example below always applies the same fixed credit.

Check it quickly

Save the code as idempotent-jobs.py and run python3 idempotent-jobs.py. It uses a temporary SQLite database and eight worker threads, each with its own database connection. The business effect adds ten units to a balance. A non-null primary key records which logical daily credit has completed within the same transaction.

from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from tempfile import TemporaryDirectory
from threading import Barrier
import json
import sqlite3

with TemporaryDirectory() as directory:
    database = str(Path(directory) / 'jobs.db')
    with sqlite3.connect(database) as connection:
        connection.executescript('''
CREATE TABLE balance (id INTEGER PRIMARY KEY, units INTEGER NOT NULL);
INSERT INTO balance VALUES (1, 0);
CREATE TABLE completed (logical_key TEXT PRIMARY KEY NOT NULL);
''')

    def run(key, fail=False, gate=None):
        connection = sqlite3.connect(database, timeout=10, isolation_level=None)
        try:
            if gate:
                gate.wait(timeout=5)
            connection.execute('BEGIN IMMEDIATE')
            inserted = connection.execute(
                'INSERT INTO completed VALUES (?) ON CONFLICT(logical_key) DO NOTHING', (key,))
            if not inserted.rowcount:
                connection.execute('COMMIT')
                return 'duplicate'
            connection.execute('UPDATE balance SET units = units + 10 WHERE id = 1')
            if fail:
                raise RuntimeError('injected failure before commit')
            connection.execute('COMMIT')
            return 'applied'
        except Exception:
            if connection.in_transaction:
                connection.execute('ROLLBACK')
            raise
        finally:
            connection.close()

    key = 'daily-credit:2026-09-13T00:00:00Z'
    try:
        run(key, fail=True)
    except RuntimeError as error:
        assert str(error) == 'injected failure before commit'
    with sqlite3.connect(database) as connection:
        assert connection.execute('SELECT units FROM balance').fetchone() == (0,)
        assert connection.execute('SELECT count(*) FROM completed').fetchone() == (0,)
    gate = Barrier(8)
    with ThreadPoolExecutor(max_workers=8) as pool:
        results = list(pool.map(lambda _: run(key, gate=gate), range(8)))
    assert results.count('applied') == 1 and results.count('duplicate') == 7
    assert run(key) == 'duplicate'
    assert run('daily-credit:2026-09-14T00:00:00Z') == 'applied'
    with sqlite3.connect(database) as connection:
        units = connection.execute('SELECT units FROM balance').fetchone()[0]
        completed = connection.execute('SELECT count(*) FROM completed').fetchone()[0]
    assert (units, completed) == (20, 2)
    print(json.dumps({'rolled_back_attempt_left_no_state': True,
                      'concurrent_applied': results.count('applied'),
                      'concurrent_duplicates': results.count('duplicate'),
                      'final_units': units, 'completed_logical_jobs': completed}))

The first attempt intentionally raises after both inserts and updates but before commit. The script confirms that rollback leaves zero units and no completed key. Eight workers then start together for the same logical occurrence. Exactly one reports applied; seven report duplicate. Their unique-key decision and balance mutation are serialized by the actual database, not a mocked lock.

A further replay after successful commit remains a duplicate. A different nominal date is a new logical task and applies another ten units. The final result is twenty units and two completed keys. This verifies rollback and replay within one SQLite database, not scheduler delivery guarantees, process-kill recovery, or external exactly-once execution.

Keep failure and completion visible

Record the logical key, attempt outcome, duration, and last successful business period. Distinguish a harmless duplicate from a database error; catching every exception and labeling it completed would hide lost work. Define how long completion records must remain available before deleting them, since a retry after deletion can apply again. Operational evidence should tell an operator whether the intended period completed, rather than merely whether a timer launched a process.

Further reading

Sources

  • SQLite: transactions, writer serialization, commit, and rollback, checked 2026-09-13.
  • SQLite: UPSERT, targeted handling of a uniqueness conflict, checked 2026-09-13.
  • systemd.timer, the scope of persistent calendar catch-up, checked 2026-09-13.
  • Kubernetes: CronJob, approximate scheduling and duplicate or missed jobs, checked 2026-09-13.