SQLite is a file that is also a database

Tue, 20 Oct 2026 00:00:00 +0000

SQLite is a database engine linked into the application rather than a separate database server. Its main database file stores schema, tables, and indexes in a portable format. That makes it useful for local application state, embedded devices, and services whose write workload fits its concurrency model. The file is still governed by transactions and locking; ordinary file-copy intuition is not enough to operate it safely.

When it breaks

The main file is not always the whole current database state. WAL mode can leave committed changes in a separate -wal file while the main file remains at an earlier checkpoint. Shared-memory coordination also commonly uses a -shm file. Copying only the main file while connections remain active can therefore omit committed data. Copying several live files one after another does not automatically produce a consistent snapshot either.

WAL mode improves the overlap of reads and writes, but it still permits only one writer at a time. A long write transaction can make another writer receive SQLITE_BUSY. Waiting longer changes patience, not the number of simultaneous writers. Keep transactions short and choose a client/server engine when the measured write concurrency or remote-access requirements exceed this architecture’s fit.

Readers have snapshot boundaries too. A reader can see a committed older value while another connection holds an uncommitted update. That is transaction isolation, not evidence that SQLite lost the update. Applications should inspect transaction state after errors and retry the appropriate transaction only when the operation’s semantics permit it.

Check it quickly

Save the following as sqlite_lab.py and run it with ordinary python3, keeping assertions enabled. It uses only the standard library and temporary local files. The recorded run used Python 3.14.4 with SQLite 3.53.0; the engine version is available as sqlite3.sqlite_version and need not match the Python version.

import shutil
import sqlite3
import tempfile
from contextlib import closing
from pathlib import Path

with tempfile.TemporaryDirectory(prefix="xbcatm-sqlite-") as directory:
 root = Path(directory)
 database = root / "source.db"
 with closing(sqlite3.connect(database, isolation_level=None, timeout=0.05)) as first, \
 closing(sqlite3.connect(database, isolation_level=None, timeout=0.05)) as second:
 assert first.execute("PRAGMA journal_mode=WAL").fetchone() == ("wal",)
 first.execute("PRAGMA wal_autocheckpoint=0")
 second.execute("PRAGMA wal_autocheckpoint=0")
 first.execute("CREATE TABLE marker (id INTEGER PRIMARY KEY, value INTEGER)")
 first.execute("INSERT INTO marker VALUES (1,10)")
 assert first.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone() == (0, 0, 0)
 first.execute("BEGIN IMMEDIATE")
 first.execute("UPDATE marker SET value=99 WHERE id=1")
 assert second.execute("SELECT value FROM marker").fetchone() == (10,)
 try:
 second.execute("BEGIN IMMEDIATE")
 except sqlite3.OperationalError as error:
 assert error.sqlite_errorcode == sqlite3.SQLITE_BUSY
 assert not second.in_transaction
 print("second_writer", error.sqlite_errorname)
 else:
 raise AssertionError("second writer unexpectedly acquired the lock")
 first.execute("ROLLBACK")
 second.execute("BEGIN IMMEDIATE")
 second.execute("UPDATE marker SET value=20 WHERE id=1")
 second.execute("COMMIT")
 assert first.execute("SELECT value FROM marker").fetchone() == (20,)
 # Deliberately unsafe copy, only as a negative control in this quiet fixture.
 shutil.copyfile(database, root / "main-only.db")
 with closing(sqlite3.connect(root / "main-only.db")) as incomplete:
 assert incomplete.execute("PRAGMA integrity_check").fetchone() == ("ok",)
 assert incomplete.execute("SELECT value FROM marker").fetchone() == (10,)
 print("main_only_copy", "integrity ok, stale value 10")
 with closing(sqlite3.connect(root / "backup.db")) as backup:
 first.backup(backup)
 assert backup.execute("PRAGMA integrity_check").fetchone() == ("ok",)
 assert backup.execute("SELECT value FROM marker").fetchone() == (20,)
 first.execute("UPDATE marker SET value=30 WHERE id=1")
 assert backup.execute("SELECT value FROM marker").fetchone() == (20,)
 print("online_backup", "value 20, source now 30")

The fixture establishes a checkpointed value of ten, then starts a write transaction that changes it to ninety-nine without committing. Another connection still reads ten. Its attempt to become a writer must fail with SQLITE_BUSY, and it must remain outside a transaction. The script then rolls back the first writer, acquires the write position from the second connection, and commits twenty successfully.

Automatic checkpointing is disabled only in this fixture so the difference between the main file and committed WAL remains visible. The deliberately incomplete copy contains only the main file. It passes integrity_check yet returns ten, proving that structural integrity does not establish the requested freshness. This negative control uses a quiet, known schedule; copying live database files is not the backup procedure being recommended.

The proper copy uses Connection.backup(), which invokes SQLite’s online backup API. The destination passes integrity checking and contains twenty. After the source changes again to thirty, the backup still contains twenty: it is a consistent captured state, not a continuously synchronized replica. All connections close and the private directory is removed after the exercise.

Fit the engine to the access pattern

Use the online backup API or another documented SQLite backup method when the source remains active. Verify application facts as well as database structure, and record the captured state or backup time needed by recovery policy. An apparently valid file can still be the wrong recovery point, as the main-only copy demonstrates.

Keep WAL databases on storage that provides the locking and shared-memory behavior SQLite requires. Ordinary WAL operation expects participating processes on the same host; a shared network path is not a substitute for a database protocol. File permissions and application boundaries control access because there is no separate SQLite server maintaining database login roles.

Measure writer occupancy, busy errors, transaction duration, and backup behavior before deciding the engine is too small or sufficiently capable. Database size alone is not the concurrency limit. A substantial mostly read workload can fit well, while a small database with many competing long writes can be a poor match. Choose from those observed requirements rather than from the file’s simplicity.

Sources

SQLite appropriate uses, embedded use cases and client/server tradeoffs, checked 2026-09-12.
SQLite write-ahead logging, sidecar files, reader/writer concurrency, and same-host requirements, checked 2026-09-12.
SQLite online backup API, consistent backups of active databases, checked 2026-09-12.
Python sqlite3 documentation, connection backup, transaction handling, and SQLite error attributes, checked 2026-09-12.

Sqlite · XB Field Notes

SQLite is a file that is also a database

When it breaks

Check it quickly

Fit the engine to the access pattern

Sources