Docs as code make claims reviewable

Mon, 29 Mar 2027 00:00:00 +0000

A guide can be stored beside the application and still describe a command that never worked. Docs as code provides a way to discover and review that mistake: plain source files, version history, ownership, a preview, and checks attached to the change. It does not make prose correct automatically. The valuable link is between a documented claim and evidence from the implementation that the reader will actually use.

Version the reader’s contract

Document the behavior of a specific release. A command copied from the development branch may require an option missing from the installed package. Keep version selection visible, and update the implementation and the affected instructions in the same review when practical. Reviewers should inspect changed defaults, required privileges, input files, expected output, and any cleanup a procedure requires. A successful page build checks only part of that contract.

Keep a single authoritative source for mechanical facts where the project already has one. An option table generated from a schema can reduce duplicate maintenance, but its generator may omit defaults, misinterpret types, or describe flags unavailable in the packaged executable. Review generator changes and compare useful examples with the built program. Generated reference material and a human explanation serve different purposes; neither guarantees the other’s accuracy.

When it breaks

An example copied into a separate test can pass while the displayed command drifts. Execute the example extracted from the document where the format permits it, or generate both from a shared tested input. Avoid teaching an undocumented abbreviation: a parser that currently accepts a shortened option may accept it only until another option makes that prefix ambiguous.

Working directories also matter. A guide can depend on a file already present on the author’s machine, a cached dependency, or an inherited credential. A clean fixture should create its own harmless inputs and state its prerequisites. Do not execute arbitrary contributed shell snippets with release credentials merely because they live in Markdown. Give documentation checks the permissions their examples require.

Links, spelling, and formatting are useful checks, but they cannot determine whether the procedure solves the reader’s problem. A real command can exit successfully while producing output in the wrong order or modifying the wrong file. Inspect the meaningful result, and keep a human review of assumptions, failure recovery, and the audience’s starting point.

Check it quickly

Save this code as docs-code.py and run python3 docs-code.py. It uses Python’s standard library and a temporary directory. The fixture creates a JSON input with keys in the order z, then a, and a short Markdown guide. Its initial fenced command claims that json.tool supports --sort-fields, which is intentionally incorrect.

from pathlib import Path
from tempfile import TemporaryDirectory
import difflib
import json
import re
import shlex
import subprocess
import sys

with TemporaryDirectory() as directory:
 root = Path(directory)
 (root / 'sample.json').write_text('{"z":2,"a":1}\n')
 doc = root / 'guide.md'
 old = '# Format JSON\n\n```sh\npython3 -m json.tool --sort-fields sample.json\n```\n'
 doc.write_text(old)

 def execute_document():
 command = re.search(r'```sh\n([^\n]+)\n```', doc.read_text())[1]
 args = shlex.split(command)
 assert args[:3] == ['python3', '-m', 'json.tool']
 return subprocess.run([sys.executable, *args[1:]], cwd=root,
 capture_output=True, text=True)

 failed = execute_document()
 assert failed.returncode == 2 and 'unrecognized arguments' in failed.stderr
 doc.write_text(old.replace('--sort-fields', '--sort-keys'))
 passed = execute_document()
 assert passed.returncode == 0
 assert list(json.loads(passed.stdout)) == ['a', 'z']
 delta = ''.join(difflib.unified_diff(old.splitlines(True), doc.read_text().splitlines(True),
 fromfile='before/guide.md', tofile='after/guide.md'))
 print(json.dumps({'broken_command_exit': failed.returncode,
 'corrected_command_exit': passed.returncode,
 'parsed_keys': list(json.loads(passed.stdout)), 'doc_diff': delta}))

The script extracts the displayed command, splits its arguments without a shell, and runs the current interpreter’s real JSON tool. The first invocation must exit with status two and report an unrecognized argument. The guide is then corrected to --sort-keys. The second invocation must succeed, and the parsed output must contain keys in the order a, then z.

The final JSON includes the documentation diff as well as both statuses. This verifies the exact example and its relevant result instead of treating an added test as evidence. The extractor deliberately handles one known, single-line command and checks its prefix; it is not a general Markdown execution engine. It does not establish that unrelated pages, links, or release packages have been checked.

Make publication traceable

Publish the rendered documentation from the reviewed revision and record which application release it describes. Check the preview for code wrapping, navigation, and language selection, because correct source can still become an unusable page. When a check fails, decide whether the implementation, the example, or the documented contract is wrong before changing either. The reviewable diff helps people make that decision; the repository itself does not enforce agreement between prose and behavior.

Sources

Write the Docs: Docs as Code, version control, review, and executable checks for documentation, checked 2026-09-13.
Python 3.14: json, the real json.tool command and sort-keys option, checked 2026-09-13.