Change windows fit people, not servers

Thu, 29 Jul 2027 00:00:00 +0000

A change that races to zero traffic but lands when nobody is watching trades one risk for another. The best window is one where the operators responsible can actually attend.

Traffic is only part of the math

Deploying at the absolute quietest hour minimizes how many users feel the change, but it collides with the people problem: the operator is most likely asleep, unreachable, or already covering another incident. If the rollout breaks at 3 a.m. and nobody is watching, a silent failure can run for hours before a human reads it. Least-traffic is a good starting estimate, but it is a traffic answer, not a risk answer. Balance reduced exposure against the ability to react, and prefer a window where you can actually observe the result.

Watch the human clock

The window has to fit the humans, not just the chart. Consider who must notice a broken deploy: the SRE on call, the feature owner, the support team, and schedule their attention into the slot. Also watch what else is scheduled — a parallel release, a job run, an adjacent change — that could tangle with yours and turn a clean rollout into a confusing blame-fest. A slot where the people who must read the result are awake, reachable, and not already overloaded beats a quieter one where nobody is watching.

Let the window be a habit

Turn deploys into a scheduled, rehearsed ritual rather than a reaction. A fixed change window — same weekday, same hour, with support present and a rollback plan standing by — becomes predictable, so the team learns to watch for it and reacts fast when it fails. Automate the repetitive parts (build, smoke test, canary) around that cadence. Over time the calm, repeated routine is what trains both the service and the people: everyone expects changes, everyone knows what they look like, and the process stops being a fire drill.

When it breaks

An urgent fix arrives outside the window and there is no emergency lane. The team either waits while the incident burns, or someone pushes straight to production with no watcher and no rollback note. Without a documented break-glass path — who approves, who watches, what gets written down — every exception becomes a secret deploy. Keep the normal window narrow and rehearse a visible emergency procedure alongside it.

A second failure is the stampede inside the window. Four services land in the same two hours, their signals blend together, and when the error rate climbs nobody can name the culprit. Rollback becomes a negotiation instead of a button press. Allow one risky change per slot, stagger the queue, and require every entry to name its owner, signal, and way back.

A third failure is the cross-timezone misread. The window says Wednesday 10:00 in Berlin, but a teammate in Los Angeles reads it as local time and ships while responders sleep. Daylight saving quietly shifts the slot by another hour. Store every window with its zone, render it locally, and confirm a human is present before counting a slot as staffed.

Check it quickly

Save the following as change_window_check.py and run python3 change_window_check.py. It uses only the standard library, treats every window as timezone-aware, and decides between allow and needs-approval. The table holds two Wednesday morning slots, one in Berlin and one in New York, plus one freeze date. The bad samples are negative controls, not recommended deploys.

from datetime import datetime, time
from zoneinfo import ZoneInfo

WINDOWS = [
 {"name": "berlin-wed", "weekday": 2, "start": "09:00", "end": "11:00", "tz": "Europe/Berlin"},
 {"name": "newyork-wed", "weekday": 2, "start": "09:00", "end": "11:00", "tz": "America/New_York"},
]
FREEZE_DATES = frozenset({"2027-12-24"})


def _hhmm(value):
 hour, minute = map(int, value.split(":"))
 return time(hour, minute)


def decide(moment, windows=WINDOWS, freeze=FREEZE_DATES):
 if moment.tzinfo is None or moment.utcoffset() is None:
 return "needs-approval", "naive datetime carries no zone; attach a ZoneInfo first"
 if moment.date().isoformat() in freeze:
 return "needs-approval", f"freeze date {moment.date()}; emergency path only"
 for window in windows:
 local = moment.astimezone(ZoneInfo(window["tz"]))
 if local.weekday() == window["weekday"] and _hhmm(window["start"]) <= local.time() < _hhmm(window["end"]):
 return "allow", f"inside {window['name']} ({local:%Y-%m-%d %H:%M %Z})"
 return "needs-approval", f"outside all windows ({moment:%Y-%m-%d %H:%M %Z})"


def check(actual, expected, label):
 status, reason = actual
 assert status == expected, f"{label}: got {status} ({reason})"
 print(f"{label}: {status} -- {reason}")


def main():
 berlin = ZoneInfo("Europe/Berlin")
 newyork = ZoneInfo("America/New_York")
 good_berlin = datetime(2027, 8, 4, 9, 30, tzinfo=berlin)
 good_ny = datetime(2027, 8, 4, 9, 30, tzinfo=newyork)
 check(decide(good_berlin), "allow", "good/berlin-wed-morning")
 check(decide(good_ny), "allow", "good/newyork-wed-morning")
 # Same UTC instant, different local reading: 07:30 UTC is 09:30 in Berlin but 03:30 in New York.
 same_instant = datetime(2027, 8, 4, 7, 30, tzinfo=ZoneInfo("UTC"))
 check(decide(same_instant), "allow", "good/same-instant-via-berlin")
 evening = datetime(2027, 8, 4, 22, 10, tzinfo=berlin)
 check(decide(evening), "needs-approval", "bad/wednesday-night")
 wrong_day = datetime(2027, 8, 5, 9, 30, tzinfo=berlin)
 check(decide(wrong_day), "needs-approval", "bad/thursday-morning")
 frozen = datetime(2027, 12, 24, 9, 30, tzinfo=berlin)
 check(decide(frozen), "needs-approval", "bad/freeze-date")
 naive = datetime(2027, 8, 4, 9, 30)
 check(decide(naive), "needs-approval", "bad/naive-datetime")
 misread = datetime(2027, 8, 4, 9, 30, tzinfo=ZoneInfo("America/Los_Angeles"))
 check(decide(misread), "needs-approval", "bad/timezone-misread")
 print("PASS: 3 allows and 5 approvals; zones and freeze list behave as documented")


if __name__ == "__main__":
 main()

The script prints three allow lines, five needs-approval lines, and a final PASS line. The Berlin and New York mornings hit their own Wednesday slots, and the shared UTC instant passes via Berlin though it is night in New York. The evening, Thursday, freeze-date, naive, and misread samples all require approval with a reason.

These assertions passed with Python 3.11 on macOS. The fixture checks only the decision logic, not paging or automation. Before relying on it, replace the demo table with your real windows, then repeat the checks against the on-call roster.

Sources

Google SRE: release engineering, release as an engineered practice, checked 2027-07-27.
12Factor: build release run, separating build from release, checked 2027-07-27.
HashiCorp Terraform docs, change as documented infrastructure, checked 2027-07-27.

Human · XB Field Notes