Deploys are where engineering risk meets production. A change flipped to 100% at once converts any mistake into an all-or-nothing incident. A rollout by phases turns the same change into a series of small, reversible measurements.

Equal percentage is not real safety

Reaching a 100% cutover in one step is not speed, it is a bet that you can read a log before the whole fleet dies. Percentage as a number is meaningless on its own, because 50% of traffic to a service you broke is still a full incident. What a progressive rollout actually buys is reversibility: the ability to stop and pull back while most users are still on the old version, so a bad change at 5% is discovered and reverted while 95% of users never felt it. That property, not the arithmetic of the percentage, is the point.

Signals to watch during a rollout

A rollout should be gated by the signals the service already produces: error rate, latency, and saturation, but computed against the canary population, not the whole fleet. The gate is only doing its job if it can halt and roll back automatically, because a human staring at a dashboard while the canary burns is practising the spirit of the system rather than its safety. Define the thresholds before you start, wire them into the pipeline, and let the rollout pause itself when the measurement goes out of bounds instead of waiting for someone to notice.

The button and the stages

Build every deploy as a series of named stages, each with a traffic percentage, a wait, and an explicit stop-and-roll-back decision, so the release script itself becomes the checklist. The final stage may simply be ‘watch after 100%’, because some failure shapes only appear once traffic saturates and the old version has fully drained out. Treat the rollout not as a moment but as a practice, and the script, its thresholds and its rollback path, as inventory you keep current long before a release night.

Pace the stages like a schedule

A ladder such as 1% → 10% → 50% → 100% works because each rung asks a different question. One percent catches failures no staging environment reproduces: the process that never starts, the query shape that always times out. Fifty percent proves the new version can carry load before the old one drains away. The waits matter as much as the numbers: a stage that promotes after thirty seconds has sampled only the fastest requests, so pair every percentage with a soak time and a health gate. This is where a rollout differs from a feature-flag dial: the flag decides who sees new code, while the rollout decides how long each group is watched, what measurement promotes it, and which command reverses it.

When it breaks

A one-step cutover fails slowly in reverse: the flip takes seconds, but rollback needs a rebuild and a health check, so the bad version serves everyone for twenty minutes. A canary would have bounded the blast to one percent.

The mirror failure is a rollout that halts unwatched: the pipeline pauses at 10% on an amber gate, versions diverge all weekend, and Monday’s resume promotes a build nobody observed. A paused rollout needs an owner and a page.

The third failure decouples percentage from health: stages advance on a timer, or the gate averages errors fleet-wide, so a canary failing at 15% reads as 1.5% overall and the rollout promotes an incident it already measured. Gate on canary-only signals, halt on breach, and keep the old version servable until the 100% soak completes.

Check it quickly

Save the block as rollout_check.py and run python3 rollout_check.py with Python 3.10 or newer. It ran with Python 3.14.4 using only the standard library. The script replays a four-stage rollout with a healthy and a faulty build, gating each stage on canary-only error rate.

"""Simulate a gated staged rollout: 1% -> 10% -> 50% -> 100% with an error-rate breaker.

Only the standard library is used. Run with: python3 rollout_check.py
"""

STAGES = (1, 10, 50, 100)
ERROR_BUDGET = 0.02  # halt the rollout if canary error rate exceeds 2%
REQUESTS_PER_STAGE = 2000


def canary_error_rate(stage_percent, faulty):
    """Fixture error rates: healthy builds hum at 0.5%; the faulty build burns at 15%."""
    if not faulty:
        return 0.005
    return 0.15


def run_rollout(faulty):
    history = []
    reached = 0
    for stage in STAGES:
        canary_errors = canary_error_rate(stage, faulty) * REQUESTS_PER_STAGE
        observed = canary_errors / REQUESTS_PER_STAGE
        if observed > ERROR_BUDGET:
            history.append((stage, observed, "HALT+ROLLBACK"))
            return {"reached": reached, "halted_at": stage,
                    "rolled_back": True, "history": history}
        history.append((stage, observed, "PROMOTE"))
        reached = stage
    return {"reached": reached, "halted_at": None,
            "rolled_back": False, "history": history}


good = run_rollout(False)
assert good["reached"] == 100
assert good["halted_at"] is None and good["rolled_back"] is False

bad = run_rollout(True)
assert bad["halted_at"] == 1, "a faulty build must trip the breaker at the first stage"
assert bad["rolled_back"] is True and bad["reached"] == 0
assert all(stage != 100 for stage, _, _ in bad["history"])

# The trap this rollout avoids: averaging canary errors across the whole fleet.
fleet_average_at_10 = 0.15 * 0.10
assert fleet_average_at_10 < ERROR_BUDGET, "fleet-wide math dilutes a 15% canary failure into a passing 1.5%"

for stage, observed, decision in good["history"]:
    print(f"healthy build at {stage:>3}%: canary errors {observed:.1%} -> {decision}")
for stage, observed, decision in bad["history"]:
    print(f"faulty build at {stage:>3}%: canary errors {observed:.1%} -> {decision}")
print(f"fleet-average trap at 10%: {fleet_average_at_10:.1%} looks fine against a {ERROR_BUDGET:.0%} budget")
print("PASS: healthy build reaches 100%; faulty build halts at 1% and rolls back")

The healthy build reaches 100%; the faulty build trips the breaker at 1% and rolls back. The final assertion captures the fleet-average trap: the same 15% canary failure diluted across 10% of traffic reads as 1.5%, which a fleet-wide gate would wave through. The fixture uses fixed error rates, so it proves the gate logic, not the choice of threshold.

Further reading

Sources