You describe the end state, not the steps. That idea underlies replicas, updates, and rollback: the cluster keeps working to match your description.

Spec beats imperative steps

A Deployment is a statement of the end state, not a list of steps: how many replicas, which image, what update strategy, which labels. The operator publishes the what, and the controller, a reconcile loop, is responsible for the how. The difference from running a command is structural: a one-off docker run or kubectl run orders something to happen now, while a Deployment says ’this is the state I want to remain true.’ If the work is interrupted, the command fails and the state may be left half-done, whereas the controller keeps trying to converge, which is exactly what turns startups and fixes into things the system will keep pursuing.

A controller converges the gap

The reconcile loop is what makes a Deployment trustworthy. Whenever the controller observes that reality differs from the spec, fewer replicas than declared, a pod evicted, a node lost, it does the smallest corrective step and repeats until the gap is closed. Nobody runs ‘create three more’ by hand; the controller re-converges every few seconds. That is the essential difference from running imperative commands, drift is not something you notice and fix, it is a condition the system continuously returns to spec on its own. It is why a crashed pod is replaced and an evicted workload comes back, without an operator watching.

Updates happen through new Revision

Change itself is described as state. A rolling update is a new desired image; the controller stages a new controller-generation and swaps traffic gradually, respecting readiness so the fleet moves forward under control. Because each change creates a new Revision with its own record, a rollout is auditable, and roll back is simply applying a prior Revision rather than writing a new, unplanned change. That keeps the cluster maintainable at scale: operators express intent once, the controller enforces it forever, and history of every change is a versioned object rather than a notebook of commands.

When it breaks

An immutable selector turns a small label edit into lasting drift. spec.selector cannot change after creation, so editing only spec.template.labels leaves new pods outside the selector: the ReplicaSet reports missing replicas while stray pods run unowned nearby. The mirror mistake is two Deployments with overlapping selectors, where each controller counts the other’s pods as observed state and they fight over scale. Treat the selector plus template labels as one atomic contract; when the selector must change, create a new Deployment rather than patching the old one.

Manual edits lose to the loop by design. Scaling the ReplicaSet directly, deleting a pod to save capacity, or running kubectl scale without updating the declared spec works for seconds, then the Deployment controller restores spec.replicas on the next sync. The symptom looks like a flaky API, but kubectl describe shows the controller closing the same gap every time. If an emergency scale is real, change the declared desired state in Git, not the observed objects.

History can outgrow its usefulness. Every spec change keeps one ReplicaSet while revisionHistoryLimit allows, so a pipeline that applies on every commit with a limit of fifty fills etcd with dead revisions and slows rollout history. A limit of zero has the opposite cost: no rollback target remains. Keep the default of ten without a measured reason, and keep the audit in version control rather than in the cluster.

Check it quickly

Save the following as desired_state_loop.py and run python3 desired_state_loop.py. It uses only the standard library and models the same loop the controller runs: compare desired against observed, emit one corrective action per tick, and repeat until the gap closes.

"""Simulate a Deployment-style desired-state reconcile loop (stdlib only)."""

def reconcile_step(desired, observed):
    if observed < desired:
        return "create-one-pod"
    if observed > desired:
        return "delete-one-pod"
    return "in-sync"


def converge(desired, observed, log):
    while observed != desired:
        action = reconcile_step(desired, observed)
        log.append(action)
        if action == "create-one-pod":
            observed += 1
        else:
            observed -= 1
    log.append("in-sync")
    return observed


def main():
    desired = 3
    observed = 1  # one survivor after a node loss
    actions = []
    observed = converge(desired, observed, actions)
    assert observed == desired
    assert actions == ["create-one-pod", "create-one-pod", "in-sync"], actions

    # Manual drift: someone adds two pods without changing desired.
    observed = 5
    drift = []
    observed = converge(desired, observed, drift)
    assert observed == desired
    assert drift == ["delete-one-pod", "delete-one-pod", "in-sync"], drift

    # Selector drift is not auto-healed: template labels outside selector never become observed.
    selector = {"app": "api"}
    template_labels = {"app": "api-v2"}  # deliberate mismatch, a negative control
    matched = all(template_labels.get(k) == v for k, v in selector.items())
    assert not matched, "mismatched labels must not count as observed"

    print(f"desired={desired} final-observed={observed}")
    print(f"recovery-actions={actions}")
    print(f"manual-scale-reverted={drift}")
    print("PASS: controller closes the desired-vs-observed gap; label mismatch stays visible")


if __name__ == "__main__":
    main()

The two recoveries both end in sync, while the mismatched labels are the negative control: pods outside the selector never count as observed, so the loop exposes the contract error instead of healing it.

The assertions passed with Python 3.14.4 on macOS. The model checks convergence arithmetic, not scheduling or storage. Before release day, confirm the live selector, replicas, and revision history.

Further reading

Sources