Cordon and drain take a node out safely

Thu, 03 Feb 2028 00:00:00 +0000

Cordon and drain are two actions people usually say in one breath. Cordon sets spec.unschedulable on the node so the scheduler stops choosing it. Drain evicts what is already running there. Cordon alone is instant and reversible; drain is a loop that talks to the eviction API once per pod.

The tension is that drain is a client-side procedure with a safety rail: it refuses before it deletes anything, and every flag that gets past a refusal trades some of that safety away. Knowing which flag covers which refusal is the difference between a maintenance window and an outage.

Cordon closes the door, drain clears the room

kubectl cordon node-1 patches spec.unschedulable: true, and the scheduler filters that node out for new pods. Pods already running keep serving, which is why you cordon first: stop accepting new work before removing existing work. Cordoning is also what makes the drain meaningful: without it the scheduler would place replacements straight back onto the node you are emptying. kubectl drain sets the same flag itself, so a separate cordon is a courtesy rather than a requirement. kubectl uncordon is the same patch in reverse, and the scheduler starts using the node again immediately — confirm the kubelet is Ready first.

Drain then evicts pods one at a time through the eviction API, falling back to a plain delete where the server does not. Eviction is a graceful termination: terminationGracePeriodSeconds is honoured and drain waits for it. Grace is a window, not a guarantee: a pod that ignores SIGTERM is killed when it expires, so the pod’s own shutdown logic protects the work. Mirror pods are the exception — the kubelet manages them from a static manifest and they cannot be deleted through the API server at all, so drain skips them rather than failing on them.

The refusal list is all or nothing

Drain refuses before it deletes anything, and one bad pod blocks every eviction on the node. The refusals map to three flags, and the binary names them: cannot delete Pods that declare no controller (use --force to override) for a bare pod or one whose managing resource has gone missing; Pods with local storage (use --delete-emptydir-data to override) for anything using emptyDir; and DaemonSet-managed Pods (use --ignore-daemonsets to ignore).

The DaemonSet case is the one where the flag does not change the outcome. Drain never deletes DaemonSet pods, with or without the flag, because the DaemonSet controller ignores the unschedulable taint and would recreate them immediately; --ignore-daemonsets only silences the refusal. Two other kinds of pod reach a drained node regardless: anything tolerating node.kubernetes.io/unschedulable, and anything that sets nodeName directly, bypassing the scheduler. Pods tolerating that taint are usually DaemonSets, which is why the docs say not to tolerate it for anything else.

The budget, not the flags, decides the rest

Once the refusals are cleared, drain starts submitting evictions and the PodDisruptionBudget has the last word. A PDB with disruptionsAllowed at zero answers 429, and drain retries with a backoff until --timeout, whose default of zero means forever. That retry loop is why a stalled drain looks like a hang, not a failure: drain is doing exactly what it was told. No error, no progress, just an evicting pod line repeating.

--disable-eviction is the flag that bypasses budgets, by deleting pods directly instead of asking the eviction API. --force does not do that; it only covers pods with no controller, the opposite of what people assume. The two flags are not interchangeable. The default unhealthy-pod policy, IfHealthyBudget, also lets one CrashLoopBackOff pod stall an entire drain until it becomes Ready.

When it breaks

The drain that never returns. A PDB with minAvailable equal to the replica count, or maxUnavailable: 0, resolves to zero allowed disruptions, so the eviction is rejected forever and the terminal just sits there. Draining several nodes in parallel does not help either; they share the same budget. Diagnose with kubectl get pdb -A: a PDB whose pods live on the node you are draining, showing ALLOWED DISRUPTIONS 0, is the culprit; the fix is a budget that permits one eviction, not a longer timeout.

The drain that takes the workload down. Faced with a stalled drain, someone reaches for --force, and the node empties — including the pods the budget was protecting. If that workload held quorum, the maintenance becomes the outage. The safe order is to fix the budget and wait, reserving --force for a pod you have verified is genuinely stateless.

Check it quickly

Confirm the node is closed and see what drain would refuse:

kubectl get node node-1 -o jsonpath='{.spec.unschedulable}{"\n"}'
kubectl drain node-1 --dry-run=client --ignore-daemonsets --delete-emptydir-data
kubectl get pods --field-selector spec.nodeName=node-1 -o custom-columns=NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind
kubectl get pdb -A -o custom-columns=NAME:.metadata.name,MIN:.spec.minAvailable,MAX:.spec.maxUnavailable,ALLOWED:.status.disruptionsAllowed

The first prints true once the node is cordoned. The dry run lists the pods drain would evict and the ones it refuses by name. In the third command an empty OWNER column means --force is required for that pod, and in the fourth an ALLOWED of 0 on a PDB whose pods are on this node means the drain will wait rather than fail.

Sources

Safely Drain a Node, the drain workflow and its interaction with PodDisruptionBudgets, checked 2026-09-16.
kubectl drain reference, the exact refusal conditions and flags, checked 2026-09-16.
Pod disruptions, voluntary versus involuntary disruption and eviction retries, checked 2026-09-16.