Nodes · XB Field Notes

Node affinity places workloads on matching nodes

Thu, 09 Dec 2027 00:00:00 +0000

Affinity is placement matching: prefer a label or force it, and use anti-affinity to keep replicas spread apart for resilience. Hardware is never uniform for long — a GPU pool here, fast local NVMe there, an isolated zone for regulated data — and workloads quietly depend on landing in the right place. Without explicit rules, the scheduler spreads pods by free capacity alone, and the pod that needs a GPU lands on the node that has none, failing at runtime instead of at schedule time.

Affinity moves that failure earlier, where it belongs. Stated as data on nodes (labels) matched against rules on pods, placement becomes reviewable configuration rather than tribal knowledge about “which nodes are the special ones.”

Match nodes by label

Affinity rules read node labels, so you can state that a pod should, or must, run only on nodes carrying a particular label. Put a stable label on your nodes once, then let scheduling match against it. Labels like accelerator=nvidia-a100, storage=nvme, or zone=regulated describe durable properties of the machine, and because they live on the node object, every pod rule referencing them stays valid across node replacements — new hardware gets the same label, old rules keep working.

Choose label names with the same care as API fields, because renaming a widely referenced label is a migration across every manifest that matches it. Validate labels in CI the way you validate image tags — a pod requiring a label no node carries is a typo that should fail the pipeline, not the 3 a.m. deploy.

Prefer or require

Preferred rules are best-effort and balance load when possible, while required rules hard-block a pod that cannot satisfy the label. Choose required only when placement genuinely matters; otherwise preference keeps the scheduler flexible. The cost model is straightforward: required rules trade availability for correctness — if no matching node has room, the pod stays Pending rather than running in the wrong place — while preferred rules trade correctness for availability, landing somewhere suboptimal but alive.

Most estates want required for the few hard constraints (GPU present, compliance zone, architecture) and preferred for everything soft (spread across racks, favor the newer instance family). Review required rules quarterly, because hardware that was scarce last year may be plentiful now, and a required rule nobody remembers is how a cluster with free capacity stops scheduling.

Anti-affinity splits replicas

Anti-affinity looks at other pods, spreading replicas of one workload across different nodes so a single node failure stings less. Pair it with replicas and topology keys so the split is meaningful rather than cosmetic. topologyKey: kubernetes.io/hostname spreads across machines; topologyKey: topology.zone spreads across failure domains.

Required anti-affinity deserves the same caution as required affinity: with three replicas and a required hostname spread, the fourth node going down is fine but the workload can never exceed the node count, and a two-node cluster cannot host it at all. For most services, preferred anti-affinity with a high weight gives nearly all the resilience with none of the scheduling fragility. Reserve the required form for the handful of workloads — quorum members, single-leader pairs — where co-location is genuinely worse than Pending.

When it breaks

The signature failure is the permanently Pending pod: events show didn't match pod affinity rules while nodes sit half empty. Causes are almost always label-side — the node label was never applied, was renamed, or carries a different value than the rule expects (nvme vs nvme-ssd). kubectl get nodes -L hardware,workload-class shows the actual label surface in one table; diff it against the pod’s requirements and the mismatch usually confesses immediately.

The subtler failure is the satisfied-but-wrong placement: preferred rules silently lose to capacity pressure, and over weeks the GPU workloads drift onto shared nodes while the GPU pool idles. Preferred means the scheduler may ignore you, and under pressure it will. Audit placement periodically against intent — a query grouping pods by workload and node label — and promote the rules that keep drifting from preferred to required, or fix the capacity shortage that forces the compromise.

Check it quickly

See labels, requirements, and actual placement together:

kubectl get nodes -L accelerator,workload-class --no-headers
kubectl describe pod api-0 | grep -A 8 "Node-Selectors\|Affinity"
kubectl get pods -o wide -l app=api | awk '{print $1, $7}' | sort | uniq -c | sort -rn | head

The first shows the label surface, the second shows what one pod demands, and the third proves whether replicas actually spread or quietly piled onto one node.

Sources

Kubernetes node affinity docs, node affinity semantics, checked 2026-06-10.
Kubernetes taints and tolerations, documents repelling pods from specialized nodes, checked 2026-09-05.

Taints repel pods that do not tolerate them

Thu, 25 Feb 2027 00:00:00 +0000

Taint is the ’no admission’ sign and toleration is the permit, so you can pin paying platforms, GPUs, or spot nodes to exactly the pods that fit.

A sign for specialists

A taint marks a node as off-limits to ordinary pods, and it is fundamentally a sign, not a command: applying a taint does not move anything that is already running. New pods avoid the node unless they carry a matching toleration, which is the ‘permit.’ That is how you reserve a subset of the cluster, GPU machines, spot nodes, nodes with special hardware, for exactly the workloads that fit, while everything else schedules as if those nodes did not exist. Because it is scoped by opt-in, the same physical fleet can host both the general pool and the specialist pool without label soup.

Effect decides the action

The taint’s effect field is what turns a mark into behavior. NoSchedule keeps new pods away but leaves running ones alone, the default you want for capacity you plan to use. PreferNoSchedule is a soft bias that discourages, but does not forbid, scheduling; useful for cost-sensitive pools where you prefer cheap nodes rather than a hard rule. NoExecute goes further and evicts pods already on the node, which also lets you cordon a node for maintenance: taint it NoExecute and the scheduler drains it on its own. Choose the effect that matches the action you really intend, because software and hard exclusion are very different guarantees.

Pair is a contract

Tolerations are always paired against a taint by key, value, and effect, so the match is explicit and it is a contract between two objects. A toleration expresses ‘I can live on nodes bearing this taint’; it does not express a preference or force placement, so a tolerant pod can still land elsewhere. That is why you can mix GPU nodes and general nodes in one cluster without accidental misplacement: pods that cannot tolerate the taint simply never get scheduled there, and specialists stay on the hardware they need, not the budget you guessed. Design the pairing deliberately, with an owner and a reason, or the contract drifts into label soup.

When it breaks

The most expensive failure is the over-broad toleration that pollutes the specialist pool. A toleration with operator: Exists and no value, or one copied into a shared chart, matches every value of that key, so ordinary Deployments suddenly qualify for GPU or spot nodes. The reserved pool fills with cheap pods while the specialists it was built for stay Pending. This is the mirror image of the DaemonSet failure: there a missing toleration leaves special nodes uncovered, here an extra toleration leaves them unprotected. Treat tolerations like RBAC wildcards.

The second failure is applying NoExecute when you meant NoSchedule. NoSchedule steers only new pods, while NoExecute evicts running pods without a matching toleration, optionally after tolerationSeconds expires. A maintenance taint with the wrong effect drains production in seconds. Unlike affinity, which never evicts, the taint effect carries destructive power, so rehearse on one node and watch kubectl get pods -o wide before repeating across the pool.

The third failure is spelling drift between taint and toleration. dedicated=gpu:NoSchedule never meets dedicated=GPU:NoSchedule, and a toleration naming PreferNoSchedule does not cover a NoSchedule taint. Specialists sit Pending with had taint that the pod didn't tolerate while the pool looks healthy. Compare both sides with kubectl describe pod against kubectl get node <name> -o jsonpath='{.spec.taints}', and keep the canonical triple in one documented place.

Check it quickly

Save the following as taint_toleration_sim.py and run python3 taint_toleration_sim.py. It uses only the standard library, touches no cluster, and models the rules above: key plus operator plus value plus effect, with soft taints never blocking.

def tolerates(taint, tol):
 """One toleration matches one taint on key, value, operator, and effect."""
 if tol.get("key") and tol["key"] != taint["key"]:
 return False
 if tol.get("operator", "Equal") == "Equal":
 if tol.get("value") != taint.get("value"):
 return False
 effect = tol.get("effect", "")
 if effect and effect != taint.get("effect"):
 return False
 return True


def schedulable(taints, tolerations):
 """A pod schedules only if every hard taint is tolerated."""
 for taint in taints:
 if taint["effect"] == "PreferNoSchedule":
 continue
 if not any(tolerates(taint, tol) for tol in tolerations):
 return False
 return True


def check(actual, expected, message):
 if actual != expected:
 raise AssertionError(f"{message}: got {actual}, want {expected}")
 print(f"OK: {message} -> {actual}")


def main():
 gpu = {"key": "dedicated", "value": "gpu", "effect": "NoSchedule"}
 exact = {"key": "dedicated", "value": "gpu", "effect": "NoSchedule"}
 check(schedulable([gpu], [exact]), True, "exact match schedules")
 check(schedulable([gpu], []), False, "missing toleration is repelled")
 wide = {"key": "dedicated", "operator": "Exists", "effect": "NoSchedule"}
 other = {"key": "dedicated", "value": "spot", "effect": "NoSchedule"}
 check(schedulable([other], [wide]), True, "Exists toleration matches any value")
 check(schedulable([gpu], [{"key": "dedicated", "value": "spot", "effect": "NoSchedule"}]), False, "wrong value is repelled")
 evict = {"key": "maintenance", "value": "drain", "effect": "NoExecute"}
 any_effect = {"key": "maintenance", "value": "drain"}
 check(schedulable([evict], [any_effect]), True, "empty effect matches any effect")
 check(schedulable([evict], []), False, "untolerated NoExecute would evict")
 soft = {"key": "cost", "value": "cheap", "effect": "PreferNoSchedule"}
 check(schedulable([soft], []), True, "PreferNoSchedule never blocks")
 print("PASS: 7 scenarios; allow and repel verified")


if __name__ == "__main__":
 main()

The seven scenarios prove an exact toleration allows scheduling while a missing one repels, an Exists toleration matches any value but a wrong value is refused, an empty effect covers NoExecute, an untolerated NoExecute would evict, and PreferNoSchedule never blocks. Expected output ends with PASS: 7 scenarios; allow and repel verified.

The assertions passed with Python 3.12.6 on macOS. The script checks matching logic only, not capacity, affinity, or live taints. Before changing production, compare pod events against the node’s actual taint list.

Sources

Kubernetes taints and tolerations, documents effects and scheduling, checked 2026-06-01.
Kubernetes node affinity, documents matching pods to node labels, checked 2026-06-01.

Nodes · XB Field Notes

Node affinity places workloads on matching nodes

Match nodes by label

Prefer or require

Anti-affinity splits replicas

When it breaks

Check it quickly

Sources

Taints repel pods that do not tolerate them

A sign for specialists

Effect decides the action

Pair is a contract

When it breaks

Check it quickly

Further reading

Sources