Affinity is placement matching: prefer a label or force it, and use anti-affinity to keep replicas spread apart for resilience. Hardware is never uniform for long — a GPU pool here, fast local NVMe there, an isolated zone for regulated data — and workloads quietly depend on landing in the right place. Without explicit rules, the scheduler spreads pods by free capacity alone, and the pod that needs a GPU lands on the node that has none, failing at runtime instead of at schedule time.

Affinity moves that failure earlier, where it belongs. Stated as data on nodes (labels) matched against rules on pods, placement becomes reviewable configuration rather than tribal knowledge about “which nodes are the special ones.”

Match nodes by label

Affinity rules read node labels, so you can state that a pod should, or must, run only on nodes carrying a particular label. Put a stable label on your nodes once, then let scheduling match against it. Labels like accelerator=nvidia-a100, storage=nvme, or zone=regulated describe durable properties of the machine, and because they live on the node object, every pod rule referencing them stays valid across node replacements — new hardware gets the same label, old rules keep working.

Choose label names with the same care as API fields, because renaming a widely referenced label is a migration across every manifest that matches it. Validate labels in CI the way you validate image tags — a pod requiring a label no node carries is a typo that should fail the pipeline, not the 3 a.m. deploy.

Prefer or require

Preferred rules are best-effort and balance load when possible, while required rules hard-block a pod that cannot satisfy the label. Choose required only when placement genuinely matters; otherwise preference keeps the scheduler flexible. The cost model is straightforward: required rules trade availability for correctness — if no matching node has room, the pod stays Pending rather than running in the wrong place — while preferred rules trade correctness for availability, landing somewhere suboptimal but alive.

Most estates want required for the few hard constraints (GPU present, compliance zone, architecture) and preferred for everything soft (spread across racks, favor the newer instance family). Review required rules quarterly, because hardware that was scarce last year may be plentiful now, and a required rule nobody remembers is how a cluster with free capacity stops scheduling.

Anti-affinity splits replicas

Anti-affinity looks at other pods, spreading replicas of one workload across different nodes so a single node failure stings less. Pair it with replicas and topology keys so the split is meaningful rather than cosmetic. topologyKey: kubernetes.io/hostname spreads across machines; topologyKey: topology.zone spreads across failure domains.

Required anti-affinity deserves the same caution as required affinity: with three replicas and a required hostname spread, the fourth node going down is fine but the workload can never exceed the node count, and a two-node cluster cannot host it at all. For most services, preferred anti-affinity with a high weight gives nearly all the resilience with none of the scheduling fragility. Reserve the required form for the handful of workloads — quorum members, single-leader pairs — where co-location is genuinely worse than Pending.

When it breaks

The signature failure is the permanently Pending pod: events show didn't match pod affinity rules while nodes sit half empty. Causes are almost always label-side — the node label was never applied, was renamed, or carries a different value than the rule expects (nvme vs nvme-ssd). kubectl get nodes -L hardware,workload-class shows the actual label surface in one table; diff it against the pod’s requirements and the mismatch usually confesses immediately.

The subtler failure is the satisfied-but-wrong placement: preferred rules silently lose to capacity pressure, and over weeks the GPU workloads drift onto shared nodes while the GPU pool idles. Preferred means the scheduler may ignore you, and under pressure it will. Audit placement periodically against intent — a query grouping pods by workload and node label — and promote the rules that keep drifting from preferred to required, or fix the capacity shortage that forces the compromise.

Check it quickly

See labels, requirements, and actual placement together:

kubectl get nodes -L accelerator,workload-class --no-headers
kubectl describe pod api-0 | grep -A 8 "Node-Selectors\|Affinity"
kubectl get pods -o wide -l app=api | awk '{print $1, $7}' | sort | uniq -c | sort -rn | head

The first shows the label surface, the second shows what one pod demands, and the third proves whether replicas actually spread or quietly piled onto one node.

Sources