Scheduling · XB Field Notes

Node affinity places workloads on matching nodes

Thu, 09 Dec 2027 00:00:00 +0000

Affinity is placement matching: prefer a label or force it, and use anti-affinity to keep replicas spread apart for resilience. Hardware is never uniform for long — a GPU pool here, fast local NVMe there, an isolated zone for regulated data — and workloads quietly depend on landing in the right place. Without explicit rules, the scheduler spreads pods by free capacity alone, and the pod that needs a GPU lands on the node that has none, failing at runtime instead of at schedule time.

Affinity moves that failure earlier, where it belongs. Stated as data on nodes (labels) matched against rules on pods, placement becomes reviewable configuration rather than tribal knowledge about “which nodes are the special ones.”

Match nodes by label

Affinity rules read node labels, so you can state that a pod should, or must, run only on nodes carrying a particular label. Put a stable label on your nodes once, then let scheduling match against it. Labels like accelerator=nvidia-a100, storage=nvme, or zone=regulated describe durable properties of the machine, and because they live on the node object, every pod rule referencing them stays valid across node replacements — new hardware gets the same label, old rules keep working.

Choose label names with the same care as API fields, because renaming a widely referenced label is a migration across every manifest that matches it. Validate labels in CI the way you validate image tags — a pod requiring a label no node carries is a typo that should fail the pipeline, not the 3 a.m. deploy.

Prefer or require

Preferred rules are best-effort and balance load when possible, while required rules hard-block a pod that cannot satisfy the label. Choose required only when placement genuinely matters; otherwise preference keeps the scheduler flexible. The cost model is straightforward: required rules trade availability for correctness — if no matching node has room, the pod stays Pending rather than running in the wrong place — while preferred rules trade correctness for availability, landing somewhere suboptimal but alive.

Most estates want required for the few hard constraints (GPU present, compliance zone, architecture) and preferred for everything soft (spread across racks, favor the newer instance family). Review required rules quarterly, because hardware that was scarce last year may be plentiful now, and a required rule nobody remembers is how a cluster with free capacity stops scheduling.

Anti-affinity splits replicas

Anti-affinity looks at other pods, spreading replicas of one workload across different nodes so a single node failure stings less. Pair it with replicas and topology keys so the split is meaningful rather than cosmetic. topologyKey: kubernetes.io/hostname spreads across machines; topologyKey: topology.zone spreads across failure domains.

Required anti-affinity deserves the same caution as required affinity: with three replicas and a required hostname spread, the fourth node going down is fine but the workload can never exceed the node count, and a two-node cluster cannot host it at all. For most services, preferred anti-affinity with a high weight gives nearly all the resilience with none of the scheduling fragility. Reserve the required form for the handful of workloads — quorum members, single-leader pairs — where co-location is genuinely worse than Pending.

When it breaks

The signature failure is the permanently Pending pod: events show didn't match pod affinity rules while nodes sit half empty. Causes are almost always label-side — the node label was never applied, was renamed, or carries a different value than the rule expects (nvme vs nvme-ssd). kubectl get nodes -L hardware,workload-class shows the actual label surface in one table; diff it against the pod’s requirements and the mismatch usually confesses immediately.

The subtler failure is the satisfied-but-wrong placement: preferred rules silently lose to capacity pressure, and over weeks the GPU workloads drift onto shared nodes while the GPU pool idles. Preferred means the scheduler may ignore you, and under pressure it will. Audit placement periodically against intent — a query grouping pods by workload and node label — and promote the rules that keep drifting from preferred to required, or fix the capacity shortage that forces the compromise.

Check it quickly

See labels, requirements, and actual placement together:

kubectl get nodes -L accelerator,workload-class --no-headers
kubectl describe pod api-0 | grep -A 8 "Node-Selectors\|Affinity"
kubectl get pods -o wide -l app=api | awk '{print $1, $7}' | sort | uniq -c | sort -rn | head

The first shows the label surface, the second shows what one pod demands, and the third proves whether replicas actually spread or quietly piled onto one node.

Sources

Kubernetes node affinity docs, node affinity semantics, checked 2026-06-10.
Kubernetes taints and tolerations, documents repelling pods from specialized nodes, checked 2026-09-05.

Requests and limits set a Pod's floor and cap

Thu, 27 May 2027 00:00:00 +0000

The cluster schedules on requests but enforces limits, so a container that sets only one side leaves a half-open contract. Requests-only pods schedule honestly but burst without bound, free to starve neighbors when they spike. Limits-only pods are capped at runtime yet scheduled as if they cost nothing, so the scheduler packs them densely until the node saturates. Either half-contract fails under pressure; the two numbers only work as a pair.

Think of requests as the booking and limits as the house rules: the booking decides which node has room, the rules decide what happens when the guest gets loud. Set both from measurement and the cluster stays predictable.

Two numbers, two jobs

A CPU request pledges what the scheduler reserves on a node, while the limit caps the share a container can hit before the cgroup throttles it. Both matter: requests decide placement, limits decide ceiling. CPU is compressible, so breaching the CPU limit only slows the container — throttling, not death. Memory is not: breaching the memory limit kills the process via OOM, with no slowdown phase in between. That asymmetry drives every sane policy: CPU limits can sit well above requests to absorb bursts, while memory limits should hug the measured working set plus headroom.

Omissions have defaults with teeth, and admission policy can change the exact result. Without a CPU request the scheduler accounts for zero requested CPU unless a LimitRange or an explicit limit supplies a default; the Pod may then be packed too densely and receive little share under contention. Without a memory limit the process can grow until node pressure makes the kubelet reclaim or evict work. Inspect the admitted Pod, not only the source YAML, because namespace defaults may have filled in values you did not write.

The OOM corner

An untuned memory limit is the classic cause of the kernel reclaiming the workload when a spike crosses the advertised cap. If you set a memory limit, give it headroom above the measured working set. The working set is not the average — it is the p95-to-max range across a full cycle including deploys, batch windows, and traffic peaks. Set the limit at average and every routine spike becomes a kill; set it at 2x max and the reservation waste shows up as Pending pods elsewhere. The honest window is max plus twenty to fifty percent, tuned per workload, rechecked after every scaling change.

Java and other runtime-managed workloads deserve special care: heap plus metaspace plus native plus off-heap must all fit under the limit, not just the -Xmx you remember setting. Size the container limit from the observed process RSS, then work backwards to heap flags — never the reverse.

Start near reality

Measure the steady state, set requests near it, keep limits a defensive step above, and revisit whenever the workload shape shifts. Iterating from real usage beats guessing a number that protects nothing. The practical loop: run a week on honest estimates, read p95 usage per container from the metrics server or Prometheus, set requests at that line and limits above max with headroom, then watch throttling and OOM counters for a week. Zero throttling with huge headroom means requests can drop; any throttling on latency-sensitive paths means limits must rise.

Encode the ratio as team policy rather than folklore: 1:1 memory (Guaranteed QoS) for critical stateful services, 1:2 for web tiers, looser for batch — written down with the workload evidence linked, so the next engineer copies the reasoning along with the numbers.

When it breaks

The signature failure is the OOMKilled loop: restarts climbing, exit code 137, last week’s deploy the only change — and the change was a ten-percent traffic lift against a limit set exactly at old max. Diagnose with the usage-versus-limit graph, not the logs: the process did nothing wrong, it simply outgrew its cap. The fix is headroom, and the prevention is alerting on usage-to-limit ratio crossing eighty percent, which pages days before the kill instead of after.

The mirror failure is the throttled-to-death service: p99 latency doubled, CPU flat at the limit line, dashboard blaming the application. CPU throttling is invisible to in-container metrics — the app sees idle time it cannot use. When latency degrades with CPU pinned exactly at the limit, raise the limit before profiling the code.

Check it quickly

Compare booked versus real per container:

kubectl top pods -n prod --sort-by=memory | head
kubectl get pods -n prod -o custom-columns=NAME:.metadata.name,REQ:.spec.containers[*].resources.requests.memory,LIM:.spec.containers[*].resources.limits.memory --no-headers | head -n 15
kubectl get events -n prod --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -n 5

Usage shows reality, the manifest dump shows the contract, and the event tail shows where the contract already broke.

Sources

Kubernetes resource management, Official doc describing how requests and limits are interpreted by the scheduler, checked 2026-06-01.
Kubernetes QoS classes, explains Guaranteed, Burstable, and BestEffort behavior, checked 2026-09-05.

Pod anti-affinity spreads peers away from each other

Thu, 18 Mar 2027 00:00:00 +0000

Anti-affinity is the scheduler rule that keeps replicas apart. Where affinity says “run me next to these pods”, anti-affinity says “keep me away from them”, and the reason is almost always failure domains: if all three replicas land on one node, one node failure is a full outage.

The tension lives in a single word. required makes the rule a filter, so a pod with nowhere legal to go stays Pending forever; preferred makes it a score, so the same rule degrades gracefully. One YAML shape produces either a deadlock or a soft preference depending on which you typed.

A rule is a selector plus a topology

The rule has two parts. A labelSelector chooses which pods count as peers, and a topologyKey names the node label that defines a domain. kubernetes.io/hostname makes every node its own domain; topology.kubernetes.io/zone groups nodes into zones, so “keep apart” comes to mean “keep to different zones”. The namespaces field defaults to the pod’s own namespace, so anti-affinity never accidentally spans tenants.

Nodes that lack the topologyKey label are not part of any domain. The docs warn that anti-affinity requires consistently labelled nodes and that a missing key leads to unintended behaviour. In practice an unlabelled node is invisible to the rule: neither filtered out nor counted, so it can collect exactly the replicas the rule was meant to spread. One more wrinkle: preferred rules attached to pods that already exist are ignored when a new pod is scheduled. Only the incoming pod’s own rules count, so retrofitting a spread rule onto a running workload does not reshuffle what is already placed.

Required is a cap, not a guarantee

Required anti-affinity is a node filter, so it caps a workload at one replica per domain. Nine replicas across six nodes leaves three Pending forever, no matter how much spare capacity exists. Across three zones, four replicas leave one Pending. Every rejected pod reports FailedScheduling, the autoscaler keeps reporting that it cannot help, and nothing ever errors.

That is why the docs warn against required anti-affinity except where the hard constraint is real, such as one instance per node for a hostPort or a local SSD. They also note that inter-pod affinity and anti-affinity need substantial processing and are not recommended above several hundred nodes, which is a second reason to reach for something cheaper. A required rule is also not retroactive: it filters placement at admission time, so a node label that changes later does not evict pods already running, which is the IgnoredDuringExecution half of the name.

Spread constraints say the same thing more cheaply

topologySpreadConstraints expresses the same intent with an explicit tolerance. maxSkew is the permitted difference between the fullest domain and the global minimum; topologyKey names the domain; whenUnsatisfiable decides whether an unsatisfiable skew is fatal (DoNotSchedule, the default) or merely a scoring penalty (ScheduleAnyway). Nine replicas over three zones with maxSkew: 1 land as three, three, three.

The docs’ worked example is worth internalising: two zones holding two and one matching pods, with maxSkew: 1, forces the next pod into the emptier zone, because placing it in the fuller one makes the skew two. minDomains raises the global minimum so a cluster with fewer zones than expected cannot call itself balanced, and it is only valid with DoNotSchedule. The reason to prefer constraints over anti-affinity is cost as much as clarity: spread counts pods per domain, while anti-affinity evaluates a term against every node. Constraints also compose, so one per topology key can spread across zones and nodes at the same time.

When it breaks

Required anti-affinity with more replicas than domains. The workload scales up to the number of nodes and stops there, every further pod sits Pending, and the horizontal autoscaler keeps asking for more. The same trap appears in a small cluster: a two-node test cluster with three replicas and a required hostname rule can never run the third. Switch to preferred or to a spread constraint with ScheduleAnyway, and keep a hard rule only where one pod per node is genuinely required.

Nodes without the topology label. A node pool that never received topology.kubernetes.io/zone is outside every domain, so it is invisible to the constraint: replicas pile onto the labelled nodes while a third of the cluster sits unused. Verify with kubectl get nodes -L topology.kubernetes.io/zone that every node carries a value.

Check it quickly

Confirm the topology is real before trusting the spread:

kubectl get nodes -L topology.kubernetes.io/zone -L kubernetes.io/hostname
kubectl get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,PHASE:.status.phase
kubectl get events --field-selector reason=FailedScheduling --sort-by=.lastTimestamp | tail -5
kubectl get pods -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort | uniq -c

A blank cell in the first command means that node belongs to no domain. FailedScheduling events naming anti-affinity rules mean the required rule has no eligible node left. The last command shows the real distribution: a count of one per node means required anti-affinity is capping the workload, while uneven counts mean a spread constraint is not being honoured.

Sources

Assigning Pods to Nodes, inter-pod affinity and anti-affinity rules, topology keys and scheduling behaviour, checked 2026-09-16.
Pod Topology Spread Constraints, maxSkew, whenUnsatisfiable, minDomains and the worked skew example, checked 2026-09-16.

Taints repel pods that do not tolerate them

Thu, 25 Feb 2027 00:00:00 +0000

Taint is the ’no admission’ sign and toleration is the permit, so you can pin paying platforms, GPUs, or spot nodes to exactly the pods that fit.

A sign for specialists

A taint marks a node as off-limits to ordinary pods, and it is fundamentally a sign, not a command: applying a taint does not move anything that is already running. New pods avoid the node unless they carry a matching toleration, which is the ‘permit.’ That is how you reserve a subset of the cluster, GPU machines, spot nodes, nodes with special hardware, for exactly the workloads that fit, while everything else schedules as if those nodes did not exist. Because it is scoped by opt-in, the same physical fleet can host both the general pool and the specialist pool without label soup.

Effect decides the action

The taint’s effect field is what turns a mark into behavior. NoSchedule keeps new pods away but leaves running ones alone, the default you want for capacity you plan to use. PreferNoSchedule is a soft bias that discourages, but does not forbid, scheduling; useful for cost-sensitive pools where you prefer cheap nodes rather than a hard rule. NoExecute goes further and evicts pods already on the node, which also lets you cordon a node for maintenance: taint it NoExecute and the scheduler drains it on its own. Choose the effect that matches the action you really intend, because software and hard exclusion are very different guarantees.

Pair is a contract

Tolerations are always paired against a taint by key, value, and effect, so the match is explicit and it is a contract between two objects. A toleration expresses ‘I can live on nodes bearing this taint’; it does not express a preference or force placement, so a tolerant pod can still land elsewhere. That is why you can mix GPU nodes and general nodes in one cluster without accidental misplacement: pods that cannot tolerate the taint simply never get scheduled there, and specialists stay on the hardware they need, not the budget you guessed. Design the pairing deliberately, with an owner and a reason, or the contract drifts into label soup.

When it breaks

The most expensive failure is the over-broad toleration that pollutes the specialist pool. A toleration with operator: Exists and no value, or one copied into a shared chart, matches every value of that key, so ordinary Deployments suddenly qualify for GPU or spot nodes. The reserved pool fills with cheap pods while the specialists it was built for stay Pending. This is the mirror image of the DaemonSet failure: there a missing toleration leaves special nodes uncovered, here an extra toleration leaves them unprotected. Treat tolerations like RBAC wildcards.

The second failure is applying NoExecute when you meant NoSchedule. NoSchedule steers only new pods, while NoExecute evicts running pods without a matching toleration, optionally after tolerationSeconds expires. A maintenance taint with the wrong effect drains production in seconds. Unlike affinity, which never evicts, the taint effect carries destructive power, so rehearse on one node and watch kubectl get pods -o wide before repeating across the pool.

The third failure is spelling drift between taint and toleration. dedicated=gpu:NoSchedule never meets dedicated=GPU:NoSchedule, and a toleration naming PreferNoSchedule does not cover a NoSchedule taint. Specialists sit Pending with had taint that the pod didn't tolerate while the pool looks healthy. Compare both sides with kubectl describe pod against kubectl get node <name> -o jsonpath='{.spec.taints}', and keep the canonical triple in one documented place.

Check it quickly

Save the following as taint_toleration_sim.py and run python3 taint_toleration_sim.py. It uses only the standard library, touches no cluster, and models the rules above: key plus operator plus value plus effect, with soft taints never blocking.

def tolerates(taint, tol):
 """One toleration matches one taint on key, value, operator, and effect."""
 if tol.get("key") and tol["key"] != taint["key"]:
 return False
 if tol.get("operator", "Equal") == "Equal":
 if tol.get("value") != taint.get("value"):
 return False
 effect = tol.get("effect", "")
 if effect and effect != taint.get("effect"):
 return False
 return True


def schedulable(taints, tolerations):
 """A pod schedules only if every hard taint is tolerated."""
 for taint in taints:
 if taint["effect"] == "PreferNoSchedule":
 continue
 if not any(tolerates(taint, tol) for tol in tolerations):
 return False
 return True


def check(actual, expected, message):
 if actual != expected:
 raise AssertionError(f"{message}: got {actual}, want {expected}")
 print(f"OK: {message} -> {actual}")


def main():
 gpu = {"key": "dedicated", "value": "gpu", "effect": "NoSchedule"}
 exact = {"key": "dedicated", "value": "gpu", "effect": "NoSchedule"}
 check(schedulable([gpu], [exact]), True, "exact match schedules")
 check(schedulable([gpu], []), False, "missing toleration is repelled")
 wide = {"key": "dedicated", "operator": "Exists", "effect": "NoSchedule"}
 other = {"key": "dedicated", "value": "spot", "effect": "NoSchedule"}
 check(schedulable([other], [wide]), True, "Exists toleration matches any value")
 check(schedulable([gpu], [{"key": "dedicated", "value": "spot", "effect": "NoSchedule"}]), False, "wrong value is repelled")
 evict = {"key": "maintenance", "value": "drain", "effect": "NoExecute"}
 any_effect = {"key": "maintenance", "value": "drain"}
 check(schedulable([evict], [any_effect]), True, "empty effect matches any effect")
 check(schedulable([evict], []), False, "untolerated NoExecute would evict")
 soft = {"key": "cost", "value": "cheap", "effect": "PreferNoSchedule"}
 check(schedulable([soft], []), True, "PreferNoSchedule never blocks")
 print("PASS: 7 scenarios; allow and repel verified")


if __name__ == "__main__":
 main()

The seven scenarios prove an exact toleration allows scheduling while a missing one repels, an Exists toleration matches any value but a wrong value is refused, an empty effect covers NoExecute, an untolerated NoExecute would evict, and PreferNoSchedule never blocks. Expected output ends with PASS: 7 scenarios; allow and repel verified.

The assertions passed with Python 3.12.6 on macOS. The script checks matching logic only, not capacity, affinity, or live taints. Before changing production, compare pod events against the node’s actual taint list.

Sources

Kubernetes taints and tolerations, documents effects and scheduling, checked 2026-06-01.
Kubernetes node affinity, documents matching pods to node labels, checked 2026-06-01.

Scheduling · XB Field Notes

Node affinity places workloads on matching nodes

Match nodes by label

Prefer or require

Anti-affinity splits replicas

When it breaks

Check it quickly

Sources

Requests and limits set a Pod's floor and cap

Two numbers, two jobs

The OOM corner

Start near reality

When it breaks

Check it quickly

Sources

Pod anti-affinity spreads peers away from each other

A rule is a selector plus a topology

Required is a cap, not a guarantee

Spread constraints say the same thing more cheaply

When it breaks

Check it quickly

Sources

Taints repel pods that do not tolerate them

A sign for specialists

Effect decides the action

Pair is a contract

When it breaks

Check it quickly

Further reading

Sources