Taint is the ’no admission’ sign and toleration is the permit, so you can pin paying platforms, GPUs, or spot nodes to exactly the pods that fit.

A sign for specialists

A taint marks a node as off-limits to ordinary pods, and it is fundamentally a sign, not a command: applying a taint does not move anything that is already running. New pods avoid the node unless they carry a matching toleration, which is the ‘permit.’ That is how you reserve a subset of the cluster, GPU machines, spot nodes, nodes with special hardware, for exactly the workloads that fit, while everything else schedules as if those nodes did not exist. Because it is scoped by opt-in, the same physical fleet can host both the general pool and the specialist pool without label soup.

Effect decides the action

The taint’s effect field is what turns a mark into behavior. NoSchedule keeps new pods away but leaves running ones alone, the default you want for capacity you plan to use. PreferNoSchedule is a soft bias that discourages, but does not forbid, scheduling; useful for cost-sensitive pools where you prefer cheap nodes rather than a hard rule. NoExecute goes further and evicts pods already on the node, which also lets you cordon a node for maintenance: taint it NoExecute and the scheduler drains it on its own. Choose the effect that matches the action you really intend, because software and hard exclusion are very different guarantees.

Pair is a contract

Tolerations are always paired against a taint by key, value, and effect, so the match is explicit and it is a contract between two objects. A toleration expresses ‘I can live on nodes bearing this taint’; it does not express a preference or force placement, so a tolerant pod can still land elsewhere. That is why you can mix GPU nodes and general nodes in one cluster without accidental misplacement: pods that cannot tolerate the taint simply never get scheduled there, and specialists stay on the hardware they need, not the budget you guessed. Design the pairing deliberately, with an owner and a reason, or the contract drifts into label soup.

When it breaks

The most expensive failure is the over-broad toleration that pollutes the specialist pool. A toleration with operator: Exists and no value, or one copied into a shared chart, matches every value of that key, so ordinary Deployments suddenly qualify for GPU or spot nodes. The reserved pool fills with cheap pods while the specialists it was built for stay Pending. This is the mirror image of the DaemonSet failure: there a missing toleration leaves special nodes uncovered, here an extra toleration leaves them unprotected. Treat tolerations like RBAC wildcards.

The second failure is applying NoExecute when you meant NoSchedule. NoSchedule steers only new pods, while NoExecute evicts running pods without a matching toleration, optionally after tolerationSeconds expires. A maintenance taint with the wrong effect drains production in seconds. Unlike affinity, which never evicts, the taint effect carries destructive power, so rehearse on one node and watch kubectl get pods -o wide before repeating across the pool.

The third failure is spelling drift between taint and toleration. dedicated=gpu:NoSchedule never meets dedicated=GPU:NoSchedule, and a toleration naming PreferNoSchedule does not cover a NoSchedule taint. Specialists sit Pending with had taint that the pod didn't tolerate while the pool looks healthy. Compare both sides with kubectl describe pod against kubectl get node <name> -o jsonpath='{.spec.taints}', and keep the canonical triple in one documented place.

Check it quickly

Save the following as taint_toleration_sim.py and run python3 taint_toleration_sim.py. It uses only the standard library, touches no cluster, and models the rules above: key plus operator plus value plus effect, with soft taints never blocking.

def tolerates(taint, tol):
    """One toleration matches one taint on key, value, operator, and effect."""
    if tol.get("key") and tol["key"] != taint["key"]:
        return False
    if tol.get("operator", "Equal") == "Equal":
        if tol.get("value") != taint.get("value"):
            return False
    effect = tol.get("effect", "")
    if effect and effect != taint.get("effect"):
        return False
    return True


def schedulable(taints, tolerations):
    """A pod schedules only if every hard taint is tolerated."""
    for taint in taints:
        if taint["effect"] == "PreferNoSchedule":
            continue
        if not any(tolerates(taint, tol) for tol in tolerations):
            return False
    return True


def check(actual, expected, message):
    if actual != expected:
        raise AssertionError(f"{message}: got {actual}, want {expected}")
    print(f"OK: {message} -> {actual}")


def main():
    gpu = {"key": "dedicated", "value": "gpu", "effect": "NoSchedule"}
    exact = {"key": "dedicated", "value": "gpu", "effect": "NoSchedule"}
    check(schedulable([gpu], [exact]), True, "exact match schedules")
    check(schedulable([gpu], []), False, "missing toleration is repelled")
    wide = {"key": "dedicated", "operator": "Exists", "effect": "NoSchedule"}
    other = {"key": "dedicated", "value": "spot", "effect": "NoSchedule"}
    check(schedulable([other], [wide]), True, "Exists toleration matches any value")
    check(schedulable([gpu], [{"key": "dedicated", "value": "spot", "effect": "NoSchedule"}]), False, "wrong value is repelled")
    evict = {"key": "maintenance", "value": "drain", "effect": "NoExecute"}
    any_effect = {"key": "maintenance", "value": "drain"}
    check(schedulable([evict], [any_effect]), True, "empty effect matches any effect")
    check(schedulable([evict], []), False, "untolerated NoExecute would evict")
    soft = {"key": "cost", "value": "cheap", "effect": "PreferNoSchedule"}
    check(schedulable([soft], []), True, "PreferNoSchedule never blocks")
    print("PASS: 7 scenarios; allow and repel verified")


if __name__ == "__main__":
    main()

The seven scenarios prove an exact toleration allows scheduling while a missing one repels, an Exists toleration matches any value but a wrong value is refused, an empty effect covers NoExecute, an untolerated NoExecute would evict, and PreferNoSchedule never blocks. Expected output ends with PASS: 7 scenarios; allow and repel verified.

The assertions passed with Python 3.12.6 on macOS. The script checks matching logic only, not capacity, affinity, or live taints. Before changing production, compare pod events against the node’s actual taint list.

Further reading

Sources