Control Placement with Taints, Tolerations and Affinity

Why this matters

Dedicated GPU nodes should not fill with web pods. Replicas of one service should not share a single rack. System pods must land everywhere including cordoned-looking nodes. Labels plus rules express all of this declaratively, and the scheduler enforces it every placement. The exam turns these into Pending-pod puzzles: the events name the failed predicate if you read them.

Concepts

A taint on a node repels pods unless they tolerate it, with three effects: NoSchedule blocks new placements, PreferNoSchedule discourages them, NoExecute evicts existing pods too (optionally after tolerationSeconds). Control-plane nodes carry a NoSchedule taint by default, which is why user pods avoid them without any configuration on your part.

Tolerations excuse pods from taints by key, operator and effect. They never attract: a tolerating pod may still land anywhere. System DaemonSets tolerate everything precisely so they cover tainted nodes as well.

Affinity attracts. Node affinity constrains which nodes qualify (required rules are hard, preferred rules weight the choice). Pod affinity co-locates with selected pods; pod anti-affinity spreads apart, which is how you keep replicas off the same node or the same zone with topologyKey. Combine them: node affinity for fit (GPU, SSD, zone), anti-affinity for spread.

Worked example

GPU work only on GPU nodes, replicas never sharing a node:

# taint the GPU pool once, per node
kubectl taint nodes gpu-1 gpu=true:NoSchedule
# workload tolerates the taint, requires the label, spreads itself
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - { key: gpu, operator: In, values: ["true"] }
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchLabels: { app: trainer }
      topologyKey: kubernetes.io/hostname
tolerations:
- { key: gpu, operator: Exists, effect: NoSchedule }

Web pods without the toleration never touch gpu-1. Trainer replicas land only on GPU nodes and never two on one host. Remove the toleration and watch them go Pending with a taint event: the error message is the lesson.

Exam notes

Toleration without affinity lands anywhere: admitting is not attracting, and questions exploit the confusion. topologyKey: kubernetes.io/hostname spreads per node; the zone key spreads per zone. And NoExecute evicts running pods, so adding it mid-day restarts workloads: check tolerationSeconds before touching live taints.