Understanding Kubernetes: Scheduling, Affinity & Taints

In the previous articles we looked at different parts of running secure workloads in Kubernetes: how Network Policies control communication, how RBAC controls access, and how Pod Security limits what containers are allowed to do.

Now we want to answer the following question: Where should a Pod actually run?

A Kubernetes cluster can consist of many nodes with different properties. So how does Kubernetes decide where a workload should run? This is where scheduling comes in.

And just like many other Kubernetes topics, the default behaviour is often good enough - until you have a real-world requirement. Then questions like these appear:

• Should this workload only run on nodes with GPUs?
• Should two replicas of an application be separated across different nodes?
• Should a workload avoid a specific node?
• Should a database only run in a certain location?

This is where scheduler rules like node selectors, affinity, taints, and tolerations come in handy. So let’s dive in.

The Kubernetes scheduler

The Kubernetes scheduler is the component responsible for finding a suitable node for a newly created Pod. A rough overview of the process can be:

1. A Pod is created.
2. The scheduler checks available nodes.
3. It evaluates constraints and requirements.
4. It selects a node.
5. It assigns the Pod to the selected node.

During evaluation, the scheduler considers many things, including:

• available CPU and memory (based on resource requests)
• node conditions
• existing workloads
• and: scheduling rules defined in the Pod specification

If Kubernetes cannot find a suitable node, the Pod remains in the Pending state. And while the default scheduler does a great job, we sometimes need to give it additional instructions.

nodeSelector

The simplest way to influence scheduling is nodeSelector. Let’s say we have some nodes with SSD storage and want a database workload to run only there.

First, we label the nodes:

kubectl label nodes node-1 storage=ssd

Then we can use this label in our Pod:

apiVersion: v1
kind: Pod
metadata:
  name: database
spec:
  nodeSelector:
    storage: ssd
  containers:
    - name: database
      image: postgres

Now Kubernetes will only consider nodes with the label storage=ssd.

This approach is simple and works well for basic requirements, but has limitations: it is either “this label exists” or “it does not”. There is no way to express more complex rules.

This is where affinity comes in. You might wonder why both nodeSelector and node affinity exist. The short answer is that nodeSelector is the simpler, older mechanism, while node affinity builds on the same idea and offers much more flexibility.

Node affinity

Node affinity is basically the more powerful version of nodeSelector. Instead of just saying: Run this Pod on nodes with this label we can express more flexible requirements, e.g. Run this Pod preferably on nodes with this label or Run this Pod only on nodes matching one of these conditions.

An example could look like this:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: environment
                operator: In
                values:
                  - production
  containers:
    - name: app
      image: nginx

The important part here is requiredDuringSchedulingIgnoredDuringExecution. This tells the scheduler that the rule must be fulfilled (required) and checks need to happen when the Pod is placed (DuringScheduling). IgnoredDuringExecution moreover tells the scheduler to not automatically evict or automatically move the Pod if the label changes later.

A less restrictive option would be e.g. preferredDuringSchedulingIgnoredDuringExecution, meaning we prefer this rule, but the Pod will still be scheduled if no matching node exists. This is useful when you have a preference but do not want workloads to get stuck.

Pod affinity and anti-affinity

So far, we have only looked at nodes. But quite often we also care about where other Pods are running. For example:

• Keep frontend and backend close together for lower latency
• Keep replicas of an application separated to avoid a single node failure

This is where Pod affinity and Pod anti-affinity come in. Pod affinity basically says: “Place this Pod close to other Pods matching this label.” while anti-affinity does, you guessed it already, says the opposite ;)

A common example is spreading replicas:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: frontend
        topologyKey: kubernetes.io/hostname

This tells Kubernetes: “Do not schedule two frontend Pods on the same node.”

The topologyKey defines the failure domain. In this example, the topology is the node itself.

Taints and tolerations

Affinity tells Kubernetes where a Pod should run. Taints work the other way around: they tell Kubernetes where a Pod should not run. A node can have a taint like

kubectl taint nodes node-1 dedicated=database:NoSchedule

so that normal Pods will not be scheduled on it. To allow a specific workload there, we need a matching toleration:

spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: database
      effect: NoSchedule

So while taints keep workloads away from a node, tolerations define which workloads are allowed to be scheduled there. A common example is dedicated nodes for databases, machine learning workloads, or system components.

Topology spread constraints

Another useful scheduling feature is topology spread constraints. They solve a similar problem as anti-affinity: spreading workloads across failure domains.

For example, imagine a cluster with multiple availability zones. We probably do not want all replicas of an application running in the same zone.

A topology spread constraint could look like:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: frontend

This tells Kubernetes: “Keep the number of Pods between zones balanced.” This is especially useful for highly available applications.

When to use what

We have covered quite a few concepts now, so here’s a small cheat sheet before we jump into the demo:

Demo time

Let’s try some of these concepts. First, we create a couple of labels on nodes:

kubectl label nodes <node-name> disk=ssd

Then, we create a Pod that requires this label:

apiVersion: v1
kind: Pod
metadata:
  name: scheduled-demo
spec:
  nodeSelector:
    disk: ssd
  containers:
    - name: nginx
      image: nginx

We apply it

kubectl apply -f pod-with-label.yaml

and check where Kubernetes placed it with

kubectl get pod scheduled-demo -o wide

And yes, the Pod is running on the node with the matching label. 🎉

When we remove the label via

kubectl label nodes <node-name> disk-

the Pod keeps running - as expected, because scheduling decisions are made when the Pod is first scheduled.

Let’s make a final check: What if no node has a label at all? When we now try to add another pod with this label (just rename the Pod in the template above to “scheduled-demo-2”), it’s stuck in Pending, because no node has the required label.

Summing up

Scheduling is Kubernetes deciding where workloads should run. By default, the scheduler makes good decisions based on available resources, but Kubernetes also gives us many ways to influence those decisions. The most important concepts are:

• nodeSelector for simple node selection
• node affinity for more flexible node rules
• pod affinity and anti-affinity for placing workloads relative to each other
• taints and tolerations for keeping workloads away from specific nodes
• topology spread constraints for improving availability across failure domains

And with that, we have covered another important piece of how Kubernetes runs workloads. Next up in this series: Requests, Limits & Resource Management. Stay tuned :)

header image created by buddy ChatGPT