Understanding Kubernetes: What Kubernetes actually does

Understanding Kubernetes: What Kubernetes actually does

One thing I keep noticing when working with Kubernetes, especially when preparing workshops, is that a lot of tutorials jump straight into YAML files, deployments, and kubectl commands. And that makes sense, because it gets you productive quickly.

But at some point, I started realizing: A lot of people (and I would count myself in there) use Kubernetes, but often don’t have a clear mental model of what is actually happening behind the scenes. Or they know they do need an Ingress, but do not get the full picture of how networking within Kubernetes works behind the scenes.

So this is the start of a small Kubernetes series where I want to dig a bit deeper into topics that are easy to use in practice, but often harder to fully understand behind the scenes.

We’ll start with the absolute basics of how Kubernetes actually works internally and then gradually move into more detailed topics like networking, scheduling, security, and beyond.

And probably the biggest spoiler right at the beginning: Working with Kubernetes is actually nothing else than talking to an API with a very smart reconciliation system behind it. 🤯

Kubernetes is an API first

Whatever you do with Kubernetes, you interact with its API server. So

  • kubectl apply
  • kubectl get pods
  • kubectl delete deployment
  • Helm charts
  • Operators
  • CI/CD pipelines

all of it is just API calls.

Also YAML manifests are just a structured way of sending data to that API. For example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-demo
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx-demo
    spec:
      containers:
        - name: nginx
          image: nginx

What you are actually doing here is not simply “creating a Deployment”, but sending an API object definition to the Kubernetes API server.

Everything in Kubernetes is an API object

This is the second key idea: Almost everything in Kubernetes is an object exposed through the API.

Pods, Deployments, Services, ConfigMaps, Secrets, Nodes or Namespaces are all just structured API resources stored in Kubernetes.

You can think of Kubernetes as something like this:

Kubernetes API server
        ↓
API Objects (desired state)
        ↓
Controllers that act on those objects

So instead of thinking: “I deploy an application” it becomes: “I create and modify API objects that describe what I want.”

Desired state stored in the API

When you apply a YAML file, Kubernetes does not immediately “do” anything in a traditional way. Instead, it stores your request as data aka the desired state. So statements like

  • “I want 2 replicas of this pod”
  • “This container should run nginx”
  • “This service should exist”

are all structured data inside the API server (backed by etcd). Nothing has necessarily changed in the cluster yet.

So who actually makes things happen?

This is where the second part of the system comes in.

Kubernetes has a set of components that constantly watch the API. The most important ones are controllers. Controllers continuously:

  • read API objects (desired state)
  • check actual cluster state
  • compare both
  • take action if they differ

This loop is the entire foundation of Kubernetes behavior - and it never stops.

Reconciliation: the invisible loop

This process is called reconciliation. And it’s one of those concepts that sounds more complicated than it actually is. Behind the scenes, Kubernetes is constantly trying to make reality match the API objects you defined.

When you define 2 replicas of a Deployment and someone manually deletes a pod, Kubernetes will create a new one, because it notices:

Desired state = 2 pods
Actual state = 1 pod

At this point, Kubernetes starts to look less like a “system you operate” and more like a continuously running control loop over an API, right? And this also means:

  • kubectl apply is just API input
  • Deployments are API objects describing desired behavior
  • Pods are runtime representations created from that state
  • “self-healing” is just reconciliation logic
  • scaling is just updating a number in an API object

So in the end no magic - just reconciliation :)

The control plane: the API brain

At the center of Kubernetes sits the control plane, which is essentially the collection of components responsible for managing and reconciling cluster state.

But instead of just listing components, it’s more useful to look at what actually happens internally once you interact with Kubernetes.

Let’s say we run:

kubectl apply -f deployment.yaml

The following then happens behind the scenes:

kubectl
    ↓
API server
    ↓
etcd stores desired state
    ↓
Controllers notice changes
    ↓
Scheduler assigns workloads to nodes
    ↓
Kubelet starts containers

The API server receives the request

kubectl itself is just a client talking to the Kubernetes API server. So when applying a Deployment, the manifest is sent as an API request to the API server. The API server validates the object and accepts it.

etcd stores the desired state

Once accepted, the Deployment object is stored inside etcd, the distributed datastore backing Kubernetes. At this point, Kubernetes now knows e.g.:

“The desired state is running 2 nginx replicas.”

Controllers react to the new object

Controllers continuously watch the API server for changes. The Deployment controller notices:

Desired replicas: 2
Current replicas: 0

and creates a ReplicaSet object.

The ReplicaSet controller then notices:

Desired pods: 2
Current pods: 0

and creates Pod objects.

The Scheduler assigns Pods to nodes

New Pods initially do not have a node assigned. The Scheduler continuously watches for unscheduled Pods and decides where they should run based on:

  • available resources
  • constraints
  • affinity rules
  • taints and tolerations

Once a suitable node is found, the Scheduler updates the Pod object.

The Kubelet starts the containers

Each Kubernetes node runs a component called Kubelet. It watches for Pods assigned to its node and ensures the containers are actually running via the container runtime (for example containerd). Only now do containers actually start.

And this is also why Kubernetes feels so different from traditional systems. Instead of executing one big deployment command from start to finish, Kubernetes consists of many independent components continuously reacting to API changes and reconciling state.

Summing up

For me, understanding Kubernetes became much easier once I stopped thinking about it as “a platform where containers run” and started seeing it as a system built around API objects, controllers, and reconciliation loops.

In the end, Kubernetes is essentially a huge reconciliation system built around declarative API objects. And once that clicks, it starts feeling less magical - and much more consistent.

This was the first part of my k8s series. Next up: Pods :)

header image created by buddy ChatGPT