Understanding Kubernetes: DNS & Service Discovery

Understanding Kubernetes: DNS & Service Discovery

In the last article we looked at Kubernetes networking fundamentals: Pods receive their own IP addresses, CNIs provide connectivity, and workloads can communicate directly across the cluster.

We also identified a problem: Pods are ephemeral. They get replaced, rescheduled, and assigned new IP addresses over time. This is where Services come in - and we’ll now have the promised deeper look at them.

Services provide stable identities

A Kubernetes Service provides a stable network endpoint in front of one or more Pods. Instead of connecting to individual Pods, applications connect to a Service. Kubernetes then takes care of routing traffic to the Pods behind it.

A simple Service might look like this:

apiVersion: v1
kind: Service
metadata:
  name: backend
spec:
  selector:
    app: backend
  ports:
    - port: 80
      targetPort: 8080

The important part here is the selector:

selector:
  app: backend

This tells Kubernetes: Find all Pods matching this label and route Service traffic to them.

But where does that label come from? Typically from the Pod template of a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: backend
          image: my-app:latest

Whenever Kubernetes finds Pods matching the selector, they automatically become backend targets of the Service. This is also why labels are such a fundamental concept in Kubernetes: Services, Deployments, NetworkPolicies, and many other resources use labels to identify the objects they should work with.

ClusterIP: the default Service type

When you create a Service, Kubernetes assigns it a virtual IP address which is called a ClusterIP.

You can inspect it via kubectl get svc and get the following example output:

NAME      TYPE        CLUSTER-IP      PORT(S)
backend   ClusterIP   10.96.142.23    80/TCP

Unlike Pod IPs, this address remains stable for the lifetime of the Service. So applications inside the cluster can now communicate with 10.96.142.23 without needing to know which Pods currently exist behind it.

Endpoints: the Pods behind the Service

A common misconception is that a Service somehow “contains” Pods. It doesn’t. Instead, Kubernetes continuously maintains a list of backend Pods associated with the Service - called Endpoints.

Let’s assume we have three Pods

backend-abc   10.244.1.5
backend-def   10.244.2.7
backend-ghi   10.244.3.4

and the Service selector matches all three. Kubernetes then creates Endpoint objects that link the Service to those Pod IPs.

You can inspect this relationship directly via kubectl get endpoints backend and get something like:

NAME      ENDPOINTS
backend   10.244.1.5:8080,10.244.2.7:8080,10.244.3.4:8080

Whenever Pods are added, removed, or replaced, Kubernetes updates this list automatically. So Services stay stable even though the workloads behind them are constantly changing.

EndpointSlices

In modern Kubernetes versions, Endpoint information is internally stored using EndpointSlices. Instead of maintaining one potentially huge Endpoint object, Kubernetes distributes backend information across multiple EndpointSlice resources.

You can inspect them using kubectl get endpointslices.

Most users rarely interact with EndpointSlices directly, but they improve scalability significantly in larger clusters. For day-to-day operations, it is enough to know that they are the modern implementation behind Service endpoint tracking.

So where does the routing happen?

At this point we have:

Service
↓
Endpoints
↓
Pods

But something still needs to route traffic. Historically, this job is handled by kube-proxy, running on every node within a cluster. When traffic is sent to a Service IP:

  1. the packet reaches the ClusterIP
  2. networking rules installed by kube-proxy match the Service
  3. a backend Pod is selected
  4. traffic is forwarded to the Pod

Modern CNIs such as Cilium can replace parts of this functionality using eBPF, but the overall concept remains the same: A Service provides a stable endpoint. Traffic is routed to dynamic backend Pods.

Service discovery through DNS

Remembering ClusterIP addresses would be inconvenient. Instead, Kubernetes provides built-in DNS-based service discovery, so every Service automatically receives a DNS record.

For a Service named “backend” inside the namespace “default”, Kubernetes automatically creates backend.default.svc.cluster.local. Luckily, applications usually don’t need the full name - inside the same namespace, running curl http://backend is enough.

The component responsible for this is CoreDNS. It runs inside the cluster and continuously watches the Kubernetes API. Whenever Services are created, modified, or deleted, CoreDNS becomes aware of those changes and can answer DNS queries accordingly.

Headless Services

Not every workload wants load balancing. Some applications need to know about the individual Pods behind a Service. This is particularly common for distributed systems such as databases, Kafka, Elasticsearch, or ZooKeeper.

For these cases, Kubernetes provides Headless Services, which basically means creating the Service without a ClusterIP:

spec:
  clusterIP: None

Instead of returning a single Service IP, DNS then returns the individual Pod addresses - allowing applications to discover and communicate with specific cluster members directly.

Demo time: DNS resolution inside the cluster

Let’s verify ourselves. We create a Deployment with kubectl create deployment nginx-demo --image=nginx and expose it by running kubectl expose deployment nginx-demo --port=80

Now we can start a temporary BusyBox Pod:

kubectl run dns-test \
 --image=busybox:1.36 \
 -it --rm --restart=Never -- sh

Inside the Pod, let’s run nslookup nginx-demo and we’ll get something like:

Name:     nginx-demo.default.svc.cluster.local
Address:  10.96.142.23

We can also test connectivity by running wget -qO- http://nginx-demo - and receive the “Welcome to nginx” message.

All of that without using any Pod IP or manual DNS config - Kubernetes automatically provided both name resolution and traffic routing for us. 🎉

Summing up

Services solve one of the fundamental problems in Kubernetes: Pods are temporary, but applications need stable communication endpoints. As we’ve seen, Kubernetes combines here several components:

  • Services provide stable identities
  • Selectors define which Pods belong to a Service
  • Endpoints and EndpointSlices track backend Pods
  • kube-proxy handles traffic routing
  • CoreDNS provides name resolution

Together, these components create the abstraction that allows applications to simply connect to http://backend without caring which Pods are currently running behind it.

In the next article we’ll look at how traffic enters the cluster from the outside world using NodePorts, LoadBalancers, Ingress and the Gateway API. Stay tuned :)

header image created by buddy ChatGPT