Understanding Kubernetes: Monitoring Basics

Understanding Kubernetes: Monitoring Basics

In the last article, we looked at autoscaling and how Kubernetes can react to changing workloads. But we haven’t touched on one question yet: How can we see what is actually happening in our cluster? Are our Pods healthy? Is something running out of memory? Are requests getting slower? And if something breaks, how do we figure out what happened?

Hello, Monitoring 👋 And once you start looking into Kubernetes monitoring, you’ll quickly run into terms like metrics, logs, traces, Prometheus, and Grafana.

So let’s start with three common types of information.

Logs, metrics and traces

Metrics are numerical measurements that change over time, for example:

  • CPU and memory usage
  • request rate
  • error rate
  • request latency
  • number of active connections
  • number of running Pods

Metrics are particularly useful when we want to see trends or define conditions that should trigger an alert.

Logs give us more detailed information about individual events, e.g.

2026-09-23 10:42:17 ERROR Failed to connect to database

A metric might tell us that the error rate has increased. A log can give us some context about what actually happened.

And then there are traces.

Traces become particularly useful once an application consists of several services. A single request might go through an API, an authentication service, another backend service and finally a database. A trace lets us follow that request through the system and see where time was spent or where something failed.

I already wrote a bit more about distributed tracing and Jaeger in an earlier article . So if you want to dive deeper into that topic, have a look there.

Where do Kubernetes metrics come from?

There are several places we can get metrics, depending on what we want to know.

If you’ve ever used:

kubectl top pods
kubectl top nodes

you’ve already interacted with the Metrics Server. It provides relatively current CPU and memory usage for Pods and nodes. While that’s useful for things like autoscaling, it’s not really a monitoring solution - we want to collect metrics over time. And we are also interested in other details, such as the state of our Kubernetes objects and metrics of the applications running within the cluster.

For the state of Kubernetes objects, kube-state-metrics is useful, exposing relevant information for us about Pods, Jobs, Deployments, etc. Applications can expose their own metrics as well, for example:

http_requests_total
http_request_duration_seconds
database_connections_active
queue_messages_waiting

So, roughly speaking, we have:

Metrics Server
└── current resource usage
    CPU / memory

kube-state-metrics
└── state of Kubernetes objects
    Pods / Deployments / Jobs / ...

Application
└── application-specific metrics
    requests / errors / latency / queues

Having these different data sources, this still leaves us with the question: How can we collect those metrics, keep them over time and make them queryable?

This is where Prometheus comes in.

Prometheus

Prometheus is one of the most common monitoring tools in the Kubernetes world. Its job is basically: collect metrics → store them → let us query them. Prometheus usually does this by scraping metrics endpoints exposed by the systems we’re monitoring.

A metric might look something like:

http_requests_total{method="GET",status="200"} 15432

This tells us that 15,432 successful GET requests have been counted for this particular metric and set of labels. Prometheus stores these values as time series, so we can look at how they change over time. And it comes with its own query language: PromQL.

For example,

rate(http_requests_total[5m])

asks Prometheus to calculate the rate at which requests are happening over the last five minutes.

Or we might want to look at container CPU usage:

rate(container_cpu_usage_seconds_total[5m])

You don’t need to become a PromQL expert just to get started with monitoring. But once you start creating useful dashboards and alerts, understanding the basics becomes pretty handy.

And where does Grafana fit in?

This leaves us with the last of our terms from above - Grafana. While Prometheus is great at storing and querying metrics, we do not want to spend our day writing PromQL queries. ;)

Grafana helps us here, as it can use Prometheus as a data source and turn the metrics into dashboards and graphs. For example, we could have a dashboard showing:

  • CPU and memory usage
  • node capacity
  • Pod restarts
  • request rates
  • error rates
  • latency

Instead of looking at individual numbers, we can see how they change over time. But there is a small trap here: A dashboard isn’t automatically useful just because it contains a lot of graphs.

If you have 47 panels but don’t know what you’re actually looking for, you’ve mostly created a very fancy wall of numbers. ;) The interesting part is deciding which questions the dashboard should help you answer.

What about alerts?

Having useful dashboards in Grafana is a great starting point, but we do not want to stare at them 24/7. Instead, we can define alerts for conditions that need attention. If we, for example, want to get informed if Pods keep restarting, we might define an alert like:

groups:
  - name: kubernetes
    rules:
      - alert: PodRestartingTooOften
        expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Pod is restarting frequently"

With this alert we say: If the number of container restarts increases by more than three within 15 minutes, and that condition continues for ten minutes, fire an alert.

Prometheus evaluates the expression and can hand the alert over to an alerting component such as Alertmanager, which can then route notifications to the appropriate people or systems.

And just like with dashboards, we should be careful here, too. We can create alerts for almost everything, but well, we shouldn’t 😅 A useful alert should tell you that something important needs attention.

So what should beginners actually monitor?

This is where I’d start when setting up initial monitoring:

  • Workload health: Keep an eye on things like Pod restarts, pending / failed Pods, unavailable replicas, or crashed containers.
  • Resource usage: Look out for CPU & memory usage, Node capacity, and resource pressure - especially when using requests, limits, and autoscaling
  • Application health: Depending on the application, watch out for request & error rates, latency, queue size, or active connections
  • Important changes: And finally, think about what should actually wake someone up. E.g. having an unusually high error rate for a longer time could indicate a problem we should take care of.

Summing up

Monitoring helps us understand what is going on within our cluster. But if there is one thing I want to leave you with from reading this article, it is the following: The goal of monitoring isn’t to collect as much data as possible.

The goal is to have enough information to answer questions like: Is my cluster healthy? And my application? Are users experiencing problems? Will I know when something important goes wrong?

To answer these questions we can use a set of tooling: Prometheus gives us time-series metrics. kube-state-metrics gives Prometheus information about Kubernetes object state. Grafana helps us visualize those metrics. Alerts turn important conditions into something that can actually get our attention. And logs and traces give us additional information when we need to understand what’s happening inside the system.

You don’t need all of this on day one. Start with a few useful signals, create alerts for things that actually matter, and add more when you have a reason to.

And as always, there are of course way more details to this - and I haven’t even mentioned OpenTelemetry once 😅 But those topics are for later, not for a basics article :)

Next up in this series is the natural next question after knowing that something is wrong: “Why?” - and we will have a look into debugging tips for broken workloads. Stay tuned :)