Understanding Kubernetes: Events

Understanding Kubernetes: Events

In the last article of my Kubernetes series, we looked at debugging broken workloads. We saw that Kubernetes Events are often one of the first places to look when something doesn’t behave as expected. But what exactly are Events? Where do they come from? And what can they actually tell us? Let’s take a closer look.

What are Kubernetes Events?

Kubernetes Events are objects that record something that happened in the cluster. They are generated by different Kubernetes components when they observe a particular situation or take an action. A typical Event could look like this:

Normal   Scheduled
Successfully assigned default/my-app-7f8c6d9f7b-x2k9m to node-2

Or:

Warning  FailedScheduling
0/3 nodes are available: 3 Insufficient memory.

Events usually contain information such as a type, a reason, a message, and information about the object the Event relates to.

The two Event types we’ll encounter most often are:

  • Normal — something happened that is generally expected or informational
  • Warning — something happened that may indicate a problem

A Warning Event doesn’t necessarily mean that the workload is completely broken, though. But it’s a signal that something happened we might want to look into.

Where do Events come from?

One useful thing to understand is that Events don’t come from a single central component. Different parts of Kubernetes can create them. For example:

  • the scheduler reports scheduling decisions and scheduling failures
  • the kubelet reports things happening on a node, such as container starts, image pulls or probe failures
  • controllers report changes and failures related to the resources they manage
  • other Kubernetes components and controllers, as well as custom controllers and operators, can create Events for their own actions

This is why Events can give us a useful view into what Kubernetes itself is doing.

For example, when creating a Pod, we might see:

Normal   Scheduled
Successfully assigned default/web-6f7d9c8b6f-4xk2m to node-2

Normal   Pulling
Pulling image "nginx:1.29"

Normal   Pulled
Successfully pulled image "nginx:1.29"

Normal   Created
Created container: nginx

Normal   Started
Started container nginx

Reading these Events in order gives us a rough idea of what happened: The Pod was scheduled, the image was pulled, the container was created and then started. If something goes wrong somewhere along the way, the Events can help us see where the process stopped.

Looking at Events

We’ve already seen Events as part of kubectl describe. For example:

kubectl describe pod my-app

will give us the Events section at the bottom of the output. But we can also query Events directly via

kubectl get events

or sort them by the time they occurred:

kubectl get events --sort-by=.lastTimestamp

This can be particularly useful when we’re trying to understand what happened across several objects and not just one Pod. For example

LAST SEEN   TYPE      REASON             OBJECT                     MESSAGE
2m          Normal    Scheduled          pod/web-...                Successfully assigned...
2m          Normal    Pulling            pod/web-...                Pulling image...
2m          Normal    Pulled             pod/web-...                Successfully pulled...
2m          Normal    Created            pod/web-...                Created container...
2m          Normal    Started            pod/web-...                Started container...
30s         Warning   Unhealthy          pod/web-...                Readiness probe failed...

gives us a rough timeline of what Kubernetes observed within our cluster.

In a very busy cluster, we can also narrow down events to either a particular object

kubectl get events -n my-namespace --field-selector involvedObject.name=my-app

or a particular namespace:

kubectl get events -n my-namespace

A real-world example: FailedScheduling

Let’s say we create a Deployment and notice that its Pods never leave Pending:

kubectl get pods

NAME                     READY   STATUS    RESTARTS   AGE
my-app-7f8c6d9f7b-x2k9m  0/1     Pending   0          2m

The Pod isn’t running, but that doesn’t tell us why.

So we check the Events

kubectl get events --sort-by=.lastTimestamp

and find:

Warning   FailedScheduling
0/3 nodes are available: 3 Insufficient memory.

We can see that the problem isn’t the application container - it hasn’t even started yet. Instead, the scheduler couldn’t find a node where the Pod could be placed. This is a good example of why Events are useful: they can tell us what Kubernetes tried to do before the workload ever got to the point where application logs exist.

Another example: image pulling

Events can also help when a container can’t start because Kubernetes can’t pull its image. For exampl, we might see events like:

Normal   Pulling
Pulling image "my-registry.example.com/my-app:v42"

Warning  Failed
Failed to pull image "my-registry.example.com/my-app:v42"

Warning  Failed
Error: ImagePullBackOff

This Event points us towards the image itself - maybe the image name or tag is incorrect or there are authentication issues with the registry. Or something completely different. While the Event itself doesn’t necessarily tell us the complete root cause, it still gives us the next place to investigate.

Events can tell a story

One of the things I find most useful about Events is that they can help us reconstruct what Kubernetes was doing. Imagine we have a Pod that isn’t serving traffic.

The Events might show:

Normal    Scheduled
Successfully assigned default/web-... to node-2

Normal    Pulled
Successfully pulled image "my-app:v1"

Normal    Created
Created container: web

Normal    Started
Started container web

Warning   Unhealthy
Readiness probe failed: HTTP probe failed with statuscode: 503

This tells us quite a lot: The Pod was scheduled successfully. The image was pulled. The container started. But the readiness probe is failing. That means we want to look at the application, its configuration, or the readiness probe itself.

But Events are not logs - and don’t stick around forever

It’s important not to confuse Events with application logs. Events are short, high-level records of things Kubernetes noticed or did. They aren’t intended to contain the complete history of everything that happened, but they will give us clues where to check first.

There is another important limitation: Events are not a permanent audit log. Kubernetes Events have a limited lifetime and can be removed after a certain amount of time. The exact retention behaviour depends on the Kubernetes setup and configuration.

Which also means: Events are primarily useful for understanding recent activity. If something happened several days ago, we shouldn’t expect to be able to query the cluster and reconstruct the entire history from Events. This is one reason why Events shouldn’t replace proper logging, monitoring or auditing.

Not every Warning Event means our application is broken

Kubernetes is constantly reconciling the desired state with the current state. During that process, things can temporarily fail and then recover. For example, we might see a warning because a probe failed once, followed shortly afterwards by a successful probe. Or a controller might temporarily encounter a problem and recover on its next reconciliation loop.

That also means not every Warning needs immediate action. What is important is to put it into context:

  • What object generated the Event?
  • What is the reason?
  • When did it happen?
  • Is it happening repeatedly?
  • Did something happen afterwards that resolved the problem?
  • Does the current state of the object still indicate a problem?

Summing up

Kubernetes Events are one of those small pieces of Kubernetes that become really useful once we know what we’re looking at. They can show us what Kubernetes components noticed, what actions they took, and where something started to go wrong. They can be especially useful when a workload hasn’t even reached the point where application logs are available.

But Events are clues, not a complete history. They have limited retention, don’t replace logs or metrics, and often won’t tell us the full root cause. We could hence think about them like this: Events help us understand what Kubernetes saw happening. And when something goes wrong, that can be exactly the piece of the puzzle we need. :)

header image created by buddy ChatGPT