Intro into OpenTelemetry
In the last articles, we were looking a lot into Kubernetes specifics - and also touched basic monitoring and how we can use it to understand what is happening in our Kubernetes cluster. We also had a look at some debugging tools to figure out what is going wrong when something breaks.
But what if we know that something is wrong, and our metrics and logs still don’t give us enough information to figure out why?
This is where observability comes in. While monitoring helps us keep an eye on known things we want to measure, observability is about using the information our systems provide to investigate what is happening - including problems we didn’t anticipate beforehand.
And whenever we dig deeper into observability in the cloud native space, we will quickly stumble across OpenTelemetry (OTel).
So what is OpenTelemetry actually doing, and how does it fit into the tooling we already know? Let’s have a look!
What is OpenTelemetry all about?
OpenTelemetry is an open-source, vendor-neutral framework for generating, collecting, and exporting telemetry data. The basic idea is pretty simple: We want our applications to provide useful information about what they are doing, and we want to get that information into the tools we use to investigate problems.
The project focuses on three main types of telemetry: Metrics, Logs and Traces, complementing each other. Metrics help us to identify that something is wrong, logs provide details about what happened, and traces help us understand where a problem occurred and how it affected the rest of our system. We have known these three types of information for quite a while. So what’s new?
The interesting part is how we collect and handle them. OpenTelemetry provides common APIs, conventions, instrumentation libraries and protocols for working with telemetry across different programming languages and systems.
Instead of having to implement a different way of collecting data for every monitoring vendor, we can use a common approach and decide later where we want to send that data.
And that brings us to an important distinction.
OpenTelemetry is not an observability backend
If you have already worked with Prometheus and Grafana, you might wonder where OpenTelemetry fits into the picture. Let’s take a step back and look at the different responsibilities.
Prometheus collects and stores metrics. Loki can store logs, while Tempo or Jaeger can be used for traces. And Grafana helps us explore and visualize the data.
OpenTelemetry does something different. It helps us generate, collect, process and export telemetry. It doesn’t provide dashboards or act as a database for storing all that information. So, for example, we could instrument an application with OpenTelemetry and send its telemetry to a Collector, which then forwards the data to our observability backends.
Why is that useful?
Imagine we decide to switch from one observability vendor to another. If our application is tightly coupled to the old vendor’s instrumentation, we might have quite some work ahead of us.
With OpenTelemetry, we can often keep the instrumentation and change how the data is exported instead. We still need to check which signals and features our new backend supports, of course, but we don’t necessarily have to rework every application.
This separation between collecting telemetry and storing or analyzing it is one of the main reasons why OpenTelemetry has become so important in the cloud native ecosystem.
Understanding the OpenTelemetry architecture
There are a few components we should know about to understand how OpenTelemetry works. Let’s look at the most important ones.
Instrumentation: Where the data comes from
Before we can observe an application, it needs to produce telemetry. This is called instrumentation. There are two common approaches to do this:
Automatic instrumentation uses existing integrations to collect telemetry from supported frameworks and libraries. For example, an HTTP framework integration might automatically record incoming requests, their response status and how long they take.
Manual instrumentation lets us add our own metrics, spans and other telemetry at specific points in the application. This is useful when we want to understand application-specific behavior that generic instrumentation doesn’t know about.
OpenTelemetry provides APIs and SDKs for different programming languages, along with instrumentation libraries for many common technologies. The important part is that your application can use the OpenTelemetry APIs without being tightly coupled to a particular telemetry backend.
Depending on the language and framework, automatic instrumentation can save us quite some work. But it won’t necessarily tell us everything we want to know. Sometimes we need to add our own instrumentation to understand what’s happening inside our application.
The OpenTelemetry Collector
Once our application generates telemetry, we need to get it somewhere. This is where the OpenTelemetry Collector comes in. You can think of it as a component between our applications and the observability backends. It receives telemetry, processes it and forwards it to one or more destinations.
The Collector uses a pipeline with three main component types:
- Receivers accept telemetry from applications or other sources.
- Processors can modify, enrich, filter, or batch the incoming data.
- Exporters send the processed telemetry to an observability backend.
For example, we might receive traces from several applications, add information about the environment, batch the data and forward it to our tracing backend. We can also configure the Collector to send telemetry to multiple destinations.
This is particularly useful when we don’t want every application to deal with all the details of exporting and processing telemetry itself. Instead, we can centralize much of that configuration. And if we want to change our backend later, we might only need to adjust the Collector configuration instead of changing every application.
OTLP: A common language for telemetry
One more term we will encounter quite often is OTLP, short for OpenTelemetry Protocol. It defines how telemetry data is transmitted between components. It supports metrics, logs and traces, giving us a common protocol for sending this information.
An application can for example send its traces to the OpenTelemetry Collector using OTLP. The Collector then forwards the data to a backend using the appropriate exporter.
So we have our instrumentation generating the data, OTLP transporting it, and the Collector processing and forwarding it. The backend then stores or analyzes the data. With that in mind, let’s put the pieces together.
Putting the pieces together
Let’s imagine a simple application running on Kubernetes. The application receives HTTP requests and communicates with a database. Everything seems to work, but suddenly requests take three seconds instead of the usual 200 milliseconds. Our monitoring tells us that response times have increased, but where exactly is the problem?
A possible setup could look like this: Our application generates telemetry using OpenTelemetry instrumentation. The Collector receives it and forwards it to the configured backends. Grafana can then help us explore the data, provided we have configured compatible data sources.
Now we can start investigating our slow requests. Our metrics show that request latency has increased. A trace might reveal that most of the additional time is spent waiting for a database query. And the corresponding logs might tell us that the database is reporting errors or that something unexpected happened.
Instead of jumping between unrelated dashboards and trying to piece everything together manually, we have a better chance of following the request through the different components.
Of course, OpenTelemetry doesn’t automatically solve every observability problem for us. We still need to instrument our applications properly, configure the pipeline and make sure the data we need is actually available.
Running OpenTelemetry in Kubernetes
Now that we know the basic components, let’s look at how we can run the Collector in Kubernetes. After all, we don’t just have applications to observe. We also have nodes, Pods, containers and all the other components that make up our cluster.
The OpenTelemetry Collector supports different deployment patterns. Two common approaches are the agent and gateway patterns.
The agent pattern
In the agent pattern, a Collector runs close to the workloads it observes. In Kubernetes, this is often implemented as a DaemonSet, which runs a Collector pod on each eligible node.
This can be useful for collecting node-local telemetry, receiving data from applications on the same node, or gathering information about the Kubernetes environment.
The exact responsibilities depend on your configuration. For example, collecting container logs from the node requires suitable receivers and access to the relevant log files.
The gateway pattern
In the gateway pattern, applications send their telemetry to a shared Collector service.
Instead of every application having to handle the details of exporting telemetry, the gateway can process and route the data centrally. It can also apply policies such as filtering or sampling before forwarding the data to our backends.
We can combine both approaches, too. For example, node-level Collectors can forward telemetry to a central gateway, which then exports it to the observability backends.
Which approach makes sense depends on our environment, traffic volume and operational requirements. We don’t necessarily need both patterns, and we shouldn’t add extra components just because we can. ;)
A few things to keep in mind
Before we start adding OpenTelemetry to every application in our cluster, there are a few things worth thinking about.
We don’t need all the data. Collecting every possible metric, log and trace can quickly become expensive and make it harder to find the information we actually need. Start with the questions you want to answer and collect the data that helps you answer them.
Telemetry can contain sensitive information. Depending on how we instrument our applications, we might end up collecting personal data, credentials or other information that shouldn’t be exported. We need to think about what we collect and where we filter or redact sensitive attributes.
The Collector needs monitoring, too. The Collector is another component in our infrastructure. We need to make sure it has enough resources, can handle the incoming telemetry and doesn’t silently become a bottleneck in our observability pipeline.
And finally, OpenTelemetry doesn’t replace good instrumentation. A standardized way of collecting data is great, but it won’t magically give us information our applications never exposed in the first place.
Summing up
OpenTelemetry gives us a common way to generate, collect and export telemetry across applications and infrastructure. Its APIs and instrumentation libraries help us collect the data, OTLP provides a common protocol, and the Collector lets us process and route telemetry to our observability backends.
This is particularly useful in Kubernetes environments, where applications and infrastructure consist of many different components that need to work together.
We don’t need to replace our existing monitoring setup or instrument everything at once. We can start with one application, figure out which information would help us investigate problems, and build from there.
And if there is one thing I want you to take away from this article, it is this: OpenTelemetry is not another dashboard or monitoring backend. It is the common layer that helps us get telemetry from our applications to the tools we use to understand them.
In the end, collecting data is only one part of observability. The interesting part is what we can learn from it when something goes wrong. :)