News About Odigos

Introducing Odigos!

Introducing Odigos! an Open-Source Observability Control Plane that allows organizations to create & maintain their observability pipeline. Odigos allows applications to provide traces, metrics, and logs within minutes - without making any code changes.

The Problem

Real-time Observability is essential to deliver a seamless digital experience. It’s tough to get a holistic view of your entire application when it’s running on hundreds of pods deployed to dozens of nodes distributed across the world. Observability has become mission critical for teams who need to track, manage, and optimize the performance and availability of these environments.

When integrated correctly, Observability tools can monitor and troubleshoot issues by centralizing your data and providing smarter insights into key metrics on performance, usage, and user behavior. Observability tools should support the languages and frameworks you use, integrate easily with your container platform and the other tools you use, including any communication or alert.

The problem is that implementing, maintaining and scaling are contninuous tasks. Without proper execution and continuous configuration, observability tools are limited at best and many times ineffectual.

For R&D teams, applying Observability capabilities in the cloud requires specific skill sets, particularly considering the shift to OpenTelemetry & eBPF. Organizations must ensure they have access to specific skills sets within a limited and fiercely competitive talent pool. Learning SDK’s, adding auto-instrumentation (for each language), writing code, deploying, and maintaining collectors – this all requires considerable time and knowledge that most organizations do not possess

In order to work around these integration issues, some large observability tool vendors offer their own agents, providing their own tailor-made solution. The problem with this is that it creates a vendor lock-in, by using proprietary agents that ingest and store data in a proprietary format. As enterprises increasingly seek compatibility with open source standards, as well as the ability to share and access data across departments, having data locked in proprietary silos hinders those efforts and raises costs.

Odigos Observability Control Plane provides a comprehensive and fully automated solutions that allows orginaizations to build their Observability pipline within minutes. Odigos is focused on 3rd-party integrations, open source, open standards, and overall a more consolidated approach, reducing the complexity involved in combining multiple observability software vendors and open-source software solutions.

The technology behind Odigos

Odigos automatically detects the programming language of every application in your cluster and performs automatic instrumentation accordingly. For compiled languages (like Go), eBPF is used in order to instrument the application. For virtual machine languages (like Java) OpenTelemetry is used. In addition, pipelines created by Odigos follow best practices such as: persisting API keys as Kubernetes secrets, use of minimal collector images, and much more.

The ability to install, configure and maintain an open-source, agnostic Observabiity Control Plane will impower organizations of all sizes the ability to adopt the right tools for them at any given time, and add more tools as needed.

Odigos makes Observability simple and accessible to all.

How to Build an End to End Open Source Observability Solution on Kubernetes

In this blog post, you will learn how to build a complete open-source solution for extracting and shipping traces, metrics, and logs, and correlating between them. The solution proposed uses open-source tools: Grafana, Prometheus, Tempo, and Loki as an observability backend stack, and Odigos as an observability control plane.

Theory

If you are new to observability, or just interested in the difference between monitoring and observability we recommend watching this short video by the creator of OpenTelemetry. In short, distributed traces, metrics, and logs, with the ability to correlate between one signal to another, are the best practice for debugging production issues when working with microservices-based applications. This is exactly what we are going to achieve for our demo application.

There is no need to learn any new technologies in order to implement and enjoy observability. With some basic Kubernetes commands — you are ready to get started.

Solution Overview

We are going to deploy 3 different systems on our Kubernetes cluster:

  • Target application — we will use a microservices-based application written in Java and Python. (for example applications with more programming languages and more complex architecture, use the example from the Odigos getting started guide)
  • Observability backend — we are going to use the following applications to store and analyze our observability data: - Grafana: for dashboards and visualization of the data - Prometheus: for storage of metrics data - Loki: for storage of logs data - Tempo: for storage of distributed tracing data
  • Observability control plane — we will use Odigos for automatic instrumentation of our applications (automatic extraction of traces, metrics, and logs), collectors deployment, and configuration.

open source solution

Prerequisites

The following tools are required to run this tutorial:

  • Kind to run a local Kubernetes cluster with Docker container nodes.
  • Kubernetes CLI (kubectl) to run commands against Kubernetes clusters.
  • Helm CLI to install Odigos and the observability backend using helm charts.

Creating the Kubernetes cluster

Create a new local Kubernetes cluster, by running the following command:

kind create cluster

Installing Target Applications

We will install a fork of bank-of-athnos, an example of a bank application created by Google. We use a modified version without any instrumentation code to demonstrate how Odigos automatically collects observability data from the application.

Deploy the application using the following command:

kubectl apply -f https://raw.githubusercontent.com/keyval-dev/bank-of-athnos/main/release/kubernetes-manifests.yaml

Installing Observability Backend

As there is currently no one database that can store traces, logs, and metrics, we will deploy three different databases alongside Grafana as a visualization tool.

The following helm chart deploy Tempo (traces database), Prometheus (metrics database), and Loki (logs database) as well as a preconfigured Grafana instance with those databases as data sources. Install the helm chart by executing:

helm install --repo https://keyval-dev.github.io/charts observability oss-observability --namespace observability --create-namespace

Connecting Everything Together Using Odigos

Now that our test application is running, our observability databases are deployed and ready to receive data, the last piece of the puzzle is to extract and ship logs, metrics, and traces from our applications to the observability databases. The simplest and easiest way to do it is by using Odigos- a control plane for observability data. Install Odigos via the helm chart by executing the following commands:

helm repo add odigos https://keyval-dev.github.io/odigos-charts/
helm install my-odigos odigos/odigos --namespace odigos-system --create-namespace

After all the pods in the odigos-system namespace are running, open the Odigos UI by running the following command:

kubectl port-forward svc/odigos-ui 3000:3000 -n odigos-system

And navigate to http://localhost:3000 to access the UI.

Selecting Applications

There are two ways to select which applications Odigos should instrument:

  • Opt out (recommended): Instrument everything, including every new application that will be deployed going forward. Users can still manually mark applications that should not be instrumented
  • Opt In: Only instrument the applications selected manually by the user.

Select Applications

For this tutorial, we recommend choosing the opt out mode.

Choosing Destinations

Choose destinations The next step is to tell Odigos how to reach the three databases that we deployed earlier. Add the following three destinations:

Tempo

Prometheus

In order to add another destination, select Destinations from the sidebar and click Add New Destination

Loki

Exploring The Data

Wait a few seconds for Odigos to finish deploying the required collectors and instrument the target applications. You can monitor the progress by running

kubectl get pods -w

Wait for all the pods to be in Running state (especially notice the transactionservice application which has a slow startup time).

The last step is to explore our observability data in Grafana. We can now see and correlate metrics to traces to logs in order to dive deeply into how our application behaves.

Connecting to Grafana

Port forward to your Grafana instance by running:

kubectl port-forward svc/observability-grafana -n observability 3000:80

And navigating to http://localhost:3000

  • Enter admin as the username
  • For the password enter the output of the following command:
kubectl get secret -n observability observability-grafana -o jsonpath=”{.data.admin-password}” | base64 --decode

Service Graph

Let’s start by viewing a service graph of our microservices application:

  1. Go to the Explore from the sidebar
  2. Select Tempo as datasource
  3. Choose the Service Graph tab
  4. Run the query

Service Graph

Metrics

Now let’s view some metrics. Click on the contacts node from the service graph and choose Request rate Metrics

A graph similar to the following should be presented: Request rate graph

There are many more metrics that Odigos collect and can be queried easily from the Prometheus data source, check out this document for the full list.

Traces

Click on the contacts application again in the Service Graph but this time choose Request Histogram. In order to correlate metrics to traces, we will use a feature called exemplars. To show exemplars:

  1. Open the options menu
  2. Turn on exemplars
  3. Notice that the histogram is now added with green diamonds.

Exemplars

Hover over one of the added points and click Query With Tempo. A trace similar to the following should be presented: Trace

In this trace, you can see exactly how much time each part of the entire request took. Digging into one of the sections will show additional information such as database queries.

Logs

To further investigate specific action you can simply query the relevant logs by pressing on the small document icon. Press on the document icon next to the balancereader to show the relevant logs: Log

Summary

We have shown how easy it is to extract and ship logs, traces, and metrics using only open-source solutions. In addition, we were also able to generate traces, metrics, and logs from an application within minutes. We now also have the ability to correlate between the different signals: We correlated metrics to traces and traces to logs. We now have all the needed data to quickly detect and fix production issues in our target applications.

What’s Next?

Notice that the observability backend that we installed is not suited for production usage. For high volumes of data, it is recommended to persist those databases to cloud storage like S3 or use a managed offering.

Managing collectors on K8s – why we chose the OpenTelemetry collector for Odigos

As organizations deploy and maintain modern applications and microservices, Observability technologies (eBPF & OpenTelemetry) are needed to expose their health & security. If you’re using these technologies for Tracing, Metrics or Logs in a production software system, you’re either sending telemetry data directly from you application or using a collector.

Configuring OpenTelemetry to send traces and metrics directly from your app to your desired destination has the benefit of not running any additional software (no ports to configure, no processes to manage). The downside is that your application must manage the entire workload of sending your data in a production environment (running resource-intensive services), as well as the need to continuously configure and update your code as you scale (or add new applications).

Offloading the job of sending data directly from the application can be a huge help. Deploying collectors allows you to process telemetry and send it out to any destination. The OpenTelemetry Collector, an open-source project backed by the CNCF, offers a vendor-agnostic implementation that uses pipelines to receive, process, and export trace data. It is a stand-alone service chaining together plugins to form an observability pipeline that buffers data, scrubs it, and sends to one or more open-source or commercial back-ends.

The OpenTelemetry Collector is configurable. When tasked with deploying Collectors in a production environment, Devops team will find two repositories containing dozens of modules, with many integration possibilities. The problem is that it’s hard to know what module will best fit your changing requirements. Without proper implementation and continuous configuration of your collectors, observability tools will be limited at best and many times ineffectual

Collectors Figure 1

There are solutions out there. Many of the large commercial observability tool vendors provide different open-source solutions that will ease implementation. Often, implementation will include vendor specific API’s that are not compatible with other tools, further complicating any efforts of working with more than one vendor.
Actual implementation and integration are still a lengthy process that requires knowledge and experience to get done right. It’s enough to take a quick look at the “getting started” pages of the different observability vendors out there to see how complicated this integration still is.

Odigos, an open-source observability control plane just released in July ’22, provides auto-instrumentation as well as a collector management capabilities. The platform is actually comprised of four fundamental parts: an Instrumentor, Scheduler, Auto-scaler & the UI.

Collectors Figure 2

This is the way it works:

  • First, Odigos recognizes the different programming languages used in each application
  • Then it auto-instruments each application using different (open-source) tools for each programming language
  • The auto-scaler deploys and configures the OpenTelemetry collectors
  • The scheduler assigns applications discovered by the instrumentor to the collector’s pipeline create by the auto-scaler
  • Finally, a UI provides a simple interface to manage the whole process

Odigos’s advantages include its easy installation and its ability to continuously adapt as well as scale to size. The auto-instrumentation will allow Odigos to recognize new application the minute it starts running on your system, and automatically deploy the appropriate collectors. No changes to the code, no developer action needed and no Devops team required! A new application will produce signals without any additional interventions needed.

Summary

Integrating Observability Tools require companies to make several changes and configurations to their applications and infrastructure. Current solutions still demand tech teams to continuously deploy, configure and update collectors, as well as learn SDK’s, add auto-instrumentation, change application code – all requiring new and different developer skillsets.

With the latest advancements in observability technologies (eBPF & OpenTelemetry), companies can now build their observability pipeline with complete end-to-end open-source solution. Using open-source (Jaeger, Grafana, etc.) or commercial back-ends together with the open-source Odigos observability control plane, companies can get full end-to-end visibility. Ultimately, Odigos provides a simple installation that allows companies to get traces, metrics, and logs within minutes – and takes care of managing (configuring, deploying and scaling) your OpenTelemetry collectors.

Effortless distributed tracing for Go applications

Today we are excited to announce version v0.6 of Go automatic instrumentation and version v0.1.3 of Odigos. These versions focus on improving observability for Go developers.

Odigos is now able to automatically perform context propagation on any Go application. An excellent example can be seen below in the trace created by Odigos:

Automatically created distributed trace

As you can see, context is propagated automatically between different libraries (an HTTP request that caused an invocation of a gRPC request) and across processes (marked in different colors).

Bringing distributed traces to everyone

We believe the automatic creation of distributed traces is a key part of making observability simpler. The idea of distributed tracing first started with a paper called Dapper that Google published in 2010. 12 years later, and still, the majority of companies have yet to implement distributed tracing.

One of the reasons implementation has been prolonged, is that distributed tracing is especially hard to implement when developing Go applications. Due to the static binaries produced by the language, until recently automatic instrumentation was just not possible. Fortunately, using eBPF we can now solve this.. Using an eBPF feature called uprobes allows us to extend binaries with minimal effect on performance (for example, the above trace took 3.1ms to complete).

New features

In addition to context propagation this version introduces the following improvements:

  • Removed dependency on goroutine id: The previous versions used goroutine id to correlate spans. Following feedback from the community that this is not an id we should count on, this version does not use goroutine id anymore. Instead, we are now using the context.Context object for span correlations. This is similar to how it happens when you write manual instrumentation and is much more stable.
  • More accurate timestamps: We improved the way we calculate the start and end timestamps of every span.
  • Bug fixes and performance improvements.

We believe in community

Both Odigos and Go automatic instrumentation are developed in open source with the community. Recently, the Go automatic instrumentation was donated to OpenTelemetry and is now developed together with the OpenTelemetry community in the form of a new SIG called Go Automatic Instrumentation.

We can’t wait to hear what you think. We invite you to follow us or reach out to us in the following ways:

Odigos - allowing a choice between the 5 leading open-source observability backends

As observability grows in popularity, new open-source solutions continue to emerge, allowing companies to implement end-to-end observability solutions without spending a dime.

This growing trend of observability open-source software is in line with developments across other technology sectors. Already, web, mobile, and cloud solutions are increasingly built on open-source infrastructure. Future architectures are highly likely to be based on open source, as they are today in mobile solutions (Android platform), web solutions, where most websites are based on open-source technology; and cloud solutions, where almost all are based on open-source solutions as well.

Observability vendors offer solutions that process signals (distributed traces, metrics & logs) that are displayed on dashboards with performance KPI’s and insights reflecting the health and security of any applications. The challenge has been implementing these technologies to deliver their full potential of observability capabilities. Looking at the “getting started” page of most observability vendors, it is clear why implementing these solutions is a challenging process that includes: changing code, using SDK’s to auto-instrument applications, configuring & deploying collectors and finally, continuously managing them to scale (for new applications, new collectors, etc.).

Enter Odigos – an open-source, agnostic observability control plane that automates the whole process. With a quick & easy installation replacing days, if not weeks, of integration, companies can have all their application produce distributed tracing, metrics and logs within minutes. As an agnostic observability enabler, Odigos allows you to connect to any observability tool vendor out there, open-source or not.

Being big believers, supporters, and maintainers in the open-source community, Odigos is happy to announce that we are now integrated with five open-source observability tools vendors.

Prometheus, Jaeger, Loki, Tempo & Signoiz are all now supported by Odigos and can be installed within minutes.

What does this mean

It means companies have a choice of vendors when implementing an end-to-end open-source observability solution. As shown in the table below, between the five vendors there are currently three different open-source solutions for processing and analyzing Distributed Tracing, Metrics & Logs.

Traces Metrics Logs
Prometheus
Tempo
Loki
Jaeger
SigNoz

Additionally, it’s now easy for companies can work with more than one Observability vendor. Do you like Prometheus for you Metrics but want to use Tempo for your distributed tracing? Not a problem. As the industry continues to grow and observability becomes ubiquitous, additional tools with new functionality are flourishing (Security, etc.) and we expect companies in the future to use more the one observability vendors.

This is just a first step. Odigos will continue to integrate open-source solutions, allowing our customers a gateway to choose the solution that best fits their needs. As additional functionality is developed, Odigos will continue supporting multiple vendors enabling companies to prepare and take advantage of the new observability-based solutions being developed

The open-source vendors supported by Odigos

Prometheus – Prometheus is an open-source software application used for event monitoring and alerting. It records real-time metrics in a time series database built using a HTTP pull model, with flexible queries and real-time alerting.

Jaeger – Jaeger is open-source software for tracing transactions between distributed services. It’s used for monitoring and troubleshooting complex microservices environments

Loki - Loki is a log aggregation system designed to store and query logs from all your applications and infrastructure.

Tempo - Tempo is an open source, easy-to-use, and high-scale distributed tracing backend.

SigNoz - Supports all three telemetry signals under a single dashboard. You can correlate your application metrics and traces, and use it for things like infrastructure monitoring

New Releases