Faster and Better - Introducing the New Odigos Language Detector

Discover the latest enhancements in language detection within Odigos, including improved speed and efficiency.

Faster and Better - Introducing the New Odigos Language Detector
/eden.jpgEden Federman
May 17 2023

Our vision is seamless automatic instrumentation, provided as a service by the underlying compute platform. This will only happen if we can automatically instrument any container image without any prior knowledge of the application.

In the past few weeks, we have been working hard on improving the language detection in Odigos. We are excited to share the latest improvements with you today.

How the old language detection worked

Let's assume that we have a pod named my-app that we want to detect the language for. Odigos (specifically the Instrumentor) would do the following:

  1. Create a new pod named my-app-lang-detection, and schedule it to run on the same node as my-app.
  2. This new pod should have the hostPID flag set to true, which means that it shares the same PID namespace as my-app. This allows the new pod to examine the processes running inside my-app.
  3. The language detection pod determines the language of my-app by examining the following:
    • Process name
    • Linked libraries
    • Environment variables
    • Language specific tools (e.g. go version for Go applications)
  4. Once the language detection pod determines the language of my-app, it reports it back to the instrumentor.
  5. The instrumentor then deletes the language detection pod, and instruments my-app using the correct instrumentation.

This worked well at first, but as Odigos became more popular, we started to see some issues with this approach.

OutOfPods error

The first issue we encountered was the OutOfPods error. Some managed Kubernetes providers (e.g. EKS) have a limit on the number of pods that can be scheduled on a node. This limit is usually set to 110 pods per node. When this limit is reached, Kubernetes will not allow any new pods to be scheduled on the node, and it will return the OutOfPods error. Failure to schedule pods on a specific node can happen for multiple reasons, and caused Odigos to fail to detect languages for applications running on that node.

Slow language detection

The second issue we encountered was the slow language detection. Creating new pod and deleting it after the language detection is complete takes time. When dealing with large clusters, this can cause a significant delay in the language detection process.

Elevated privileges

The last issue we encountered was the need for elevated privileges. This method required the instrumentor component to have a ClusterRole that allows it to delete pods in any namespace. This is not ideal, and it can be a potential security risk.

A better way to detect languages

To solve the issues mentioned above, we rewrote the language detection mechanisem to be integrated into the Odiglet. Odiglet is a small agent we developed that runs on every node in the cluster and is responsible for providing nodes with an instrumentation virtual device.

Moving the language detection to Odiglet solved all the issues mentioned above. Detecting languages is now much faster, and it does not require permissions to delete pods.

Try out the latest version of Odigos, and let us know what you think!

If you want to learn more about how you can generate distributed traces instantly check out our GitHub repository. We'd really appreciate it if you could throw us a ⭐👇

Related posts

Introduction to Log Pipelines

Introduction to Log Pipelines

What are log pipelines, why we need them, what purpose do they server and how to design them?

Introduction to Log Pipelines

Amir Blum

Oct 15 2023

Odigos Version v0.1.4

Odigos Version v0.1.4

Explore the latest features, improvements, and additions in Odigos version v0.1.4, including ARM support and new destinations.

Odigos Version v0.1.4

Eden Federman

Feb 13 2023

Distributed Tracing in 2025: What the future holds

Distributed Tracing in 2025: What the future holds

Explore our predictions for the evolution of distributed tracing, a critical component in achieving comprehensive observability, as we foresee widespread adoption and groundbreaking advancements.

Distributed Tracing in 2025: What the future holds

Eden Federman

Nov 9 2022