Our vision is seamless automatic instrumentation, provided as a service by the underlying compute platform. This will only happen if we can automatically instrument any container image without any prior knowledge of the application.
In the past few weeks, we have been working hard on improving the language detection in Odigos. We are excited to share the latest improvements with you today.
Let's assume that we have a pod named my-app that we want to detect the language for. Odigos (specifically the Instrumentor) would do the following:
This worked well at first, but as Odigos became more popular, we started to see some issues with this approach.
The first issue we encountered was the OutOfPods error. Some managed Kubernetes providers (e.g. EKS) have a limit on the number of pods that can be scheduled on a node. This limit is usually set to 110 pods per node. When this limit is reached, Kubernetes will not allow any new pods to be scheduled on the node, and it will return the OutOfPods error. Failure to schedule pods on a specific node can happen for multiple reasons, and caused Odigos to fail to detect languages for applications running on that node.
The second issue we encountered was the slow language detection. Creating new pod and deleting it after the language detection is complete takes time. When dealing with large clusters, this can cause a significant delay in the language detection process.
The last issue we encountered was the need for elevated privileges. This method required the instrumentor component to have a ClusterRole that allows it to delete pods in any namespace. This is not ideal, and it can be a potential security risk.
To solve the issues mentioned above, we rewrote the language detection mechanisem to be integrated into the Odiglet. Odiglet is a small agent we developed that runs on every node in the cluster and is responsible for providing nodes with an instrumentation virtual device.
Moving the language detection to Odiglet solved all the issues mentioned above. Detecting languages is now much faster, and it does not require permissions to delete pods.
Try out the latest version of Odigos, and let us know what you think!
What are log pipelines, why we need them, what purpose do they server and how to design them?
Explore the latest features, improvements, and additions in Odigos version v0.1.4, including ARM support and new destinations.
Explore our predictions for the evolution of distributed tracing, a critical component in achieving comprehensive observability, as we foresee widespread adoption and groundbreaking advancements.