The new Source of truth: introducing persistent, stateless instrumentation with Source objects

We are excited to announce a change to the way you control instrumented workloads with Odigos: Source objects.

Beginning in Odigos v1.0.144, the new Source object is a Custom Resource that allows declarative control over which workloads and namespaces are instrumented in your cluster. This is an evolution from the current approach in which you enable or disable instrumentation by setting the odigos-instrumentation=enabled label directly on your workload.

An example Source object looks like this:

apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: my-source
  namespace: default
spec:
  workload:
    name: my-app
    namespace: default
    kind: Deployment

With these objects, we can provide the same functionality and control that's offered today with several benefits, which we'll explain below.

But first, a few things to note:

• Sources are namespaced objects, created in the same namespace as the workload they represent. This gives application teams the independence to own their own instrumentation as well as providing flexible RBAC control for system admin, DevOps, and SRE teams.
• Sources identify a workload by name, namespace, and kind. This allows Odigos to uniquely identify the workload to be instrumented. In the future, we hope to add wider identification capabilities such as LabelSelectors to allow you to define custom application topologies.
• Sources are entirely decoupled from applications. Instrumenting a workload with Sources means that there is no change required to the workload's definition. This allows Sources to persist beyond the lifecycle of a workload, idempotently enabling workload instrumentation even if it is deleted and re-deployed.
• Sources exist 1:1 with applications. Only one Source can exist for each workload (except in the case of namespace instrumentation – more on that later).
• Source workload fields are immutable once created. When a Source is linked to a workload, it is linked to that workload forever (until the Source is deleted).

Why this change?

In Odigos today, instrumenting your applications is as simple as applying a label to a workload or namespace. So why make the switch to a resource?

While the simplicity of instrumentation labels is certainly beneficial, this approach also has its drawbacks. When talking to users about this, some issues became clear:

• Using a label on the workload creates a stateful relationship on the workload's definition, meaning that if the workload is modified then care must be taken to persist the label for instrumentation to remain effective. This causes problems with continuous deployment systems that might overwrite labels by default. It also means that distributed teams need to be aware of their auto-instrumentation, which shouldn't be a concern for them.
• Relying on the label's value means that our backend needs to not only monitor the presence of the label, but also changes to its value. This creates an edge-based system that is difficult for Kubernetes operators to resiliently consume at scale.
• Kubernetes labels are intended for identifying and grouping workloads, not as a control mechanism. In contrast, CRDs are a common config pattern for users to interact with Kubernetes operators.

Switching to a resource-based control mechanism addresses these by leveraging idiomatic Kubernetes operator patterns to reconcile the state of the cluster in a truly level-based design. It creates a user-facing config surface to declare the desired state of the cluster through a persistent and reliable Source of truth.

With Sources, you are declaring that a workload (or namespace) should be instrumented, forever, as long as that Source exists. This means the workload can be deleted, re-deployed, or modified in any way and Odigos will still continue to recognize it.

This decoupled persistence is critical to the role of Odigos as a low-overhead, non-invasive tool for auto-instrumentation.

How to use sources?

Functionally, Source objects provide full feature parity with the auto-instrumentation controls in Odigos today. That includes:

• Instrumenting individual workloads
• Instrumenting entire namespaces
• Excluding individual workloads from namespace instrumentation

Instrumenting individual workloads

Say you have a simple app consisting of two deployments: frontend and backend. To instrument both of these, you will create two Source objects:

apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: frontend
  namespace: default
spec:
  workload:
    name: frontend
    namespace: default
    kind: Deployment
---
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: backend
  namespace: default
spec:
  workload:
    name: backend
    namespace: default
    kind: Deployment

When these objects are created, Odigos will find these workloads, detect their runtimes, and automatically instrument them just as it does today.

You can confirm that these Sources were created with kubectl:

$ kubectl get sources
NAME      WORKLOAD  KIND        NAMESPACE
frontend  frontend  Deployment  default
backend   backend   Deployment  default

Instrumenting entire namespaces

Instrumenting an entire namespace is similar to instrumenting individual workloads, except it only requires one Source object for the entire namespace. Following the above example, that would be:

apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: myapp-namespace
  namespace: default
spec:
  workload:
    name: default
    namespace: default
    kind: Namespace

In this case, a Source object must set kind: Namespace with the name and namespace equal to the namespace to instrument. This will instrument all of the possible workloads in the namespace under the single Source:

$ kubectl get sources
NAME            WORKLOAD  KIND      NAMESPACE
myapp-namespace default   Namespace default

You can also create an individual workload Source in the same namespace. For example, you could also create the above frontend Source, and the list for the namespace would now look like this:

$ kubectl get sources
NAME            WORKLOAD  KIND        NAMESPACE
myapp-namespace default   Namespace   default
frontend        default   Deployment  default

Doing so will keep the frontend deployment instrumented even if the myapp-namespace Source is deleted.

Excluding individual workloads from namespace instrumentation

Similar to the last example, a Source can exclude individual workloads from namespace-wide instrumentation. This is done by setting disableInstrumentation: true on the Source.

apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: backend-excluded
  namespace: default
spec:
  disableInstrumentation: true
  workload:
    name: backend
    namespace: default
    kind: Deployment

Now, if the namespace is instrumented, the backend deployment will not be instrumented (as long as the Source is present).

Working with Sources

When a Source object is created, Odigos will automatically apply labels to that Source that mirror the workload field. For example:

apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: backend
  namespace: default
  labels:
    odigos.io/workload-name: backend # added automatically
    odigos.io/workload-namespace: default # added automatically
    odigos.io/workload-kind: Deployment # added automatically
spec:
  workload:
    name: backend
    namespace: default
    kind: Deployment

This is intended to provide flexible management of Sources using native Kubernetes LabelSelectors:

$ kubectl get sources -l odigos.io/workload-kind=Deployment
NAME      WORKLOAD  KIND        NAMESPACE
frontend  frontend  Deployment  default
backend   backend   Deployment  default

This approach was chosen to support older versions of Kubernetes that do not yet have default support for selectable fields.

As mentioned earlier, keep in mind that Sources are immutable and must exist 1:1 with the workload or namespace they instrument. Odigos will validate both of these on creation and any attempts to update existing Sources.

Continued support for label-based instrumentation

As mentioned above, the current support for label-based instrumentation is convenient and we understand that some users may prefer it. While we encourage users to migrate to Source objects, we will continue to provide support for the current label-based approach as a shorthand method for creating Sources.

Going forward, when Odigos finds a workload or namespace that is labeled with odigos-instrumentation=enabled, the backend will automatically create a matching Source object.

This operation is idempotent, meaning that re-creating the same workload with odigos-instrumentation=enabled will not create a new Source object if one already exists for that workload.

Similarly, labeling a workload or namespace with odigos-instrumentat=disabled will create a Source for that workload with disableInstrumentation: true.

However, both of these actions will log a warning in the Instrumentor.

Future plans

While Source objects today are functionally simply a way to enable or disable instrumentation, in the future we plan to expand this object to provide more functionality such as:

• LabelSelector-based workload selection
• Source-to-destination grouping configuration
• Instrumentation status conditions

With features like these in mind, we plan for Sources to be the main user-facing Kubernetes resource within Odigos going forward.

Check out our documentation on Sources to learn more.