At Keyval, we are on a mission to instrument any application, anywhere, at any scale. To achieve this, we need to make instrumentation as easy and accessible as possible. A key part of this is to not require any code changes by extending containers. In this blog post, we will compare different ways to extend containers, and why we chose to use Kubernetes Device Plugins at Odigos.
Adding files and environment variables to containers after they are deployed is a common practice. For example, you may want to add a configuration file, a secret, or in our case, automatic instrumentation. Kubernetes provides several ways to extend containers:
In the upcoming sections, we will test these methods and see how they compare.
Kubernetes YAMLs are already very verbose. Adding more YAML to extend containers only makes it even more difficult to read and maintain. This also makes it harder to roll back the added changes if something goes wrong. Our first reason for choosing Device Plugins is that they require minimal YAML changes. For example, using a ConfigMap to add a configuration file and environment variables to a container requires the following YAML:
apiVersion: v1 kind: Pod metadata: name: my-pod spec: containers: - name: my-container image: my-container-image + volumeMounts: + - name: python-instrumentation + mountPath: /instrumentation + env: + - name: PYTHONPATH + value: /instrumentation + - name: OTEL_RESOURCE_ATTRIBUTES + value: service.name=myservice + volumes: + - name: python-instrumentation + configMap: + name: python-instrumentation
This is a lot of YAML for a simple task. Device Plugin, allows us to achieve the same result with the following (much shorter) YAML:
apiVersion: v1 kind: Pod metadata: name: my-pod spec: containers: - name: my-container image: my-container-image + resources: + limits: + instrumentation.odigos.io/python: 1
Init containers with shared volume are not shown, but they require even more YAML changes than ConfigMaps and Secrets.
Kubernetes allows to specify security contexts for containers. This allows to run containers as a specific user, group, or with specific capabilities. For example, you may want to run a container as a non-root user to improve security. This is not possible when using init containers with shared volumes, as the shared volume is mounted as root. However, this is possible when using device plugins, as the device is mounted as the user specified in the security context. For this reason, operators like Vault Agent Injector require adding many annotations to work with security contexts.
Automatic instrumentation may not be available on every node. For example, Odigos uses eBPF to auto instrument Go applications. Requiring Linux kernel with eBPF support is a reasonable requirement, but it may not be available on every node. This is not a problem when using device plugins, as the scheduler will only schedule pods that require the device to nodes that have the device. When using init containers or ConfigMaps there is no way to specify this requirement, and the pod may be scheduled to a node that does not support automatic instrumentation.
ConfigMaps and Secrets are great, they are simple to use and work well for straightforward use cases. However, they are not a good fit for more complex cases like automatic instrumentation. Init containers with shared volumes are a bit more flexible, but they require more YAML changes and do not work well with security contexts. Device Plugins are the best option for automatic instrumentation, as they require minimal YAML changes, work well with security contexts, and are scheduler friendly. In the future, device plugins will get even better, when KEP-3063 is implemented.