ebpf

opentelemetry

instrumentation

Integrating manual with automatic instrumentation

This blog shares the results of combining manual with automatically generated spans to get the best of both worlds.

Integrating manual with automatic instrumentation
Author
/Ron.pngRon Federman
Nov 30 2023

We're developing Odigos, an open-source project for effortless distributed tracing. See more at https://github.com/keyval-dev/odigos.

Distributed tracing is essential for monitoring and improving the performance of applications in distributed environments. To achieve this, applications need to be instrumented to allow for the collection of data on their behavior and performance. There are two main approaches to instrumenting applications for distributed tracing:

  • Manual instrumentation - the developer manually adds code to the application to create spans and export them to the tracing backend.
  • Automatic instrumentation - the developer uses an agent to automatically generate spans for common operations such as HTTP requests and database queries.

While manual instrumentation provides the most control over the data collected, it is also the most time-consuming and error-prone. Automatic instrumentation, on the other hand, is easy to use and requires minimal code changes. However, it is usually limited to the open-source libraries that are supported by the agent.

In this post, we introduce a hybrid approach that seamlessly combines the precision of manual instrumentation with the comfort, efficiency, and performance of automatic instrumentation. We'll explore how to easily combine both approaches to produce meaningful data with minimal code changes.

Simple example application

To demonstrate this feature we will use the following simple HTTP server application.

The server has a single endpoint for which it will generate a single random integer between 1 and 6 (roll a dice) and will save the result in a database. This can be seen in the following snippet:

func (s *Server) rollDice(w http.ResponseWriter, req *http.Request) {
	// Roll a dice
	n := s.rand.Intn(6) + 1

	_, err := s.db.ExecContext(req.Context(), "INSERT INTO results (roll_value) VALUES (?)", n)
	if err != nil {
		panic(err)
	}

	// Write the result to the response
	fmt.Fprintf(w, "%v", n)
}

Automatically generated trace

For the above code, we can leverage Go-auto-instrumentation (https://github.com/open-telemetry/opentelemetry-go-instrumentation) to generate a trace for each roll dice operation.

Since HTTP handling and database queries are common operations, these libraries from the Go standard library are automatically instrumented.

We can gain quite a lot of information from the generated trace such as the SQL query, the HTTP endpoint, and the timing of the operations.

Automatic instrumentation

Combining with manual spans

In many cases, the users would like to add some custom information related to their business logic and internal functions. This level of flexibility is not currently possible with automatic instrumentation.

However, we can achieve a solution using an integration of manually created spans with those created automatically.

In the following code snippet, we used the OpenTelemetry API for Go in order to create a span within the HTTP handler. In our example, we added the value of the dice as an attribute for the span.

import (
    ...
    "go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
    ...
)
...

var tracer = otel.Tracer("rolldice")

...

func (s *Server) rollDice(w http.ResponseWriter, req *http.Request) {
	ctx, span := tracer.Start(req.Context(), "roll")
	defer span.End()

	// Roll a dice
	n := s.rand.Intn(6) + 1

	_, err := s.db.ExecContext(ctx, "INSERT INTO results (roll_value) VALUES (?)", n)
	if err != nil {
		panic(err)
	}

	span.SetAttributes(attribute.Int("roll.value", n))

	// Write the result to the response
	fmt.Fprintf(w, "%v", n)
}

An important thing to note is that only the OpenTelemetry API is being imported which means the code changes are minimal.

Automatic with manual

As a result, the generated trace combines the manually created span with the automatically created ones. We can see that the manual span was inserted in between the HTTP and database spans. This is due to the way we are passing Go's context. We are using the context from the HTTP request to generate the internal span, and the context returned by tracer.Start as the context for the database query.

This approach combines the best of both worlds. We can easily add custom information to the trace with minimal code changes while still benefiting from the automatic instrumentation.

Another benefit of this approach is the performance gains achieved by using eBPF-based automatic instrumentation as described in this blog post.

If you want to learn more about how you can generate distributed traces instantly check out our GitHub repository. We'd really appreciate it if you could throw us a ⭐👇
https://github.com/keyval-dev/odigos

Related posts

Better application discovery with Kubernetes labels

Better application discovery with Kubernetes labels

Discover how Odigos simplifies application discovery using Kubernetes labels and provides more flexible instrumentation options.

Better application discovery with Kubernetes labels

Eden Federman

May 16 2023

Introducing the new Odigos UI

Introducing the new Odigos UI

Discover the enhanced Odigos UI, designed to simplify application discovery and instrumentation management for your observability pipeline.

Introducing the new Odigos UI

Eden Federman

Aug 22 2023

Odigos Version v0.1.4

Odigos Version v0.1.4

Explore the latest features, improvements, and additions in Odigos version v0.1.4, including ARM support and new destinations.

Odigos Version v0.1.4

Eden Federman

Feb 13 2023