We're developing Odigos, an open-source project for effortless distributed tracing. See more at https://github.com/keyval-dev/odigos.

Distributed tracing is essential for monitoring and improving the performance of applications in distributed environments. To achieve this, applications need to be instrumented to allow for the collection of data on their behavior and performance. There are two main approaches to instrumenting applications for distributed tracing:

• Manual instrumentation - the developer manually adds code to the application to create spans and export them to the tracing backend.
• Automatic instrumentation - the developer uses an agent to automatically generate spans for common operations such as HTTP requests and database queries.

While manual instrumentation provides the most control over the data collected, it is also the most time-consuming and error-prone. Automatic instrumentation, on the other hand, is easy to use and requires minimal code changes. However, it is usually limited to the open-source libraries that are supported by the agent.

In this post, we introduce a hybrid approach that seamlessly combines the precision of manual instrumentation with the comfort, efficiency, and performance of automatic instrumentation. We'll explore how to easily combine both approaches to produce meaningful data with minimal code changes.

Simple example application

To demonstrate this feature we will use the following simple HTTP server application.

The server has a single endpoint for which it will generate a single random integer between 1 and 6 (roll a dice) and will save the result in a database. This can be seen in the following snippet:

func (s *Server) rollDice(w http.ResponseWriter, req *http.Request) {
	// Roll a dice
	n := s.rand.Intn(6) + 1

	_, err := s.db.ExecContext(req.Context(), "INSERT INTO results (roll_value) VALUES (?)", n)
	if err != nil {
		panic(err)
	}

	// Write the result to the response
	fmt.Fprintf(w, "%v", n)
}

Automatically generated trace

For the above code, we can leverage Go-auto-instrumentation (https://github.com/open-telemetry/opentelemetry-go-instrumentation) to generate a trace for each roll dice operation.

Since HTTP handling and database queries are common operations, these libraries from the Go standard library are automatically instrumented.

We can gain quite a lot of information from the generated trace such as the SQL query, the HTTP endpoint, and the timing of the operations.

Combining with manual spans

In many cases, the users would like to add some custom information related to their business logic and internal functions. This level of flexibility is not currently possible with automatic instrumentation.

However, we can achieve a solution using an integration of manually created spans with those created automatically.

In the following code snippet, we used the OpenTelemetry API for Go in order to create a span within the HTTP handler. In our example, we added the value of the dice as an attribute for the span.

import (
    ...
    "go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
    ...
)
...

var tracer = otel.Tracer("rolldice")

...

func (s *Server) rollDice(w http.ResponseWriter, req *http.Request) {
	ctx, span := tracer.Start(req.Context(), "roll")
	defer span.End()

	// Roll a dice
	n := s.rand.Intn(6) + 1

	_, err := s.db.ExecContext(ctx, "INSERT INTO results (roll_value) VALUES (?)", n)
	if err != nil {
		panic(err)
	}

	span.SetAttributes(attribute.Int("roll.value", n))

	// Write the result to the response
	fmt.Fprintf(w, "%v", n)
}

An important thing to note is that only the OpenTelemetry API is being imported which means the code changes are minimal.

As a result, the generated trace combines the manually created span with the automatically created ones. We can see that the manual span was inserted in between the HTTP and database spans. This is due to the way we are passing Go's context. We are using the context from the HTTP request to generate the internal span, and the context returned by tracer.Start as the context for the database query.

This approach combines the best of both worlds. We can easily add custom information to the trace with minimal code changes while still benefiting from the automatic instrumentation.

Another benefit of this approach is the performance gains achieved by using eBPF-based automatic instrumentation as described in this blog post.