Sending OpenTelemetry data to Google Cloud Storage

In this blog post, we will show you how to send OpenTelemetry data to Google Cloud Storage using Odigos.

Since version v0.1.4, Odigos supports sending OpenTelemetry data to Google Cloud Storage (GCS). For the first time, you can use storage services like GCS to store your OpenTelemetry data, and build custom solutions on top of it.

Creating a GCS bucket

First, you need to create a GCS bucket. You can do it using the Google Cloud Console or using the gsutil command line tool. For example, to create a bucket named otel-data in the us-central1 region, run the following command:

gsutil mb -p <project-id> -c regional -l us-central1 gs://otel-data

Creating a GKE cluster

Next, you need to create a GKE cluster with permissions to write to GCS. You can do it using the Google Cloud Console or using the gcloud command line tool. For example, to create a cluster named otel-cluster in the us-central1 region, run the following command (replace PROJECT_ID with your project ID):

gcloud container clusters create otel-cluster \
--region us-central1 \
--workload-pool=PROJECT_ID.svc.id.goog

Installing demo application

For this demo, we will use the emojivoto application. You can install it by running the following command:

kubectl apply -k kubectl apply -k github.com/odigos-io/opentelemetry-go-instrumentation/docs/getting-started/emojivoto

Installing Odigos

After installing the demo application, you need to install Odigos to send data to GCS. The simplest way to install Odigos is via the helm chart by executing the following commands:

helm repo add odigos https://odigos-io.github.io/odigos-charts/
helm install my-odigos odigos/odigos --namespace odigos-system --create-namespace

Granting permissions using workload identity

To grant permissions to the cluster, you need to create a service account and bind it to the cluster. You can do it using the Google Cloud Console or using the gcloud command line tool. For example, to create a service account named otel-sa and bind it to the cluster, run the following commands:

gcloud iam service-accounts create otel-sa
gcloud iam service-accounts add-iam-policy-binding \
  --role roles/storage.objectCreator \
  --member "serviceAccount:otel-sa@<project-id>.iam.gserviceaccount.com" \
  otel-sa@<project-id>.iam.gserviceaccount.com

Finally, you need to bind the odigos-gateway service account to the GCP service account, run the following command:

kubectl annotate serviceaccount -n odigos-system odigos-gateway \
  iam.gke.io/gcp-service-account=otel-sa@<project-id>.iam.gserviceaccount.com

Sending data to GCS

The last step is to configure Odigos to send data to GCS. This can be done easily by accessing the Odigos UI and creating a new Google Cloud Storage destination. Make sure to specify the bucket name you created earlier.

Summary

In this blog post, we showed you how to send OpenTelemetry data to Google Cloud Storage using Odigos, an open-source observability control plane. Odigos supports more that 20 different destinations, including cloud providers like Azure and Google Cloud, as well as open-source solutions like Loki, Elasticsearch, and Prometheus.

LEARN MORE