odigos

open-source

s3

aws

opentelemetry

Sending OpenTelemetry data to AWS S3

Learn how to generate and send OpenTelemetry data to Amazon Web Services S3 using Odigos, improving your data storage and analysis capabilities.

Sending OpenTelemetry data to AWS S3
Author
/eden.jpgEden Federman
Mar 26 2023

In this blog post, we will show you how to generate and send OpenTelemetry data to Amazon Web Services S3 using Odigos.

Since version v0.1.46, Odigos supports sending OpenTelemetry data to AWS S3. For the first time, you can use S3 to store your OpenTelemetry data, and build custom solutions on top of it.

Creating an S3 bucket

First, you need to create an S3 bucket. You can do it using the AWS Console or using the aws command line tool. For example, to create a bucket named otel-data in the us-east-1 region, run the following command:

aws s3api create-bucket \
--bucket otel-data \
--region us-east-1

Creating an EKS cluster

Next, you need to create an EKS cluster. You can do it using the AWS Console or using the eksctl command line tool. For example, to create a cluster named otel-cluster in the us-east-1 region, run the following command:

eksctl create cluster \
--name otel-cluster \
--region us-east-1 \
--version 1.24 \
--nodegroup-name standard-workers \
--node-type t3.medium \
--nodes 3 \
--nodes-min 1 \
--nodes-max 4 \
--managed

eksctl utils associate-iam-oidc-provider --region=us-east-1 --cluster=otel-cluster --approve

Installing demo application

For this demo, we will use the emojivoto application. You can install it by running the following command:

kubectl apply -k github.com/keyval-dev/opentelemetry-go-instrumentation/docs/getting-started/emojivoto

Installing Odigos

After installing the demo application, you need to install Odigos to send data to S3. The simplest way to install Odigos is via the helm chart by executing the following commands:

helm repo add odigos https://keyval-dev.github.io/odigos-charts/
helm install my-odigos odigos/odigos --namespace odigos-system --create-namespace

Configuring IAM and RBAC

In order to send data to S3, Odigos needs to have the right permissions. Create a new IAM policy named odigos-s3 with the following content:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:DeleteObject"
      ],
      "Resource": ["arn:aws:s3:::otel-data", "arn:aws:s3:::otel-data/*"]
    }
  ]
}

Next, you need to configure the IAM role to be used by Odigos. You can do it by creating a service account and attaching the IAM role to it (replace the account ID with your own):

eksctl create iamserviceaccount \
--name default \
--namespace odigos-system \
--cluster otel-cluster \
--attach-policy-arn arn:aws:iam::123456789012:policy/odigos-s3 \
--override-existing-serviceaccounts \
--region=us-east-1 \
--approve

Sending data to S3

The last step is to configure Odigos to send data to S3. This can be done easily by accessing the Odigos UI and creating a new AWS S3 destination. Make sure to specify the bucket name you created earlier.

Summary

In this blog post, we showed you how to send OpenTelemetry data to AWS S3 using Odigos, an open-source observability control plane. Odigos supports more that 20 different destinations, including cloud providers like Azure and Google Cloud, as well as open-source solutions like Loki, Elasticsearch, and Prometheus.

If you want to learn more about how you can generate distributed traces instantly check out our GitHub repository. We'd really appreciate it if you could throw us a ⭐👇
https://github.com/keyval-dev/odigos

Related posts

Introduction to Log Pipelines

Introduction to Log Pipelines

What are log pipelines, why we need them, what purpose do they server and how to design them?

Introduction to Log Pipelines

Amir Blum

Oct 15 2023

Odigos Version v0.1.4

Odigos Version v0.1.4

Explore the latest features, improvements, and additions in Odigos version v0.1.4, including ARM support and new destinations.

Odigos Version v0.1.4

Eden Federman

Feb 13 2023

Combining Odigos and Jaeger for instant distributed tracing

Combining Odigos and Jaeger for instant distributed tracing

This article explains how to simplify distributed tracing using Odigos and Jaeger for improved observability in microservices-based applications.

Combining Odigos and Jaeger for instant distributed tracing

Eden Federman

Apr 12 2023