Golang: Jaeger and OpenTelemetry

Currently, in Software Development most of us already apply the concept of microservices. Microservices is a distributed system based on domain/areas, where each area is divided into several services and connected to each other.

What must be considered from microservices is performance and consistently data, so to make sure our function is running well and complete the data we need some tools monitoring that can show the process from A to Z in our systems, for example, when we need to check the performance of login API or if we need to find the error in our code.

We usually look up logs monitoring tools to find root causes, but sometimes seeing logs one by one is very heavy and exhausting. In my experience, I’m using Jaeger and OpenTelemetry tools for traces, and logs for my project on the server.

So what are Jaeger and OpenTelemetry and how to implement them in Go ?

Jaeger

Based on the Jaeger website :

Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and troubleshooting microservices-based distributed systems, including:

Distributed context propagation

Distributed transaction monitoring

Root cause analysis

Service dependency analysis

Performance / latency optimization

Uber published a blog post, Evolving Distributed Tracing at Uber, where they explain the history and reasons for the architectural choices made in Jaeger. Yuri Shkuro, creator of Jaeger, also published a book Mastering Distributed Tracing that covers in-depth many aspects of Jaeger design and operation, as well as distributed tracing in general.

OpenTelemetry

OpenTelemetry is instrumentation for emitting traces, metrics, and logs. Otel will be sent to observability backends, in this case, we using Jaeger.

Implementation

let’s say we are making a system login application where there are several steps to do it.

Hit API Login(router) -> Login Handler -> Login Services -> Login Repository

Setup Jaeger Config
Jaeger using docker

docker run -d — name jaeger \
 -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
 -e COLLECTOR_OTLP_ENABLED=true \
 -p 6831:6831/udp \
 -p 6832:6832/udp \
 -p 5778:5778 \
 -p 16686:16686 \
 -p 4317:4317 \
 -p 4318:4318 \
 -p 14250:14250 \
 -p 14268:14268 \
 -p 14269:14269 \
 -p 9411:9411 \
 jaegertracing/all-in-one:1.37 

• The entry point for collector HTTP will be http://localhost:14268/api/traces
• You can then navigate to http://localhost:16686 access the Jaeger UI.

Install OpenTelemetry in Go

go get go.opentelemetry.io/otel \
go.opentelemetry.io/otel/trace \
go.opentelemetry.io/otel/sdk \
go.opentelemetry.io/otel/sdk/resource \
go.opentelemetry.io/otel/exporters/jaeger\
go.opentelemetry.io/otel/semconv\

Create pkg OpenTelemetry and Create Implementation
Adding wrapper package for OpenTelemetry.

import (
 “go.opentelemetry.io/otel/attribute”
 “go.opentelemetry.io/otel/exporters/jaeger”
 “go.opentelemetry.io/otel/sdk/resource”
 sdktrace “go.opentelemetry.io/otel/sdk/trace”
 semconv “go.opentelemetry.io/otel/semconv/v1.7.0”
)
func NewExporter(url string) (*sdktrace.TracerProvider, error) {
 // Your preferred exporter: console, jaeger, zipkin, OTLP, etc.
 exp, err := jaeger.New(jaeger.WithCollectorEndpoint(jaeger.WithEndpoint(url)))
 if err != nil {
 return nil, err
 }
 tp := sdktrace.NewTracerProvider(
 // Always be sure to batch in production.
 sdktrace.WithBatcher(exp),
 // Record information about this application in a Resource.
 sdktrace.WithResource(resource.NewWithAttributes(
 semconv.SchemaURL,
 semconv.ServiceNameKey.String(“flip-ewallet”),
 attribute.String(“environment”, “development”),
 attribute.Int64(“ID”, 1),
 )),
 )
 return tp, nil
}

Implement package OpenTelemetry
In main.go we will init OpenTelemetry and add tracer. example component-main

// opentelemetry
tp, err := opentelemetry.NewExporter(cfg.Jaeger.Endpoint)
if err != nil {
   log.Fatal(err)
}
otel.SetTracerProvider(tp)

tp.Tracer("component-http")

Implement tracer in 1 cycle of endpoint, for this case we assume we already implement API login (router, handler, repository). So in every step, we will put a tracer so that we can see the traces in Jaeger UI.

Login Handler :

func (u userHandler) Login(w http.ResponseWriter, r *http.Request) {
   ctx := context.Background()
   ctx, span := otel.Tracer("users").Start(ctx, "UserHandler_Login")
   defer span.End()
.....
}

Login Services :

func (u userService) Login(ctx context.Context, username string) (string, error) {
 ctx, span := otel.Tracer(“users”).Start(ctx, “UserService_Login”)
 defer span.End()
.....
}

User Repository :

func (u *userRepo) Find(ctx context.Context, username string) (*dao.User, error) {
   ctx, span := otel.GetTracerProvider().Tracer("users").Start(ctx, "UserRepository_Find")
   defer span.End()
.....
}

JWT Service :

func JWTSigning(ctx context.Context, key string) (string, error) {
   ctx, span := otel.Tracer("users").Start(ctx, "UserService_jwtSigning")
   defer span.End()
.....
}

Test Endpoint and Access Jaeger UI
Go to http://localhost:16686 to see the Operation and Logs.

We can see in operation of one cycle of login endpoint from A to Z and process, so we can see traces, logs, and performance in every step.

With Jaeger and OpenTelemetry we can monitor our service operations easily.

References :

• Jaeger
• OpenTelemetry