Learn About Tracing, Metrics, and Logs in the Golang Application Without Changing a Single Line of Code

This article introduces how to leverage Alibaba Cloud's open-source Golang Agent to obtain tracing, metrics, and logs in a Golang application without modifying the code.

By Musi

Background

Previously, monitoring Golang applications usually required certain code modifications, such as introducing SDKs like OpenTelemetry. Such modifications often led to disagreements between the O&M and R&D teams. The O&M team said, "If the R&D team does not access the relevant monitoring, it is difficult for us to understand the application's running status and make corresponding operations." The R&D team, on the other hand, would complain, "Originally, We already have many business requirements and need to add various instrumentation in the code. If the SDK version changes, the code needs to be adjusted again." This reduced the efficiency of the Golang application accessing observability. In this article, Alibaba Cloud's open-source Golang Agent is used to help users obtain various observation data generated by their applications without changing a single line of code. This approach improves the happiness of O&M and R&D teams.

Quick Start

Golang developers usually use the go build command to build the source code into a binary package that is executed when developing applications. Alibaba Cloud's open-source Golang Agent can be considered an alternative to the go build. When compiling a Golang application, users can replace the go build with the Golang Agent binary package to compile a binary Golang program with observability capabilities. Below is a detailed introduction to the build and usage process of Golang Agent:

Step 1: Download/build the Golang Agent binary package

First, you can visit https://github.com/alibaba/opentelemetry-go-auto-instrumentation page to download the latest version of the Golang Agent binary package.

Suppose you want to experience the latest unreleased capabilities of the Golang Agent. In that case, you can also compile the latest Golang Agent binary package from the main branch with the following command:

# Download the project to your local computer first.
git clone https://github.com/alibaba/opentelemetry-go-auto-instrumentation.git
# Enter the root directory of the agent.
cd opentelemetry-go-auto-instrumentation
# Compile the Golang Agent binary package.
make clean && make build
chmod u+x ./otel

After executing the above command, a binary file named Otel will appear in the project's root directory. This indicates that the Golang Agent binary package is compiled.

Step 2: Compile Golang applications using the Golang Agent binary package

After you have the Golang Agent binary package, you can use it to compile the binary program of your Golang application instead of using the go build. The Golang Agent provides the corresponding Golang Demo in the https://github.com/alibaba/opentelemetry-go-auto-instrumentation/tree/main/example directory:

cd ./example/demo
../../otel go build .

Take the build of the demo directory under example as an example. The original build command may be the go build. Now you need to add a compilation prefix ../../otel go build. This is similar to adding sudo before the original command in a Linux system. After executing the above command, you can find a Golang binary program with observability in the root directory of the corresponding application.

Step3: Configure the reporting endpoint and start the Golang application

Tracing reporting

First, refer to Step 2 to compile the binary file demo in the example/demo directory.

Before reporting, you can start some dependencies of the application. This application will call MySQL and Redis, so we start these two dependencies first:

docker run -d -p 3306:3306 -p 33060:33060 -e MYSQL_USER=test -e MYSQL_PASSWORD=test -e MYSQL_DATABASE=test -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql:8.0.36
docker run -d -p 6379:6379 redis:latest

Then you can start Jaeger to receive and display the reported tracing information:

docker run --rm --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.53.0

After starting the relevant dependencies, you can configure the relevant reporting endpoints according to Otel's specifications and start the application:

cd example/demo
export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4318" 
export OTEL_EXPORTER_OTLP_INSECURE=true 
export OTEL_SERVICE_NAME=demo 
./demo

After the application is started, the application is continuously requested to generate relevant tracing information:

curl localhost:9000/http-service

Finally, visit http://localhost:16686/jaeger/ui/search to search for the tracing information in Jaeger. You can find that the tracing has been generated and successfully connected. You can fully trace the entire execution path of the request through tracing.

Metric reporting

First, refer to Step 2 to compile the binary file metrics in the example/metrics directory.

Similarly, before reporting, you can start some dependencies of the application. This application will report metric information to Prometheus through the Otel Collector, so we can run the following command to start Otel Collector and Prometheus:

# Enter the metrics directory.
cd example/metrics
# Start Otel Collector and Prometheus.
docker compose up --force-recreate --remove-orphans --detach

After the related dependencies are started, you can configure the related reporting endpoints according to Otel's specifications and start the application:

# Enter the metrics directory.
cd example/metrics
export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4318" 
export OTEL_EXPORTER_OTLP_INSECURE=true
export OTEL_SERVICE_NAME=metrics 
./metrics

After the application is started, the application is continuously requested to generate relevant metric information:

# For Golang GC metrics
curl localhost:9000/gc-metrics
# For Golang Memory metrics
curl localhost:9000/mem-metrics

Finally, visit http://localhost:9090 to search for the relevant metric information in Prometheus. You will find that the relevant metrics have been generated and reported normally. The above metrics can help users better understand Golang's runtime information and the overall status of HTTP requests.

Log association

The Golang Agent supports two modes of log association. If you use a logging framework supported by the Golang Agent, the Agent will automatically inject the traceID and spanID into the logs. In addition, you can manually obtain the traceID and spanID using the Otel SDK to print them into the logs. Below is a brief demonstration of log association in these two modes:

Automatic log association

First, refer to Step 2 to compile the binary file test in the example/log directory. After the application is compiled, you can start it.

# Enter the metrics directory.
cd example/log
./test

After the application is started, the application is continuously requested to print the relevant logs:

curl localhost:9999/log

You will find that the application prints logs as shown below, with the corresponding traceID and spanID information visible at the beginning of the logs:

{"level":"debug","msg":"this is debug message","trace_id":"d62a8fea286cc66de9c68ca17d4faa88","span_id":"7cb6d692769ffd32"}
{"level":"info","msg":"this is info message","trace_id":"d62a8fea286cc66de9c68ca17d4faa88","span_id":"7cb6d692769ffd32"}
{"level":"warn","msg":"this is warn message","trace_id":"d62a8fea286cc66de9c68ca17d4faa88","span_id":"7cb6d692769ffd32"}
{"level":"error","msg":"this is error message","trace_id":"d62a8fea286cc66de9c68ca17d4faa88","span_id":"7cb6d692769ffd32"}
{"level":"debug","msg":"this is debug message","trace_id":"e56a6f1e7ed7af48cce8f64d045ed158","span_id":"def0b8cf10fe8844"}
{"level":"info","msg":"this is info message","trace_id":"e56a6f1e7ed7af48cce8f64d045ed158","span_id":"def0b8cf10fe8844"}
{"level":"warn","msg":"this is warn message","trace_id":"e56a6f1e7ed7af48cce8f64d045ed158","span_id":"def0b8cf10fe8844"}
{"level":"error","msg":"this is error message","trace_id":"e56a6f1e7ed7af48cce8f64d045ed158","span_id":"def0b8cf10fe8844"}

Manual log association

In addition, you can manually obtain the traceID and spanID using the Otel SDK to print them into the logs. The relevant example code is shown below:

package main

import (
  "go.opentelemetry.io/otel/sdk/trace"
  "go.uber.org/zap"
  "net/http"
)

func main() {
  http.HandleFunc("/logwithtrace", func(w http.ResponseWriter, r *http.Request) {
    logger := zap.NewExample()
      // Obtain the corresponding traceID and spanID through the SDK.
      // Note: This usage requires the Golang Agent to compile successfully.
    traceId, spanId := trace.GetTraceAndSpanId()
    logger.Info("this is info message with fileds",
      zap.String("traceId", traceId),
      zap.String("spanId", spanId),
    )
  })
  http.ListenAndServe(":9999", nil)
}

Refer to Step 2 to compile the binary file test in the example/log directory. After the application is compiled, you can start it.

# Enter the metrics directory.
cd example/log
./test

After the application is started, the application is continuously requested to print the relevant logs:

curl localhost:9999/logwithtrace

You can see that the application prints logs as shown below, with the corresponding traceID and spanID information visible at the beginning of the logs:

{"level":"info","msg":"this is info message with fileds","traceId":"92d63797010a2040484222a74c5ce304","spanId":"5a2c84c807a6e12c"}

Summary and Outlook

The Golang Agent successfully solves the cumbersome problem of manual instrumentation in Golang application monitoring and has been commercially launched on Alibaba Cloud public cloud, providing customers with powerful monitoring capabilities. This technology was originally designed to allow users to insert monitoring code easily without changing the existing code, enabling real-time monitoring and analysis of application performance. However, its application areas have exceeded expectations, including service governance, code audit, application security, and code debugging. It has also shown potential in many unexplored areas.

We have decided to open-source this innovative solution and donate it to the OpenTelemetry community. In the future, our code will be migrated to the OpenTelemetry community repository. The solution's open source promotes technical sharing and improvement and helps us continuously explore its potential in more fields with the help of the community.

Community

Learn About Tracing, Metrics, and Logs in the Golang Application Without Changing a Single Line of Code

Background

Quick Start

Step 1: Download/build the Golang Agent binary package

Step 2: Compile Golang applications using the Golang Agent binary package

Step3: Configure the reporting endpoint and start the Golang application

Tracing reporting

Metric reporting

Log association

Automatic log association

Manual log association

Summary and Outlook

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

Managed Service for OpenTelemetry

Cloud-Native Applications Management Solution

Function Compute

Lindorm

A Free Trial That Lets You Build Big!