By Qingfeng, Guqi, Musi and Ruman
This article aims to provide in-depth technical analysis and practical guidance to help developers understand and apply this innovative technology to improve the monitoring and service governance capabilities of Golang applications. In the following sections, we will further demonstrate how to apply this technology in different scenarios through some practical cases to provide more practical insights.
In the development process of the Go language, despite its reputation for outstanding performance and efficient coding capabilities, significant challenges in costs and technologies remain in application monitoring and service governance. Traditional solutions often require developers to manually adjust source code, which not only increases the workload but also affects the existing architecture, making seamless integration extremely difficult. Especially in complex heterogeneous systems, achieving comprehensive and detailed monitoring and service optimization is almost a time-consuming task that requires expert experience. In this context, seeking a methodology that can minimize intrusion and improve O&M efficiency has become a common goal in the industry.
To solve this problem, Alibaba Cloud Compiler Team[0], ARMS Team, and MSE Team worked together to release and open-source [1] the compile-time automatic instrumentation technology for Go [2]. With its non-intrusive feature, this technology offers Golang applications monitoring capabilities on par with those available for Java. Developers do not need to make any modifications to the existing code. They only need to replace the go build with new compilation commands to implement comprehensive monitoring and governance of Go applications.
In the open-source version, we support 16 mainstream open-source frameworks (38 in the commercial version). At the same time, considering the diverse needs of users, especially those using unsupported frameworks or requiring advanced customization, we further introduced a modular instrumentation extension feature. Users can inject custom code into any target function with zero intrusion through simple JSON configurations. There is no need to modify the code in the original code repository. The code injection is completed through modular instrumentation extension, allowing for finer-grained control, monitoring, governance, and security.
Under normal circumstances, the go build command compiles a Go application through six main steps: source code analysis, type checking, semantic analysis, compilation optimization, code generation, and linking. However, with automatic instrumentation, these steps are preceded by two additional steps: preprocessing and instrumentation.
At this stage, the tool first reads the user-defined rule.json configuration file, which details versions of frameworks or standard libraries that require the instrumentation of custom hook code. The content of the rule.json configuration file is entirely controlled by the user. A typical example is as follows:
[{
"ImportPath": "google.golang.org/grpc",
"Function": "NewClient",
"OnEnter": "grpcNewClientOnEnter",
"OnExit": "grpcNewClientOnExit",
"Path": "/path/to/my/code"
}]This configuration indicates that the user wants to insert the code segments grpcNewClientOnEnter and grpcNewClientOnExit at the entry and exit points of the NewClient function in the google.golang.org/grpc library, respectively. The code for these two functions to be inserted is located at the local path /path/to/my/code.
Next, the tool will analyze the project's third-party library dependencies and match them with the custom instrumentation rules defined in rule.json, while also pre-configuring any additional dependencies required by these rules. When all the preprocessing work is completed, the tool will intercept the regular compilation process and add an instrumentation phase before the compilation of each package.
At this stage, the tool will insert trampoline code for the target function, such as NewClient, based on the configuration in rule.json. The main function of the trampoline code is to serve as a logical springboard to handle exceptions and fill in the context, and eventually, it will jump to the user-defined grpcNewClientOnEnter and grpcNewClientOnExit functions to complete the collection of monitoring data or the governance of service traffic. Since the trampoline code is performance-critical, we also perform a series of optimizations at the AST (Abstract Syntax Tree) level to ensure that its overhead is minimized. Readers interested in the optimization section can access the project’s source code, which will not be further elaborated on here.
Through the above steps, the tool effectively inserts the user-specified code logic while ensuring the integrity of the code's functionality. Subsequently, the tool modifies the necessary compilation parameters and then performs the regular compilation process to generate the final application.
After illustrating the above principles, we will demonstrate how to use the modular extension of Go automatic instrumentation through a few examples.
Taking net/http as an example, many users are interested in request parameters and bodies to locate issues. Here, we use custom instrumentation to describe how to obtain request headers and returned headers.
Step 1: Create a hook folder, initialize it with go mod init hook, and then add the following hook.go code to be injected:
package hook
import (
"encoding/json"
"fmt"
"github.com/alibaba/opentelemetry-go-auto-instrumentation/pkg/api"
"net/http"
)
// Note: The first parameter of the inserted code must be api.CallContext. The subsequent parameters must match the target function's parameters.
func httpClientEnterHook(call api.CallContext, t *http.Transport, req *http.Request) {
header, _ := json.Marshal(req.Header)
fmt.Println("request header is ", string(header))
}
// Note: The first parameter of the inserted code must be api.CallContext. Subsequent parameters must match the target function's return values.
func httpClientExitHook(call api.CallContext, res *http.Response, err error) {
header, _ := json.Marshal(res.Header)
fmt.Println("response header is ", string(header))
}Step 2: Write the following conf.json configuration to instruct the tool on where to inject the hook code: net/http::(*Transport).RoundTrip.
[{
"ImportPath":"net/http",
"Function":"RoundTrip",
"OnEnter":"httpClientEnterHook",
"ReceiverType": "*Transport",
"OnExit": "httpClientExitHook",
"Path": "/path/to/hook" # Change to the local path of the hook code
}]Step 3: Write a test demo. Create a folder and initialize it with go mod init demo, and then add main.go.
package main
import (
"context"
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
// Define the requested URL.
req, _ := http.NewRequestWithContext(context.Background(), "GET", "http://www.aliyun.com", nil)
req.Header.Set("otelbuild", "true")
client := &http.Client{}
resp, _ := client.Do(req)
// Ensure that the body of the response is closed after the function ends.
defer resp.Body.Close()
}Step 4: Switch to the demo directory, and compile and execute the program using the otelbuild tool to verify the effect.
$ ./otelbuild -rule=conf.json -- main.go
$ ./mainThe following output is returned, indicating that the instrumentation is successful:
The example can be found in the following link:
https://github.com/alibaba/opentelemetry-go-auto-instrumentation/tree/main/example/extension/netHttp
The sorting algorithm currently used in the Golang standard library is pdqsort (Pattern-defeating Quick Sort)[3], invented by computer scientist Orson R. L. Peters. pdqsort detects specific patterns in the input data, such as partial sort, ascending sort, or descending sort, and selects the appropriate strategy to address these patterns. For example, when the data is nearly sorted, pdqsort will switch to instrumentation sort. The name pattern-defeating also reflects its special optimization for specific data patterns.
Suppose you are developing a new quicksort algorithm, or find that another quicksort algorithm such as DualPivot Quick Sort is faster under a specific workload. At this time, with the help of the instrumentation tool, you can easily replace the sorting algorithms in the standard library and quickly verify the new algorithm.
Step 1: Create a hook folder, initialize it with go mod init hook, and then add the following hook.go code to be injected:
package hook
import (
"github.com/alibaba/opentelemetry-go-auto-instrumentation/pkg/api"
)
func partition(arr []int, low, high int) (int, int) {
if arr[low] > arr[high] {
arr[low], arr[high] = arr[high], arr[low]
}
lp := low + 1
g := high - 1
k := low + 1
p := arr[low]
q := arr[high]
for k <= g {
if arr[k] < p {
arr[k], arr[lp] = arr[lp], arr[k]
lp++
} else if arr[k] >= q {
for arr[g] > q && k < g {
g--
}
arr[k], arr[g] = arr[g], arr[k]
g--
if arr[k] < p {
arr[k], arr[lp] = arr[lp], arr[k]
lp++
}
}
k++
}
lp--
g++
arr[low], arr[lp] = arr[lp], arr[low]
arr[high], arr[g] = arr[g], arr[high]
return lp, g
}
func dualPivotQuickSort(arr []int, low, high int) {
if low < high {
lp, rp := partition(arr, low, high)
dualPivotQuickSort(arr, low, lp-1)
dualPivotQuickSort(arr, lp+1, rp-1)
dualPivotQuickSort(arr, rp+1, high)
}
}
func sortOnEnter(call api.CallContext, arr []int) {
//Use dual pivot qsort
dualPivotQuickSort(arr, 0, len(arr)-1)
//Skip the original sort algorithm
call.SetSkipCall(true)
}Step 2: Write the following conf.json configuration to instruct the tool to inject hook code into the sort.Ints.
[{
"ImportPath":"sort",
"Function":"Ints",
"OnEnter":"sortOnEnter",
"Path":"/path/to/hook" # Change to the local path of the hook code
}]Step 3: Write a test demo. Create a folder and initialize it with go mod init demo, and then add main.go.
package main
import (
"fmt"
"sort"
)
func main() {
arr := []int{6, 3, 7, 9, 4, 4}
sort.Ints(arr)
fmt.Printf("== %v\n", arr)
}Step 4: Switch to the demo directory, and compile and execute the program using the otelbuild tool to verify the dual pivot quicksort effect.
$ ./otelbuild -rule=conf.json -- main.go
$ ./main
== [3 4 4 6 7 9]To prevent SQL code injection, additional code can be injected into the database/sql::(*DB).Query() to check for potential injection risks in SQL statements and intercept them promptly.
Step 1: Create a hook folder, initialize it with go mod init hook, and then add the following hook.go code to be injected:
package hook
import (
"database/sql"
"errors"
"github.com/alibaba/opentelemetry-go-auto-instrumentation/pkg/api"
"log"
"strings"
)
func checkSqlInjection(query string) error {
patterns := []string{"--", ";", "/*", " or ", " and ", "'"}
for _, pattern := range patterns {
if strings.Contains(strings.ToLower(query), pattern) {
return errors.New("potential SQL injection detected")
}
}
return nil
}
func sqlQueryOnEnter(call api.CallContext, db *sql.DB, query string, args ...interface{}) {
if err := checkSqlInjection(query); err != nil {
log.Fatalf("sqlQueryOnEnter %v", err)
}
}Step 2: Write the following conf.json configuration to instruct the tool to inject hook code into the database/sql::(*DB). Query().
[{
"ImportPath": "database/sql",
"Function": "Query",
"ReceiverType": "*DB",
"OnEnter": "sqlQueryOnEnter",
"Path": "/path/to/hook" # Change to the local path of the hook code
}]Step 3: Write a test demo. Create a folder and initialize it with go mod init demo, and then add main.go.
package main
import (
"context"
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
"os"
"time"
)
func main() {
mysqlDSN := "test:test@tcp(127.0.0.1:3306)/test"
db, _ := sql.Open("mysql", mysqlDSN)
db.ExecContext(context.Background(), `CREATE TABLE IF NOT EXISTS usersx (id char(255), name VARCHAR(255), age INTEGER)`)
db.ExecContext(context.Background(), `INSERT INTO usersx (id, name, age) VALUE ( ?, ?, ?)`, "0", "foo", 10)
# Inject malicious code into SQL statements to capture all the information in the database table
maliciousAnd := "'foo' AND 1 = 1"
injectedSql := fmt.Sprintf("SELECT * FROM userx WHERE id = '0' AND name = %s", maliciousAnd)
db.Query(injectedSql)
}Step 4: Switch to the demo directory, and compile and execute the program using the otelbuild tool to verify the SQL injection prevention effect.
$ ./otelbuild -rule=conf.json -- main.go
$ docker run -d -p 3306:3306 -p 33060:33060 -e MYSQL_USER=test -e MYSQL_PASSWORD=test -e MYSQL_DATABASE=test -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql:8.0.36
$ ./mainIt can be observed that the binary file compiled with the otelbuild tool successfully detects a potential SQL injection attack and prints the corresponding log:
2024/11/04 21:12:47 sqlQueryOnEnter potential SQL injection detectedThe example can be found in the following link:
https://github.com/alibaba/opentelemetry-go-auto-instrumentation/tree/main/example/extension/sqlinject
If we plan to add traffic protection to grpc-go unary requests based on sentinel-golang, we can also inject middleware into the grpc client through automatic instrumentation.
Step 1: Create a hook folder, initialize it with go mod init hook, and then add the following hook.go code to be injected:
package hook
import (
"context"
"google.golang.org/grpc"
sentinel "github.com/sentinel-golang/api"
"github.com/sentinel-golang/core/base"
pkgapi "github.com/alibaba/opentelemetry-go-auto-instrumentation/pkg/api"
)
// Add traffic protection middleware at the gRPC client entry point
func newClientOnEnter(call pkgapi.CallContext, target string, opts ...grpc.DialOption) {
opts = append(opts, grpc.WithChainUnaryInterceptor(unaryClientInterceptor))
}
// Traffic protection middleware based on sentinel-golang
func unaryClientInterceptor(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
entry, blockErr := sentinel.Entry(
method,
sentinel.WithResourceType(base.ResTypeRPC),
sentinel.WithTrafficType(base.Outbound),
)
defer func() {
if entry != nil {
entry.Exit()
}
}()
if blockErr != nil {
return blockErr
}
return invoker(ctx, method, req, reply, cc, opts...)
}Step 2: Write the following conf.json configuration to instruct the tool to inject hook code into the google.golang.org/grpc::NewClient.
[{
"ImportPath": "google.golang.org/grpc",
"Function": "NewClient",
"OnEnter": "newClientOnEnter",
"Path": "/path/to/hook" # Change to the local path of the hook code
}]Step 3: Write a test demo. Create a folder and initialize it with go mod init demo, and then add main.go.
package main
import (
"context"
"fmt"
"google.golang.org/grpc"
pb "path/to/your/protobuf" // Replace with the path to your protobuf file
)
func main() {
// Connect to the GRPC server
conn, _ := grpc.Dial("localhost:50051", grpc.WithInsecure())
client := pb.NewYourServiceClient(conn)
// Send a gRPC request
response, _ := client.YourMethod(context.Background(), &pb.YourRequest{})
fmt.Println("Response: ", response)
}Step 4: Switch to the demo directory, and compile and execute the program using the otelbuild tool to verify the effect.
$ ./otelbuild -rule=conf.json -- main.go
$ ./mainThe approach is similar if you want to add protection rules for gRPC-go stream requests. In addition, if you want to enable your requests with canary routing, tag-based routing, percentage routing, and other capabilities for canary releases, you can also enhance the framework's load balancer as needed, which provides a high degree of autonomy and extensibility.
Golang's compile-time automatic instrumentation has successfully solved the cumbersome manual instrumentation issues in microservice monitoring and has been commercially launched on Alibaba Cloud's public cloud to provide customers with powerful monitoring capabilities. This technology was originally designed to allow users to easily insert monitoring code without changing the existing code, thereby enabling real-time monitoring and analysis of application performance. However, the applications of this technology have exceeded expectations, including service governance, code audit, application security, and code debugging. It has also shown potential in many unexplored areas.
We decided to open-source this innovative solution and contribute it to the OpenTelemetry community [4]. At present, we have reached an agreement on the contribution, and our code will be migrated to the OpenTelemetry community repository in the future. The open-sourcing of the solution not only promotes technical sharing and improvement but also helps us continuously explore its potential in more fields with the help of the community.
Finally, we sincerely invite you to try our commercial products [5] [6] and join our DingTalk group (open-source group: 102565007776, commercial group: 35568145) to jointly enhance the monitoring and service governance capabilities of Golang applications. Through collective effort, we believe we can bring a greater cloud-native experience to the Golang developer community.
[0] Programming Language and Compiler SIG in OpenAnolis: https://openanolis.cn/sig/java
[1] Go Automatic Instrumentation Open Source Project: https://github.com/alibaba/opentelemetry-go-auto-instrumentation
[2] Non-intrusive Instrumentation Technology for Golang Applications in OpenTelemetry: https://www.alibabacloud.com/blog/non-intrusive-instrumentation-technology-for-golang-applications-in-opentelemetry_601665
[3] Pattern-defeating Quicksort Algorithm Paper: https://arxiv.org/pdf/2106.05123
[4] Discussion of Project Contribution in the OpenTelemetry Community: https://github.com/open-telemetry/community/issues/1961
[5] Alibaba Cloud ARMS Go Agent Commercial Edition: https://www.alibabacloud.com/help/en/arms/tracing-analysis/monitor-go-applications/
[6] Alibaba Cloud MSE Go Agent Commercial Edition: https://www.alibabacloud.com/help/en/mse/getting-started/ack-microservice-application-access-mse-governance-center-golang-version
Analysis of Coolbpf Latest Features: Open-Sourcing of eNetSTL in the OpenAnolis Community
87 posts | 5 followers
FollowAlibaba Cloud Native - August 7, 2024
Alibaba Cloud Native Community - April 6, 2023
Alibaba Cloud Native Community - October 15, 2024
Alibaba Cloud Native Community - April 26, 2024
Alibaba Cloud Native Community - August 30, 2022
Alibaba Cloud Native Community - May 18, 2023
87 posts | 5 followers
Follow
Managed Service for OpenTelemetry
Allows developers to quickly identify root causes and analyze performance bottlenecks for distributed applications.
Learn More
Managed Service for Prometheus
Multi-source metrics are aggregated to monitor the status of your business and services in real time.
Learn More
Function Compute
Alibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.
Learn More
Elastic High Performance Computing Solution
High Performance Computing (HPC) and AI technology helps scientific research institutions to perform viral gene sequencing, conduct new drug research and development, and shorten the research and development cycle.
Learn MoreMore Posts by OpenAnolis