How to collect performance data from Go programs in a Kubernetes cluster -

pprof is a profiling tool of Go. It is used to analyze and improve the performance of Go programs and provides performance data on Go programs. You can use the Kubernetes Golang Pull module of the Full-stack Observability application provided by Simple Log Service to collect performance data from all Go programs in a Kubernetes cluster in a convenient manner.

Prerequisites

A Full-stack Observability instance is created. For more information, see Create an instance.
A monitoring component is installed in a Kubernetes cluster. For more information, see Install a monitoring component.
Note
You must use a Logtail image of v1.7.1.0-monitor-aliyun or later to install the monitoring component.

Implementation

In Kubernetes clusters, the DaemonSet deployment model is used for both the component that collects performance data from Go programs and the component that collects logs. During log collection, Logtail supports Kubernetes resource selection from different dimensions. During performance data collection from Go programs, Logtail can also filter data by using environment variables, Kubernetes labels, namespaces, pod names, and container names. The collection of performance data from Go programs involves the data of communication between Go processes, and Kubernetes resource selection may hit a large number of unrelated processes. In this case, Logtail introduces a special environment variable named ILOGTAIL_PROFILE_PORT to further narrow down the data scope.

Go provides a built-in exposure mechanism for the performance data provided by pprof. After you enable the built-in exposure mechanism in a Go program, Logtail can obtain the performance data over HTTP.

Preparations

Enable an exposure mechanism for performance data.
Before you collect pprof performance data from Go programs, you must enable an exposure mechanism. You can use the following code to enable the built-in exposure mechanism. You can also configure and enable a custom exposure mechanism. Make sure that the pprof performance data is exposed at the following address: {host}:{port}/debug/pprof/Port. In the following code, pprof performance data is exposed on port 8080.
```
package main

import (
	"net/http"
	_ "net/http/pprof"
)

func main() {

	// do something...

	_ = http.ListenAndServe(":8080", nil)
}
```
Configure the ILOGTAIL_PROFILE_PORT environment variable.
Set the ILOGTAIL_PROFILE_PORT environment variable to the port on which data is exposed. Example: ILOGTAIL_PROFILE_PORT=8080.

Create a Logtail configuration

Log on to the Simple Log Service console.
In the Log Application section, click the Intelligent O&M tab. Then, click Full-stack Observability.
On the Simple Log Service Full-stack Observability page, click the instance that you want to manage.
In the left-side navigation pane, click Performance Monitoring.
Click Enable if this is your first time to use Performance Monitoring in the instance.
In the left-side navigation tree, click Data Import. On the Data Access Configurations page, find Kubernetes Golang Pull.
The first time you create a Logtail configuration for host monitoring data, turn on the switch to go to the configuration page. If you created a Logtail configuration, click the icon to go to the configuration page.
Click Use Existing Machine Groups.
After the monitoring component is installed, Simple Log Service automatically creates a machine group named {instanceID}-{clusterID}-k8s-nodes. You can use this machine group.
Select the {instanceID}-{clusterID}-k8s-nodes machine group, move the machine group from the Source Server Groups section to the Applied Server Groups section, and then click Next.
Important
If the heartbeat status of the machine group is FAIL, you can click Automatic Retry. If the issue persists, see What do I do if a Logtail machine group has no heartbeats?

In the Specify Data Source step, configure the following parameters and click Complete.

Parameter	Description
General configurations
Config Name	The name of the Logtail configuration.
Cluster	The name of the Kubernetes cluster. After you configure this parameter, Simple Log Service adds a cluster=Cluster name tag to the performance data that is collected by using the Logtail configuration. Important Make sure that the cluster name is unique. Otherwise, data conflicts may occur.
Profile Whitelist	The type of data that you want to collect.
Collection Interval	The interval at which data is collected. Unit: seconds.
Timeout Period	The timeout period for data collection. Unit: seconds.
Maximum Body Size	The maximum size of data that can be collected. Unit: KB.
Kubernetes Selector configurations
Namespace	The regular expression that is used to match the names of namespaces. This parameter specifies the namespaces from which data is collected.
Pod Name	The regular expression that is used to match the names of pods. This parameter specifies the pods from which data is collected.
Container Name	The regular expression that is used to match the names of containers. This parameter specifies the containers from which data is collected.
Environment Variable Whitelist	The environment variable whitelist. The whitelist specifies the containers from which data is collected. This parameter is empty by default. If you leave this parameter empty, data is collected from all containers that run Go programs in a Kubernetes cluster. When you configure the environment variable whitelist, the Key parameter is required, and the value parameter is optional. If the EnvValue parameter is empty, containers whose environment variables contain the keys specified by EnvKey are matched. If the EnvValue parameter is not empty, containers whose environment variables consist of the key-value pairs specified by EnvKey and EnvValue are matched. By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to `NGINX_SERVICE_PORT` and set the EnvValue parameter to `^(80\|6379)$`, containers whose port number is 80 and containers whose port number is 6379 are matched. Key-value pairs are connected by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is matched.
Environment Variable Blacklist	The environment variable blacklist. The blacklist specifies the containers from which data is not collected. This parameter is empty by default, which indicates that data is collected from all containers. When you configure the environment variable blacklist, the Key parameter is required, and the value parameter is optional. If the EnvValue parameter is empty, containers whose environment variables contain the keys specified by EnvKey are filtered out. If the EnvValue parameter is not empty, containers whose environment variables consist of the key-value pairs specified by EnvKey and EnvValue are filtered out. By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to `NGINX_SERVICE_PORT` and set the EnvValue parameter to `^(80\|6379)$`, containers whose port number is 80 and containers whose port number is 6379 are matched. Key-value pairs are connected by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is filtered out.
K8s Label Whitelist	The Kubernetes label whitelist. The whitelist specifies the containers from which data is collected. When you configure the Kubernetes label whitelist, the Key parameter is required, and the value parameter is optional. If the LabelValue parameter is empty, containers whose Kubernetes labels contain the keys specified by LabelKey are matched. If the LabelValue parameter is not empty, containers whose Kubernetes labels consist of the key-value pairs specified by LabelKey and LabelValue are matched. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`), regular expression matching is performed. For example, if you set the LabelKey parameter to `app` and set the LabelValue parameter to `^(test1\|test2)$`, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched. Key-value pairs are connected by using the OR operator. If a container has a Kubernetes label that consists of one of the specified key-value pairs, the container is matched.
K8s Label Blacklist	The Kubernetes label blacklist. The blacklist specifies the containers from which data is not collected. When you configure the Kubernetes label blacklist, the Key parameter is required, and the value parameter is optional. If the LabelValue parameter is empty, containers whose Kubernetes labels contain the keys specified by LabelKey are filtered out. If the LabelValue parameter is not empty, containers whose Kubernetes labels consist of the key-value pairs specified by LabelKey and LabelValue are filtered out. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`), regular expression matching is performed. For example, if you set the LabelKey parameter to `app` and set the LabelValue parameter to `^(test1\|test2)$`, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched. Key-value pairs are connected by using the OR operator. If a container has a Kubernetes label that consists of one of the specified key-value pairs, the container is filtered out.
Container Label Whitelist	The container label whitelist. The whitelist specifies the containers from which data is collected. This parameter is empty by default. If you leave this parameter empty, data is collected from all containers that run Go programs in a Kubernetes cluster. When you configure the container label whitelist, the Key parameter is required, and the value parameter is optional. If the LabelValue parameter is empty, containers whose container labels contain the keys specified by LabelKey are matched. If the LabelValue parameter is not empty, containers whose container labels consist of the key-value pairs specified by LabelKey and LabelValue are matched. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to `io.kubernetes.container.name` and set the LabelValue parameter to `^(nginx\|cube)$`, a container named nginx and a container named cube are matched. Key-value pairs are connected by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is matched.
Container Label Blacklist	The container label blacklist. The blacklist specifies the containers from which data is not collected. This parameter is empty by default, which indicates that data is collected from all containers. When you configure the container label blacklist, the Key parameter is required, and the value parameter is optional. If the LabelValue parameter is empty, containers whose container labels contain the keys specified by LabelKey are filtered out. If the LabelValue parameter is not empty, containers whose container labels consist of the key-value pairs specified by LabelKey and LabelValue are filtered out. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to `io.kubernetes.container.name` and set the LabelValue parameter to `^(nginx\|cube)$`, a container named nginx and a container named cube are matched. Key-value pairs are connected by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is filtered out.

After you complete the configuration, Simple Log Service automatically creates assets such as Metricstores. For more information, see Assets.

What to do next

After you collect the pprof performance data from Go programs to Full-stack Observability, you can use the performance monitoring feature to troubleshoot performance issues. For more information, see Data query and Data comparison.