Optimal Multi-language Application Monitoring: ARMS eBPF Edition

By Guqi, Qianlu and Yanhong

As cloud-native technologies like Kubernetes and Serverless continue to transform R&D and O&M modes, the shift from monolithic to distributed, microservice-oriented application architectures has become more evident. As enterprises integrate more microservices using multiple languages, frameworks, and protocols, the complexity of microservices has increased. Quickly identifying and locating microservice issues through observability has become crucial for O&M personnel and developers.

Observability products need to ensure that the application's current operating architecture, upstream and downstream dependencies, and traffic are normal. However, challenges arise as the number of microservice types increases, particularly in implementing event tracking with different languages, frameworks, and protocols, and the potential impact on performance. To address these issues and meet the end-to-end application monitoring requirements, Application Real-Time Monitoring Service (ARMS) has introduced the application monitoring eBPF edition to enhance the entire application monitoring system through eBPF technology. The following section will focus on the application monitoring eBPF edition.

1. Scenario

As the number of microservices surges with the development of businesses, common application observability cannot handle complex scenarios involving specific languages used across different business sectors. Additionally, most non-Java languages require the use of an SDK to access monitoring, significantly reducing the efficiency of research and development personnel. This is where the application monitoring eBPF edition comes into play. It is primarily designed to address application observability challenges in multi-language scenarios, providing non-intrusive and language-independent observability capabilities. For microservices in the Go language, specific monitoring capabilities, such as GC monitoring, memory dump, and profiling, are supported.

2. What is eBPF?

The eBPF technology is a sandbox program running on the Linux kernel without changing any source code or loading any kernel modules. It has the following features:

Non-intrusiveness: Low cost, no need to restart the application, no need to modify the code.
Dynamic programmability: No need to restart the probe and dynamically issue the collection script.
High performance: Native code compiled with JIT.
Security: The verifier mechanism ensures stable kernel operation.

3. Use eBPF to Collect Application Monitoring Data

In the Kubernetes environment, the eBPF agent is deployed on each node in Daemonset mode. It runs in the kernel through the program of Load eBPF in the user state, collects data, sends it to the user state for data processing, and then forms different types of metrics.

eBPF has three most important advantages in the observability field:

Non-intrusive: You do not need to modify any business code. The eBPF agent is deployed on nodes to monitor the network data of applications without business interruptions.
Multi-language, multi-framework, and multi-protocol: The eBPF agent is non-intrusive, so there are no restrictions on the languages, frameworks, or protocols of microservices. After eBPF collects network data requested by applications, the eBPF agent can parse and identify different types of protocols and support different development frameworks.
Full-stack coverage: All data from the network layer to the application layer are collected to realize full stack coverage. Different types of data in kernel space and user space can be collected, such as TCP/UDP and Socket in kernel space and Net/HTTP libraries in the Go language in user space.

4. Use eBPF for Application Observation

Global topology: This topology can correctly reflect the calling relationships of the currently running applications, help users understand the overall microservice architecture, and detect problems by comparing the expected architecture. This topology is usually used in scenarios such as new application launches, server launches in new regions, and overall trace sorting.

Application details: The information about microservices is displayed here, such as the number of requests, number of errors, average time consumption, number of instances, interfaces, CPU usage, and memory usage.

Application topology: You can view the status of upstream requests, downstream dependencies, and service instance running state. It can help you locate exceptions and is usually used after locating a certain exception node.

Instance monitoring: You can view the number of requests, number of errors, and average latency of a single instance of the running application. You can also view the container monitoring of pods that run on a single instance.

Intelligent alerts: You can configure active alert capability for applications, such as the number of requests, latency, number of errors, and HTTP status code.

5. How to Access eBDF

5.1 Prerequisites

You have activated ARMS. For more information, see Activate and upgrade ARMS[1].
A Kubernetes cluster is created. Currently, ARMS eBPF only supports access to ACK clusters. You can create each cluster as follows:
- Create an ACK dedicated cluster[2]
- Create an ACK managed cluster[3]
You have checked that the current environment supports ARMS eBPF. For more information, see Requirements and limits for the operating environment of ARMS eBPF[4].

5.2 Access Description

Install the following monitoring components for a Kubernetes cluster to connect the cluster to ARMS eBPF:

Install the Prometheus monitoring agent

Note: Metric data of ARMS eBPF depends on Prometheus monitoring. Therefore, you must install the Prometheus monitoring agent first.

ARMS eBPF component

Before you install the ARMS application monitoring agent, check whether your ACK Serverless cluster has ARMS Addon Token. Otherwise, an error report will occur because you do not have permission. If the cluster has ARMS Addon Token, ARMS performs passwordless authentication. After the component is installed, you can use the ARMS eBPF. If the cluster does not have ARMS Addon Token, you need to manually add permission policies on ARMS and Tracing Analysis to complete authentication. The following describes how to check whether ARMS Addon Token exists and how to attach permission policies:

How to check whether ARMS Addon Token is enabled for a cluster:

1) Log on to the ACK console[5]. In the left-side navigation pane, click Clusters. On the Clusters page, click the name of the cluster to go to the cluster details page.

2) In the left-side navigation pane, choose Configurations > Secrets. In the upper part of the page, select kube-system from the Namespace drop-down list and check whether addon.arms.token is displayed on the Secrets page.

Complete authorization on the Cloud Resource Access Authorization page.

1) Log on to the ACK console.

2) In the left-side navigation pane, click Clusters. On the Clusters page, click the name of the cluster that you want to view.

3) On the Cluster Information page, click the Cluster Resources tab and click the link next to Worker RAM Role.

4) On the Permissions tab, click Grant Permission.

5) Select AliyunARMSFullAccess and AliyunTracingAnalysisFullAccess System Permissions and click OK.

Note: Managed Kubernetes clusters: For some managed Kubernetes clusters, the ARMS Addon Token may not exist. Therefore, for managed Kubernetes clusters, it is recommended that you first check whether the ARMS Addon Token exists. If ARMS Addon Token does not exist, you need to manually grant permissions.

By default, ACK dedicated clusters do not have ARMS Addon Token. You need to manually grant permissions.

5.3 Install the Prometheus Monitoring Agent

1) Log on to the ACK console. On the Clusters page, click the name of the cluster that you want to manage.

2) In the left-side navigation pane, choose O&M Management> Prometheus Monitoring.

3) If the Prometheus Monitoring page shows that the Prometheus Service agent is not installed, you need to install it.

4) On the Prometheus Monitoring page, click Install.

ACK console automatically installs the Prometheus Monitoring agent.

Note: The default namespace is arms-prom.

5.4 Install the ARMS eBPF Component

a. Install the component for an existing application

1) Log on to the ACK console. On the Clusters page, click the name of the cluster that you want to manage.

2) In the left-side navigation pane, choose Operations> Cluster Topology.

3) On the Cluster Topology page, click Install. The ARMS eBPF component is automatically installed by the ACK console.

Note: The default namespace is arms-prom.

b. Install the component when you create a new application

1) Log on to the ACK console. On the Clusters page, find the cluster that you want to manage and click Applications in the Actions column.

2) On the Deployments page, click Create from YAML in the upper-right corner.

3) Select Sample Template and add the following labels to the spec.template.metadata hierarchy in the Template (YAML format) section.

labels:
armseBPFAutoEnable: "on"
armseBPFCreateAppName: "<your-deployment-name>" // Replace <your-deployment-name> with the name of your application.

The following YAML template provides an example of how to create a deployment application and enable ARMS eBPF:

apiVersion: v1
kind: Namespace
metadata:
  name: arms-demo
---
apiVersion: apps/v1 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
  name: arms-springboot-demo
  namespace: arms-demo
  labels:
    app: arms-springboot-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: arms-springboot-demo
  template:
    metadata:
      labels:
        app: arms-springboot-demo
        armseBPFAutoEnable: "on"
        armseBPFCreateAppName: "arms-k8s-demo"
    spec:
      containers:
        - resources:
            limits:
              cpu: 0.5
          image: registry.cn-hangzhou.aliyuncs.com/arms-docker-repo/arms-springboot-demo:v0.1
          imagePullPolicy: Always
          name: arms-springboot-demo
          env:
            - name: SELF_INVOKE_SWITCH
              value: "true"
            - name: COMPONENT_HOST
              value: "arms-demo-component"
            - name: COMPONENT_PORT
              value: "6666"
            - name: MYSQL_SERVICE_HOST
              value: "arms-demo-mysql"
            - name: MYSQL_SERVICE_PORT
              value: "3306"
---
apiVersion: apps/v1 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
  name: arms-springboot-demo-subcomponent
  namespace: arms-demo
  labels:
    app: arms-springboot-demo-subcomponent
spec:
  replicas: 2
  selector:
    matchLabels:
      app: arms-springboot-demo-subcomponent
  template:
    metadata:
      labels:
        app: arms-springboot-demo-subcomponent
        armseBPFAutoEnable: "on"
        armseBPFCreateAppName: "arms-k8s-demo-subcomponent"
    spec:
      containers:
        - resources:
            limits:
              cpu: 0.5
          image: registry.cn-hangzhou.aliyuncs.com/arms-docker-repo/arms-springboot-demo:v0.1
          imagePullPolicy: Always
          name: arms-springboot-demo-subcomponent
          env:
            - name: SELF_INVOKE_SWITCH
              value: "false"
            - name: MYSQL_SERVICE_HOST
              value: "arms-demo-mysql"
            - name: MYSQL_SERVICE_PORT
              value: "3306"
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: arms-demo-component
  name: arms-demo-component
  namespace: arms-demo
spec:
  ports:
    # the port that this service should serve on
    - name: arms-demo-component-svc
      port: 6666
      targetPort: 8888
  # label keys and values that must match in order to receive traffic for this service
  selector:
    app: arms-springboot-demo-subcomponent
---
apiVersion: apps/v1 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
  name: arms-demo-mysql
  namespace: arms-demo
  labels:
    app: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - resources:
            limits:
              cpu: 0.5
          image: registry.cn-hangzhou.aliyuncs.com/arms-docker-repo/arms-demo-mysql:v0.1
          name: mysql
          ports:
            - containerPort: 3306
              name: mysql
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: mysql
  name: arms-demo-mysql
  namespace: arms-demo
spec:
  ports:
    # the port that this service should serve on
    - name: arms-mysql-svc
      port: 3306
      targetPort: 3306
  # label keys and values that must match in order to receive traffic for this service
  selector:
    app: mysql
--

6. ARMS and ARMS eBPF Edition

ARMS provides non-intrusive observability for Java services. For Java applications, we recommend you access ARMS. ARMS eBPF edition provides language-independent observability because it is non-intrusive. Currently, ARMS eBPF edition only supports the HTTP protocol.

Scenarios	ARMS	ARMS eBFP Edition
● Pure Java applications ● Deployment methods: ECS, container, EDAS/SAE hosting, IDC/ other cloud service providers ● Expect no intrusion into business code ● Scenario: application health monitoring/alerting ● Scenario: distributed full Tracing Analysis ● Scenario: perform code-level analysis and troubleshooting for errors, slowdowns, and exceptions ● Scenario: continuous performance profiling to find code hotspots	★★★★★ Highly recommended	Not recommended
● Multi-language applications, including Java ● Deployment methods: ECS, container, EDAS/SAE hosting, IDC/other cloud service providers ● Scenario: application health monitoring/alerting ● Scenario: distributed full Tracing Analysis ● Scenario: perform code-level analysis and troubleshooting for errors, slowdowns, and exceptions ● Scenario: continuous performance profiling to find code hotspots	★★★★★ Highly recommended for Java applications	★★★ Recommended when a non-intrusive solution is strongly desiredEquipped with regular application performance monitoring/alerting capabilities Limits: weak Tracing Analysis and performance profiling capability
● Multi-language applications, including Java ● Deployment method: Container deployment, and the system kernel is high (or can accept kernel upgrade) ● Expect no intrusion into business code ● Scenario: application health monitoring/alerting	★★★★★ Highly recommended for Java applications	★★★★ Equipped with regular application performance monitoring/alerting capabilities No advanced application diagnostics or Tracing Analysis is required

References

[1] Activate and upgrade ARMS
https://www.alibabacloud.com/help/en/arms/getting-started/activate-arms#concept-65257-zh
[2] Create an ACK dedicated cluster
https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/create-an-ack-dedicated-cluster#task-skz-qwk-qfb
[3] Create an ACK managed cluster
https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/create-an-ack-managed-cluster-2#task-skz-qwk-qfb
[4] AMS eBPF edition operating environment requirements and limits
https://www.alibabacloud.com/help/en/arms/application-monitoring-ebpf/getting-started/untitled-document-1698028648754
[5] ACK console
https://account.aliyun.com/login/login.htm?oauth_callback=https%3A%2F%2Fcs.console.aliyun.com%2F

Community

Optimal Multi-language Application Monitoring: ARMS eBPF Edition

1. Scenario

2. What is eBPF?

3. Use eBPF to Collect Application Monitoring Data

4. Use eBPF for Application Observation

5. How to Access eBDF

5.1 Prerequisites

5.2 Access Description

How to check whether ARMS Addon Token is enabled for a cluster:

Complete authorization on the Cloud Resource Access Authorization page.

5.3 Install the Prometheus Monitoring Agent

5.4 Install the ARMS eBPF Component

a. Install the component for an existing application

b. Install the component when you create a new application

6. ARMS and ARMS eBPF Edition

References

Read previous post:

Read next post:

Alibaba Cloud Native

You may also like

Comments

Alibaba Cloud Native

Related Products

Application Real-Time Monitoring Service

Managed Service for Prometheus

Real-Time Livestreaming Solutions

Enterprise Distributed Application Service